Training architecture, method and system for graph neural network model, and server

Abstract

The present application relates to the field of memory design. Disclosed are a training architecture, method and system for a graph neural network model, and a server, which solve the problem of limited memory capacity in traditional servers. In the architecture, by means of the connection of N processors, N compute express link devices and N graphics processing units, the processors divide graph structure data into N pieces of first sub-graph structure data, and allocate the N pieces of first sub-graph structure data to the N compute express link devices, so that the N compute express link devices perform neighbor node sampling on the first sub-graph structure data to obtain second sub-graph structure data, and send the second sub-graph structure data to the graphics processing units for the training of a graph neural network model. By providing N compute express link devices, the present application realizes memory expansion of a single server and parallel processing and storage in a single server, thereby solving the problems of storage and training of large-scale graph structure data, and also reducing the network communication overheads and architecture design complexity and improving the training and inference efficiencies.

Description

A training architecture, method, system and server for a graph neural network model

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to a Chinese patent application filed with the Patent Office of China on February 27, 2024, with application number 202410214274.4 and entitled “A training architecture, method, system and server for a graph neural network model,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of memory design, and in particular to a training architecture, method, system, and server for a graph neural network model. Background Art

[0004] In recent years, with the widespread application of GNN (Graph Neural Network) models in various fields, the demand for training and reasoning on large-scale graph-structured data has become increasingly urgent. However, unlike image or language models, the input data of GNN models is unstructured graph data, and real-world graph data, such as social networks and protein networks, is extremely large. To achieve better training and reasoning results on larger graph data, the current approach is to increase the scale of training graph data. However, traditional servers have limited memory capacity, and the increasing number of nodes in graph neural networks means that the storage requirements for graph data are also increasing.

[0005] Due to the limitations of current server architectures, the scalable memory capacity on a single server is limited. Therefore, distributed computing frameworks are currently used to address this issue. Specifically, graph-structured data is partitioned and distributed across multiple servers for parallel processing. However, this introduces additional network communication overhead and is relatively complex in architectural design.

[0006] Summary of the Invention

[0007] In a first aspect, the present application provides a training architecture for a graph neural network model, applied to a server, including:

[0008] N processors, where N is an integer greater than one;

[0009] One or more graphics processors connected to each processor in a one-to-one correspondence, and each graphics processor is connected to each other;

[0010] Each processor is configured to send a node feature vector corresponding to each node in each second subgraph structure data of the graph structure data input by the user to a correspondingly connected graphics processor;

[0011] Each graphics processor is used to store the graph neural network model, and upon receiving the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data, train the graph neural network model according to the second subgraph structure data and the node feature vector to obtain gradient information;

[0012] a master graphics processor, configured to receive gradient information sent by each graphics processor, aggregate and optimize the received gradient information to obtain optimized gradient information, and send the optimized gradient information to each graphics processor, wherein the master graphics processor is any graphics processor among the multiple graphics processors connected to each processor, or a graphics processor not connected to each processor;

[0013] Each graphics processor is also used to update the model parameters of the stored graph neural network model based on the received optimization gradient information.

[0014] Preferably, a training architecture of a graph neural network model further includes:

[0015] N computing fast link devices are connected to N processors in a one-to-one correspondence;

[0016] Each processor is further configured to distribute corresponding first subgraph structure data from the N first subgraph structure data divided from the graph structure data to a corresponding connected fast computing link device, and when the corresponding connected fast computing link device obtains second subgraph structure data, send a node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor;

[0017] The computing fast link device is used to store the first subgraph structure data corresponding to itself, sample neighbor nodes of the nodes in the first subgraph structure data corresponding to itself, obtain the sampled second subgraph structure data, and send the second subgraph structure data to the corresponding connected graphics processor.

[0018] Preferably, it also includes:

[0019] N host memories, connected one-to-one with N processors, for storing node feature vectors corresponding to each node in the graph structure data;

[0020] The processor is further configured to store a node feature vector corresponding to each node in the graph structure data into a host memory;

[0021] When the corresponding connected computing fast link device obtains the second subgraph structure data, sending the node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor includes:

[0022] When the corresponding connected computing fast link device obtains the second subgraph structure data, determining the node identifier of each node in the second subgraph structure data; and

[0023] A node feature vector corresponding to the node identifier is read from the host memory, and the node feature vector is sent to the corresponding connected graphics processor.

[0024] Preferably, the plurality of graphics processors connected to each processor include a master graphics processor and a slave graphics processor;

[0025] The slave graphics processor is further configured to obtain gradient information of the first slave model parameters after training the graph neural network model according to the second subgraph structure data and the node feature vector, and send the gradient information of the first slave model parameters to the master graphics processor;

[0026] The main graphics processor is also used to obtain the gradient information of the first main model parameters after training the graph neural network model according to the second subgraph structure data and the node feature vector, receive the gradient information of the first slave model parameters sent by each slave graphics processor, and obtain optimized gradient information based on the gradient information of the first main model parameters and the gradient information of each first slave model parameter through aggregation and optimization processing.

[0027] Preferably, the optimized gradient information is obtained by aggregating and optimizing the gradient information of the first master model parameters and the gradient information of each first slave model parameter, including:

[0028] Average processing is performed on the gradient information of the first master model parameter and the gradient information of each first slave model parameter to obtain average gradient information, and the optimized gradient information includes the average gradient information.

[0029] Preferably, the master graphics processor and the slave graphics processor are determined in the following manner: the master graphics processor and the slave graphics processor are determined according to the user's configuration instructions, or, after each training is completed, a graphics processor is randomly selected to be the master graphics processor, and other graphics processors except the master graphics processor are determined to be the slave graphics processors.

[0030] Preferably, the processor is further configured to divide each first subgraph structure data into M third subgraph structure data, and store the M third subgraph structure data corresponding to each first subgraph structure data in the computing fast link device, and when the computing fast link device obtains the fourth subgraph structure data, send the node feature vector corresponding to each node in the fourth subgraph structure data to the graphics processor, where M is an integer greater than one;

[0031] a computing fast link device, specifically configured to store M third subgraph structure data corresponding to itself, and sequentially sample neighboring nodes for each node in each third subgraph structure data to obtain sampled fourth subgraph structure data, and send the fourth subgraph structure data to a graphics processor;

[0032] The graphics processor is specifically used to store the graph neural network model, and sequentially receive the fourth subgraph structure data sent by the computing fast link device and the node feature vector corresponding to each node in the fourth subgraph structure data sent by the processor, train the graph neural network model according to the fourth subgraph structure data and the node feature vector, and obtain gradient information.

[0033] Preferably, the multiple graphics processors connected to each processor include a master graphics processor and a slave graphics processor;

[0034] The slave graphics processor is further configured to train the graph neural network model based on the node feature vector corresponding to each node in the Mth fourth subgraph structure data sent by the computing fast link device and the Mth fourth subgraph structure data sent by the processor, obtain gradient information of the second slave model parameters, and send the gradient information of the second slave model parameters to the master graphics processor;

[0035] The main graphics processor is also used to train the graph neural network model based on the node feature vector corresponding to each node in the Mth fourth subgraph structure data sent by the computing fast link device and the Mth fourth subgraph structure data sent by the processor, obtain the gradient information of the second main model parameters, receive the gradient information of the second slave model parameters sent by the slave graphics processor, and obtain optimized gradient information based on the gradient information of the second main model parameters and the gradient information of each second slave model parameter through aggregation and optimization processing.

[0036] Preferably, the first subgraph structure data corresponding to the N first subgraph structure data divided from the graph structure data is obtained by the following steps:

[0037] Obtaining graph structure data input by a user, and determining the number of first nodes in the graph structure data; and

[0038] The graph structure data is divided into N first sub-graph structure data according to the first number of nodes, and the difference in the number of nodes between every two first sub-graph structure data is not greater than a first preset number.

[0039] Preferably, each first subgraph structure data is divided into M third subgraph structure data, including:

[0040] Obtaining the second node number and edge number of the target node included in the first subgraph structure data; and

[0041] Dividing the first subgraph structure data into M third subgraph structure data according to the second number of nodes and the number of edges;

[0042] The difference in the number of target nodes between any two third subgraph structure data is not greater than the second preset number, and the difference in the number of edges is not greater than the third preset number.

[0043] Preferably, the first subgraph structure data is divided into M third subgraph structure data according to the second number of nodes and the number of edges, including:

[0044] Divide the target node into M groups evenly according to the number of second nodes, and obtain M groups of target child nodes;

[0045] Divide the edges into M groups evenly according to the number of target sub-nodes and edges in each group, and obtain M groups of edge data corresponding to the M groups of target sub-nodes; and

[0046] The M groups of target sub-nodes and the edge data corresponding to themselves are determined as M third sub-graph structure data.

[0047] Preferably, the processor is further configured to obtain the number of first neighbor node samples configured for each target node in the first subgraph structure data, divide the number of first neighbor node samples into M equal parts, obtain M groups of second neighbor node sample numbers, and distribute the second neighbor node sample numbers to the M groups of target subnodes in a one-to-one correspondence;

[0048] Neighbor node sampling is performed on each node in each third subgraph structure data to obtain sampled fourth subgraph structure data, including:

[0049] Neighbor sampling is performed on each target node according to the number of second neighbor node samples corresponding to itself to obtain the sampled fourth subgraph structure data.

[0050] Preferably, the processor is also used to determine the edge nodes in each first subgraph structure data and the edge neighbor nodes in other first subgraph structures connected to the edge nodes, and store the edge neighbor nodes and the connection relationship between the edge neighbor nodes and the edge nodes in the computing fast link device corresponding to the first subgraph structure where the edge nodes are located.

[0051] Preferably, the computing fast link device includes a computing unit, a private second device memory, and a processor-managed first device memory, and a one-to-one mapping between a portion of the processor's physical addresses and the physical addresses of the first device memory;

[0052] Allocating corresponding first subgraph structure data among the N first subgraph structure data divided from the graph structure data to corresponding connected computing fast link devices includes:

[0053] Allocating corresponding first subgraph structure data among the N first subgraph structure data to a second device memory of a corresponding connected computing fast link device;

[0054] A second device memory of the computing fast link device is used to store first subgraph structure data corresponding to the computing fast link device;

[0055] a computing unit, configured to sample neighbor nodes of nodes in the first subgraph structure data according to the first subgraph structure data in the memory of the second device to obtain sampled second subgraph structure data, send the second subgraph structure data to the graphics processor, and write node identifiers in the second subgraph structure data into the memory of the first device;

[0056] The first device memory is used to store node identifiers in the second subgraph structure data;

[0057] When the corresponding connected computing fast link device obtains the second subgraph structure data, sending the node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor includes:

[0058] The node identifiers in the memory of the first device are obtained, and node feature vectors corresponding to the node identifiers are determined according to the node identifiers, and the node feature vectors are sent to the graphics processor.

[0059] Preferably, the processor is also used to divide the device memory of the computing fast link device into a first device memory and a second device memory according to the device memory capacity of the computing fast link device and a preset ratio when initializing the computing fast link device, and map the physical address in the first device memory to part of its own physical address one by one.

[0060] Preferably, the preset ratio is determined according to the type of graph structure data, the data size of the first subgraph structure data, and the data size of the node identifier corresponding to the first subgraph structure data.

[0061] Preferably, the processor is further configured to divide each first subgraph structure data into M third subgraph structure data, and store each node identifier in the M third subgraph structure data corresponding to each first subgraph structure data in a first device memory of the fast link computing device; when the fast link computing device obtains fourth subgraph structure data, obtain a node identifier of each node in the fourth subgraph structure data stored in the first device memory, obtain a node feature vector of each node in the fourth subgraph structure data based on the node identifier of each node in the fourth subgraph structure data, and send the node feature vector to the graphics processor, where M is an integer greater than one;

[0062] The first device memory of the computing fast link device is specifically used to store each node identifier in the M third subgraph structure data corresponding to the first device memory;

[0063] a computing unit of the fast link computing device, specifically configured to sequentially sample neighbor nodes for each node in each of the third subgraph structure data according to each node identifier in the third subgraph structure data, obtain sampled fourth subgraph structure data, and write the node identifier of each node in the fourth subgraph structure data into the memory of the first device;

[0064] The graphics processor is specifically used to store the graph neural network model, and sequentially receive the fourth subgraph structure data sent by the computing fast link device and the node feature vector corresponding to each node in the fourth subgraph structure data sent by the processor, train the graph neural network model according to the fourth subgraph structure data and the node feature vector, and update the model parameters of the graph neural network model stored in itself after the training is completed.

[0065] Preferably, the computing fast link device further comprises a memory buffer connected to the computing unit;

[0066] The processor is further configured to, after dividing the graph structure data into N first subgraph structure data, sort the N first subgraph structure data according to the size of the degree distribution;

[0067] The memory buffer is used to store the data of nodes and edges in the first subgraph structure data with the largest degree distribution, so as to provide the data to the computing unit.

[0068] Preferably, the memory buffer is specifically used to store data of part of the nodes and part of the edges in the first subgraph structure data with the largest degree distribution when the data of the nodes and edges in the first subgraph structure data with the largest degree distribution is larger than its own memory capacity.

[0069] In a second aspect, the present application further provides a training method for a graph neural network model, which is applied to a computing fast link device. The training method for the graph neural network model includes:

[0070] Obtaining the first subgraph structure data corresponding to itself after the graph structure data is divided and sent by the processor;

[0071] Sampling neighbor nodes of nodes in the first subgraph structure data to obtain sampled second subgraph structure data;

[0072] Sending the second subgraph structure data to the graphics processor, and triggering the processor to send the node feature vector corresponding to each node in the second subgraph structure data to the graphics processor; and

[0073] The graphics processor is triggered to train the graph neural network model based on the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data.

[0074] In a third aspect, the present application further provides a training system for a graph neural network model, which is applied to a computing fast link device. The training system for the graph neural network model includes:

[0075] an acquiring unit, configured to acquire first subgraph structure data corresponding to itself after the graph structure data is divided and sent by the processor;

[0076] a sampling unit, configured to sample neighboring nodes of nodes in the first subgraph structure data to obtain sampled second subgraph structure data;

[0077] a sending unit, configured to send the second subgraph structure data to the graphics processor, and trigger the processor to send a node feature vector corresponding to each node in the second subgraph structure data to the graphics processor; and

[0078] The training unit is used to trigger the graphics processor to train the graph neural network model based on the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data.

[0079] Fourthly, the present application also provides a server, including a training architecture for the graph neural network model as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the prior art and the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0081] FIG1 is a schematic diagram of a training architecture of a graph neural network model provided by one or more embodiments of the present application;

[0082] FIG2 is a schematic diagram of data divided into a third subgraph structure according to one or more embodiments of the present application;

[0083] FIG3 is a schematic diagram of K-order neighbor node sampling provided by one or more embodiments of the present application;

[0084] FIG4 is a schematic diagram of dividing graph structure data into first sub-graph structure data according to one or more embodiments of the present application;

[0085] FIG5 is a flowchart of the workflow of each module in a training architecture of a graph neural network model provided by one or more embodiments of the present application;

[0086] FIG6 is a schematic diagram of a single processor training method according to one or more embodiments of the present application;

[0087] FIG7 is a schematic diagram of a training method for a graph neural network model provided by one or more embodiments of the present application;

[0088] Figure 8 is a schematic diagram of a training system for a graph neural network model provided by one or more embodiments of the present application. DETAILED DESCRIPTION

[0089] The core of this application is to provide a training architecture, method, system and server for a graph neural network model, set up N fast computing link devices, realize memory expansion of a single server, and parallel processing and storage within a single server, solve the storage and training problems of large-scale graph structure data, while reducing network communication overhead and architectural design complexity, and improving training and reasoning efficiency.

[0090] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0091] In a first aspect, the present application provides a training architecture for a graph neural network model, which is applied to a server and includes:

[0092] N processors, where N is an integer greater than one;

[0093] One or more graphics processors connected to each processor in a one-to-one correspondence, and each graphics processor is connected to each other;

[0094] Each processor is configured to send a node feature vector corresponding to each node in each second subgraph structure data of the graph structure data input by the user to a correspondingly connected graphics processor;

[0095] Each graphics processor is used to store the graph neural network model, and upon receiving the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data, train the graph neural network model according to the second subgraph structure data and the node feature vector to obtain gradient information;

[0096] a master graphics processor, configured to receive gradient information sent by each graphics processor, aggregate and optimize the received gradient information to obtain optimized gradient information, and send the optimized gradient information to each graphics processor, wherein the master graphics processor is any graphics processor among the multiple graphics processors connected to each processor, or a graphics processor not connected to each processor;

[0097] Each graphics processor is also used to update the model parameters of the stored graph neural network model based on the received optimization gradient information.

[0098] Specifically, any processor can be connected to one or more graphics processors, and each graphics processor can independently perform model training on the graph neural network model. Each graphics processor trains the graph neural network model based on the second subgraph structure data and the node feature vector to obtain gradient information. The obtained gradient information is used to guide the adjustment of the model parameters of the graph neural network model. The main graphics processor can be an independently configured graphics processor other than one or more graphics processors connected to each processor in a one-to-one correspondence, or it can be any graphics processor connected thereto. The main graphics processor is responsible for collecting the gradient information of other graphics processors, obtaining optimized gradient information, and transmitting the optimized gradient information to other graphics processors so that other graphics processors can update the model parameters. The optimized gradient information can be obtained by averaging the gradient information output by each graphics processor, that is, the optimized gradient information is the average gradient information.

[0099] In one embodiment, a training architecture of a graph neural network model of the present application further includes:

[0100] N computing fast link devices are connected to N processors in a one-to-one correspondence;

[0101] Each processor is further configured to distribute corresponding first subgraph structure data from the N first subgraph structure data divided from the graph structure data to a corresponding connected fast computing link device, and when the corresponding connected fast computing link device obtains second subgraph structure data, send a node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor;

[0102] The computing fast link device is used to store the first subgraph structure data corresponding to itself, sample neighbor nodes of the nodes in the first subgraph structure data corresponding to itself, obtain the sampled second subgraph structure data, and send the second subgraph structure data to the corresponding connected graphics processor.

[0103] Specifically, the process of dividing the graph structure data input by the user into N first sub-graph structure data can be that another processing device independent of the processor divides the graph structure data input by the user and transmits the N first sub-graph structure data obtained by the division to the processor. The processor distributes each first sub-graph structure data to the corresponding connected computing fast link device.

[0104] In one example, as shown in Figure 1, a processor can be represented as a non-uniform memory access node.

[0105] N non-uniform memory access nodes, where N is an integer greater than one;

[0106] N computing fast link devices are connected to N non-uniform memory access nodes in a one-to-one correspondence;

[0107] K×N graphics processors are connected to N non-uniform memory access nodes in a one-to-one correspondence, and the N graphics processors are connected to each other, where K is an integer greater than one;

[0108] The non-uniform memory access node is used to obtain graph structure data input by a user, divide the graph structure data into N first subgraph structure data, and distribute the N first subgraph structure data one-to-one to N computing fast link devices. When the computing fast link device obtains second subgraph structure data, the node feature vector corresponding to each node in the second subgraph structure data is sent to the graphics processor;

[0109] A computing fast link device is configured to store first subgraph structure data corresponding to itself, perform neighbor node sampling on nodes in the first subgraph structure data corresponding to itself, obtain sampled second subgraph structure data, and send the second subgraph structure data to a graphics processor;

[0110] A graphics processor is used to store the graph neural network model and, upon receiving the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data, train the graph neural network model according to the second subgraph structure data and the node feature vector.

[0111] This embodiment describes a training architecture for a graph neural network model, which is used for parallel computing on a single server. Specifically, the processor is used to obtain the graph structure data input by the user and divide it into N first sub-graph structure data. Each processor is connected to a computing fast link device and a graphics processor. The computing fast link device is used to store and process the first sub-graph structure data corresponding to it. They sample neighbor nodes for the nodes in the first sub-graph structure data to obtain the sampled second sub-graph structure data and send it to the corresponding graphics processor. The graphics processor is used to store the graph neural network model and train the graph neural network model upon receiving the second sub-graph structure data and the node feature vector corresponding to each node in the second sub-graph structure data. The graphics processors are also connected to each other to achieve parallel training on a single server.

[0112] The purpose of the entire architecture is to set up N processors on a single server and increase the number of computing fast link devices accordingly, so that a single server can process graph structure data in parallel faster, thereby improving training speed. In addition, each computing fast link device can store and process more first subgraph structure data, thereby expanding the memory capacity of a single server. These beneficial effects can greatly improve the training efficiency and performance of graph neural network models. In practical applications, expanding N computing high-speed link devices can also reduce network communication overhead, because more processing tasks can be completed locally without the need for frequent data transmission.

[0113] Specifically, Figure 1 includes four non-uniform memory access domains (non-uniform memory access domain 0, non-uniform memory access domain 1, non-uniform memory access domain 2, and non-uniform memory access domain 3). Each non-uniform memory access domain includes a host memory, a non-uniform memory access node, a computing fast link device, a graphics processor, and a memory buffer connected to the graphics processor.

[0114] By dividing graph-structured data into multiple subgraphs and processing them in parallel on a single server, we address the memory limitations of individual servers. When processing its own subgraph, each node uses a fast-link computing device to sample neighboring nodes, obtain the required data, and send it to the corresponding GPU for training. This architectural design fully utilizes the computing and storage resources of multiple servers, improving the training efficiency of graph neural network models.

[0115] Based on the above embodiment:

[0116] In one embodiment, it further includes:

[0117] N host memories, connected one-to-one with N processors, for storing node feature vectors corresponding to each node in the graph structure data;

[0118] The processor is further configured to, after obtaining the graph structure data input by the user, store the node feature vector corresponding to each node in the graph structure data in the host memory;

[0119] When the corresponding connected computing fast link device obtains the second subgraph structure data, sending the node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor includes:

[0120] When the corresponding connected computing fast link device obtains the second subgraph structure data, determining the node identifier of each node in the second subgraph structure data;

[0121] A node feature vector corresponding to the node identifier is read from the host memory, and the node feature vector is sent to the corresponding connected graphics processor.

[0122] In this embodiment, in addition to the components described above, N host memories are introduced, and each host memory is connected to a corresponding processor. The host memory is used to store the node feature vector corresponding to each node in the graph structure data. After the processor obtains the graph structure data input by the user, the node feature vector corresponding to each node in the graph structure data is stored in the host memory. In this way, the node feature vector associated with each node in the graph structure data is saved in the host memory. When the fast link device is calculated to obtain the second sub-graph structure data, the node feature vector corresponding to each node in the second sub-graph structure data needs to be sent to the graphics processor. In order to implement this step, it is necessary to determine the node identifier of each node in the second sub-graph structure data. Then, the node feature vector corresponding to the node identifier is read from the host memory, and the node feature vector is sent to the corresponding graphics processor.

[0123] In summary, the host memory enables the fast link device to obtain the second subgraph structure data through calculation, and conveniently obtain the feature vector corresponding to each node, and send it to the graphics processor for training.

[0124] In one embodiment, the plurality of graphics processors include a master graphics processor and a slave graphics processor;

[0125] The slave graphics processor is further configured to obtain gradient information of the first slave model parameters after training the graph neural network model according to the second subgraph structure data and the node feature vector, and send the gradient information of the first slave model parameters to the master graphics processor;

[0126] The main graphics processor is also used to obtain the gradient information of the first main model parameters after training the graph neural network model according to the second subgraph structure data and the node feature vector, receive the gradient information of the first slave model parameters sent by each slave graphics processor, and obtain optimized gradient information based on the gradient information of the first main model parameters and the gradient information of each first slave model parameter through aggregation and optimization processing.

[0127] In this embodiment, the multiple graphics processors include a master graphics processor and a slave graphics processor. Under this architecture, the master graphics processor and the slave graphics processors implement collaborative training and updating of model parameters to improve the training efficiency and accuracy of the model. First, after each slave graphics processor trains the graph neural network model based on the second subgraph structure data and the node feature vector, it obtains the first slave model parameters and the gradient information of the first slave model parameters, and sends the gradient information of these first slave model parameters to the master graphics processor. The master graphics processor receives the gradient information of the first slave model parameters sent by the slave graphics processor, and obtains the gradient information of the first master model parameters based on its own training results. After receiving the gradient information of the first slave model parameters sent by all slave graphics processors, the master graphics processor determines the gradient information of the first comprehensive model parameters based on the gradient information of the first master model parameters and the gradient information of each first slave model parameter, that is, optimizes the gradient information. Finally, the master graphics processor triggers itself and the slave graphics processors to update the model parameters of the graph neural network model based on the gradient information of the first comprehensive model parameters.

[0128] This architecture effectively leverages the computing power and storage resources of multiple GPUs, enabling centralized management of model parameters and collaborative training. The collaboration between the master and slave GPUs accelerates the training of graph neural network models and effectively processes large-scale graph-structured data, improving training efficiency and model accuracy.

[0129] In one embodiment, the optimized gradient information is obtained by aggregating and optimizing the gradient information of the first master model parameters and the gradient information of each first slave model parameter, including: averaging the gradient information of the first master model parameter and the gradient information of each first slave model parameter to obtain average gradient information, and the optimized gradient information includes the average gradient information.

[0130] This embodiment describes a method for updating model parameters between graphics processors. Specifically, after each slave graphics processor obtains the first slave model parameters after training, the gradient information of these first slave model parameters is sent to the master graphics processor. After the master graphics processor receives the gradient information of the first slave model parameters sent by all slave graphics processors, it determines the gradient information of the first comprehensive model parameters based on the gradient information of the first master model parameters and the gradient information of each first slave model parameter, and triggers itself and the slave graphics processors to update the model parameters of the graph neural network model based on the gradient information of the first comprehensive model parameters. In the process of determining the gradient information of the first comprehensive model parameters, a method of calculating the average gradient information can be used, that is, the average gradient information is calculated based on the gradient information of the first master model parameters and the gradient information of each first slave model parameter, and the average gradient information is determined as the gradient information of the first comprehensive model parameter. In this way, the model parameters on each graphics processor can be effectively integrated to achieve the effect of collaborative training. Through this method, the model parameter update and integration between multiple graphics processors can be achieved, thereby improving the training efficiency and performance of the graph neural network model.

[0131] This embodiment describes the process of how to trigger the master graphics processor and the slave graphics processor to update the model parameters of the graph neural network model based on the gradient information of the first comprehensive model parameters. Specifically, the master graphics processor first sends the calculated gradient information of the first comprehensive model parameters to all slave graphics processors to ensure that all processors have the latest gradient information of the comprehensive model parameters as the basis for updating the model parameters. After receiving the gradient information of the first comprehensive model parameters, the master graphics processor and each slave graphics processor begin to update the model parameters of the graph neural network model based on this optimized gradient information. The specific update algorithm and strategy are adjusted accordingly according to the weights, biases and other parameters of the neural network to make the model better adapt to the training data set and improve accuracy and generalization ability. Through this process, the master graphics processor and the slave graphics processor can work together to jointly update the parameters of the graph neural network model, thereby continuously optimizing the performance and effect of the model. This parallel parameter update method on a single server can speed up the model training and make full use of the computing resources of multiple processors to improve the overall training efficiency and effect.

[0132] In one embodiment, the master graphics processor and the slave graphics processor are determined in the following manner: the master graphics processor and the slave graphics processor are determined according to a user's configuration instruction, or, after each training is completed, a graphics processor is randomly selected to be the master graphics processor, and other graphics processors other than the master graphics processor are determined to be the slave graphics processors.

[0133] This embodiment describes how to determine the master and slave GPUs. Specifically, there are two methods. First, the master and slave GPUs are determined based on user configuration instructions: users can specify which GPU serves as the master and which as slaves through configuration instructions, allowing for flexible task allocation based on user needs and priorities. Second, a GPU is randomly selected as the master: After each training session, a GPU is randomly selected as the master, with the remaining GPUs serving as slaves. This allows for balanced task distribution and avoids over-reliance on a single processor.

[0134] Through the above two methods, the master GPU and slave GPU can be determined according to the user's configuration or random selection to achieve task allocation and training of graph neural network models.

[0135] In one embodiment, the processor is further configured to divide each first subgraph structure data into M third subgraph structure data, and store the M third subgraph structure data corresponding to each first subgraph structure data in a computing fast link device; when the computing fast link device obtains fourth subgraph structure data, send a node feature vector corresponding to each node in the fourth subgraph structure data to a graphics processor, where M is an integer greater than one;

[0136] a computing fast link device, specifically configured to store M third subgraph structure data corresponding to itself, and sequentially sample neighboring nodes for each node in each third subgraph structure data to obtain sampled fourth subgraph structure data, and send the fourth subgraph structure data to a graphics processor;

[0137] The graphics processor is specifically used to store the graph neural network model, and sequentially receive the fourth subgraph structure data sent by the computing fast link device and the node feature vector corresponding to each node in the fourth subgraph structure data sent by the processor, train the graph neural network model according to the fourth subgraph structure data and the node feature vector, and update the model parameters of the graph neural network model stored in itself after the training is completed, and obtain gradient information.

[0138] Specifically, to achieve more efficient and accurate graph neural network model training. First, the processor divides each first subgraph structure data into M third subgraph structure data, and stores the M third subgraph structure data corresponding to each first subgraph structure data in the computing fast link device. This can further subdivide the task and improve the parallelism and efficiency of training. Secondly, the computing fast link device can not only store the first subgraph structure data corresponding to itself, but also sample the neighboring nodes of each node in each third subgraph structure data in turn to obtain the sampled fourth subgraph structure data (the node sampling process is shown in Figure 2), and send the fourth subgraph structure data to the graphics processor, so that the graph structure data can be processed more flexibly and the accuracy and efficiency of training can be improved. Finally, the graphics processor not only receives the node feature vectors corresponding to each node in the fourth subgraph structure data sent by the computing fast link device and the fourth subgraph structure data sent by the processor in turn, and trains the graph neural network model according to the fourth subgraph structure data and the node feature vectors, but also updates the model parameters of the graph neural network model stored in itself after the training is completed, so that the training of the model can be more complete and accurate. In summary, this embodiment further refines and optimizes the subgraph structure data to achieve more efficient and accurate graph neural network model training.

[0139] In one embodiment, the plurality of graphics processors include a master graphics processor and a slave graphics processor;

[0140] The slave graphics processor is further configured to train the graph neural network model based on the calculation of the Mth fourth subgraph structure data sent by the fast link device and the node feature vector corresponding to each node in the Mth fourth subgraph structure data sent by the processor, and when obtaining the gradient information of the second slave model parameters, send the gradient information of the second slave model parameters to the master graphics processor;

[0141] The main graphics processor is also used to train the graph neural network model based on the node feature vector corresponding to each node in the Mth fourth subgraph structure data sent by the computing fast link device and the Mth fourth subgraph structure data sent by the processor to obtain the gradient information of the second main model parameters, receive the gradient information of the second slave model parameters sent by the slave graphics processor, and obtain optimized gradient information based on the gradient information of the second main model parameters and the gradient information of each second slave model parameter.

[0142] This embodiment mainly involves the collaboration between multiple graphics processors and how to train and update the parameters of the graph neural network model under this architecture. First, the slave graphics processor and the master graphics processor each assume different roles. The slave graphics processor is responsible for receiving and processing the Mth fourth subgraph structure data sent from the computing fast link device and the node feature vector corresponding to each node in the Mth fourth subgraph structure data sent by the processor, for training the graph neural network model; after the training is completed, the slave graphics processor will obtain the second slave model parameters and the gradient information of the second slave model parameters, and send the gradient information of these second slave model parameters to the master graphics processor. The master graphics processor receives the Mth fourth subgraph structure data and the node feature vector corresponding to each node sent from the computing fast link device and the processor, for training the graph neural network model; after the training is completed, the master graphics processor will obtain the second master model parameters and the gradient information of the second master model parameters; then, the master graphics processor will receive the gradient information of the second slave model parameters sent by the slave graphics processor, and summarize and optimize the gradient information of the second master model parameters and the gradient information of each second slave model parameter to obtain optimized gradient information. Finally, the master GPU triggers itself and the slave GPUs to update the model parameters of the graph neural network model based on the optimized gradient information. This layered parameter update method effectively integrates the training results of each GPU, thereby improving the performance and accuracy of the overall model.

[0143] In general, this embodiment realizes information exchange and parameter update across multiple graphics processors on the same server through the design of a master-slave structure, thereby improving the efficiency and accuracy of model training.

[0144] In one embodiment, the first subgraph structure data corresponding to the N first subgraph structure data divided from the graph structure data is obtained by the following steps:

[0145] Obtaining graph structure data input by a user, and determining the number of first nodes in the graph structure data;

[0146] The graph structure data is divided into N first sub-graph structure data according to the first number of nodes, and the difference in the number of nodes between every two first sub-graph structure data is not greater than a first preset number.

[0147] This embodiment describes a specific process of dividing the graph structure data input by the user into N first sub-graph structure data. Specifically, first, the system receives the graph structure data input by the user, and the graph structure data represents the nodes of a graph and the connection relationship between them. The received graph structure data is processed to determine the number of nodes therein, and the number of nodes in the graph structure data is the first number of nodes. According to the first number of nodes, the graph structure data is divided into N first sub-graph structure data, and each first sub-graph structure data contains a node corresponding to the sub-graph structure data and information about the edges related thereto. During the division process, in order to ensure load balancing, try to keep the difference in the number of nodes in any two first sub-graph structure data not exceeding the first preset number, so that computing resources can be fully utilized and communication overhead can be reduced while maintaining data balance.

[0148] Through the above steps, the graph structure data input by the user can be divided into N first sub-graph structure data, and the balance of the number of nodes can be maintained during the division process. This can provide suitable data distribution for subsequent graph neural network model training to improve training effects and performance.

[0149] In one embodiment, each first subgraph structure data is divided into M third subgraph structure data, including:

[0150] Obtain the number of second nodes and the number of edges of the target node included in the first subgraph structure data;

[0151] Dividing the first subgraph structure data into M third subgraph structure data according to the second number of nodes and the number of edges;

[0152] The difference in the number of target nodes between any two third subgraph structure data is not greater than the second preset number, and the difference in the number of edges is not greater than the third preset number.

[0153] This embodiment describes a process for partitioning first subgraph structure data, including obtaining the number of second nodes and edges of a target node and, based on this information, partitioning the first subgraph structure data into M third subgraph structure data. Specifically, for each first subgraph structure data, the number of second nodes and edges of the target node in the subgraph must be determined. This information can be used to describe the structural characteristics of the subgraph. Based on the obtained number of second nodes and edges, the first subgraph structure data is partitioned to obtain M third subgraph structure data. This partitioning process ensures that each third subgraph structure data contains a similar number of second nodes and edges to achieve balanced data distribution. This ensures that the image processor samples neighbor nodes at the same time for each batch of third subgraph structure data, or the time difference does not exceed a preset threshold. Specifically, the difference in the number of target nodes between each pair of third subgraph structure data does not exceed a second preset number, and the difference in the number of edges does not exceed a third preset number. This standard setting ensures that the partitioned subgraph data have similar structural characteristics, facilitating synchronization of subsequent processing and training processes.

[0154] In one embodiment, the first subgraph structure data is divided into M third subgraph structure data according to the second number of nodes and the number of edges, including:

[0155] Divide the target node into M groups evenly according to the number of second nodes, and obtain M groups of target child nodes;

[0156] Divide the edges into M groups evenly according to the number of target sub-nodes and edges in each group, and obtain M groups of edge data corresponding to the M groups of target sub-nodes;

[0157] The M groups of target sub-nodes and the edge data corresponding to themselves are determined as M third sub-graph structure data.

[0158] This embodiment describes the specific steps of dividing the first subgraph structure data into M third subgraph structure data. First, the target nodes are evenly divided into M groups according to the number of second nodes to obtain M groups of target sub-nodes, so as to ensure that the number of target sub-nodes in each group is similar. Secondly, the edges are evenly divided into M groups according to the number of target sub-nodes and edges in each group to obtain M groups of edge data corresponding one-to-one to the M groups of target sub-nodes, so as to ensure that each group of edge data corresponds one-to-one to the corresponding target sub-nodes. Finally, the M groups of target sub-nodes and the edge data corresponding to themselves are determined as M third subgraph structure data, thus completing the division and determination of the first subgraph structure data into M third subgraph structure data. Such a division and determination method can effectively divide complex graph structure data into multiple subgraph structure data, so that each image processor can process different subgraph structure data in parallel, thereby improving the training efficiency and performance of the entire graph neural network model.

[0159] In one embodiment, the processor is further configured to obtain a sample number of first neighbor nodes configured for each target node in the first subgraph structure data, divide the sample number of first neighbor nodes into M equal parts, obtain M groups of sample numbers of second neighbor nodes, and distribute the sample numbers of second neighbor nodes to the M groups of target subnodes in a one-to-one correspondence;

[0160] Neighbor node sampling is performed on each node in each third subgraph structure data to obtain sampled fourth subgraph structure data, including:

[0161] Neighbor sampling is performed on each target node according to the number of second neighbor node samples corresponding to itself to obtain the sampled fourth subgraph structure data.

[0162] In this embodiment, the processor is used not only to obtain the first subgraph structure data but also to obtain the number of first neighbor node samples configured for each target node and evenly divide these sampled numbers into M parts to obtain M groups of second neighbor node samples. This is done to more evenly sample neighbor nodes for each target node, ensuring that the fourth subgraph structure data obtained by the fast link computing device is more balanced. Next, the fast link computing device performs neighbor node sampling for each node in each third subgraph structure data to obtain sampled fourth subgraph structure data. This step involves sampling neighbors for each target node based on the number of second neighbor node samples obtained from the processor, ensuring that the sampled fourth subgraph structure data contains more information about associated nodes, facilitating subsequent training by the graphics processor.

[0163] As shown in Figure 3, if a target node needs to be sampled in k orders, k is an integer greater than 1. After the number of neighbor nodes for each order of sampling is set, it is divided by M as the number of sampled neighbor nodes in the third subgraph structure data.

[0164] In general, by evenly distributing the number of first neighbor node samples to each group of target sub-nodes, it can be ensured that when sampling neighbor nodes, each group of target sub-nodes can obtain a similar number of neighbor node samples, thereby ensuring the balance of data samples and the effectiveness of the training model.

[0165] In one embodiment, the processor is also used to determine the edge nodes in each first subgraph structure data and the edge neighbor nodes in other first subgraph structures connected to the edge nodes after dividing the graph structure data into N first subgraph structure data, and store the edge neighbor nodes and the connection relationship between the edge neighbor nodes and the edge nodes in the computing fast link device corresponding to the first subgraph structure where the edge node is located.

[0166] During the training process, graph structure data needs to be processed and communicated in parallel. Since the graph structure data is divided and distributed to each CXL (Compute Express Link) device, the edge node of each CXL device may need to access the data corresponding to the nodes connected to the edge node of this CXL device under other CXL devices, etc. This requires cross-device access to the data corresponding to edge neighbor nodes, which will cause communication delay problems.

[0167] In this embodiment, by determining the edge nodes in each subgraph structure data and the edge neighbor nodes in other subgraph structures connected to the edge nodes in the processor, and storing this information in the corresponding computing fast link device, it is possible to effectively avoid cross-device access to the data corresponding to the edge neighbor nodes, thereby reducing communication latency. Specifically, by storing the edge neighbor nodes and the connection relationships between the edge neighbor nodes and the edge nodes in the corresponding computing fast link device, when calculations are required, the required data corresponding to the edge neighbor nodes can be directly obtained in the local computing fast link device without the need for cross-device data communication, thereby greatly reducing communication latency and improving training efficiency and performance.

[0168] For example, in Figure 4, the feature vectors of each node are identified as V_1, V_2, V_3, V_4, V_5...V_n. After the graph structure data is divided and allocated to each processor, a schematic diagram of each non-uniform memory access domain and its assigned first subgraph structure data is obtained. The dots (including black dots and white dots) in each non-uniform memory access domain and their connection relationships represent the first subgraph structure data in the domain. For example, each black dot in non-uniform memory access domain 0 represents an edge node in the domain. The black dots in other non-uniform memory access domains 1, 2, and 3 corresponding to the edge node and the nodes connected to the edge node are neighbor nodes of the edge node, and the data corresponding to this neighbor node is stored in non-uniform memory access domain 0.

[0169] In one embodiment, a computing fast link device includes a computing unit, a private second device memory, and a processor-managed first device memory, and a one-to-one mapping between a portion of a physical address of the processor and a physical address of the first device memory;

[0170] Allocating corresponding first subgraph structure data among the N first subgraph structure data divided from the graph structure data to corresponding connected computing fast link devices includes:

[0171] Allocating corresponding first subgraph structure data among the N first subgraph structure data to a second device memory of a corresponding connected computing fast link device;

[0172] A second device memory of the computing fast link device is used to store first subgraph structure data corresponding to the computing fast link device;

[0173] a computing unit, configured to sample neighbor nodes of nodes in the first subgraph structure data according to the first subgraph structure data in the memory of the second device to obtain sampled second subgraph structure data, send the second subgraph structure data to the graphics processor, and write node identifiers in the second subgraph structure data into the memory of the first device;

[0174] The first device memory is used to store node identifiers in the second subgraph structure data;

[0175] When the corresponding connected computing fast link device obtains the second subgraph structure data, sending the node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor includes:

[0176] The node identifiers in the memory of the first device are obtained, and node feature vectors corresponding to the node identifiers are determined according to the node identifiers, and the node feature vectors are sent to the graphics processor.

[0177] In this embodiment, the computational fast link device includes a computing unit, a processor-managed first device memory, and a private second device memory. In this design, the first device memory and the second device memory are used separately. The first device memory is primarily used to store node identifiers in the second subgraph structure data, which enables more organized and efficient data storage. Node identifiers are frequently read and updated during neighbor node sampling. Storing them separately in the first device memory reduces data access conflicts and improves data read speed. The second device memory is used to store and process the first subgraph structure data, ensuring that the computing unit can quickly access and process the required data, reducing data read latency and improving overall computational efficiency. Allocating N pieces of first subgraph structure data to the second device memories of N computational fast link devices in a one-to-one correspondence enables parallel computation across multiple computational fast link devices, allowing each computational fast link device to independently process its own data, improving overall system computational efficiency. When the computing unit processes the first subgraph structure data in the second device memory, it can directly send the results to the graphics processor, reducing data transmission complexity and communication latency, and improving the real-time and efficiency of data processing. Because the first and second device memories manage different types of data separately, data consistency and reliability can be better controlled. Separate storage of node identifiers and node feature vectors reduces the possibility of data confusion or garbled data, ensuring data accuracy and integrity. Separate management of the primary and secondary device memories makes the system more flexible and controllable. When it's necessary to expand computing fast link devices or increase node processing power, the primary and secondary device memories can be expanded or optimized relatively independently without significantly impacting the overall system.

[0178] In summary, dividing the first device memory and the second device memory brings many advantages to the training architecture of the graph neural network model, including data management optimization, parallel computing efficiency, data consistency and reliability, as well as improvements in flexibility and scalability, which helps to improve the performance and efficiency of the overall system.

[0179] In one embodiment, the processor is further configured to, when initializing the computing fast link device, divide the device memory of the computing fast link device into a first device memory and a second device memory according to the device memory capacity of the computing fast link device and a preset ratio, and map the physical address in the first device memory to a part of its own physical address one by one.

[0180] In this embodiment, first, the device memory of the computing fast link device is divided into a first device memory and a second device memory based on the device memory capacity and a preset ratio of the computing fast link device. This division process is intended to effectively manage and utilize the memory resources of the computing fast link device, ensuring that the computing fast link device can efficiently store and process graph structure data. Secondly, the processor maps the physical addresses in the first device memory to part of its own physical addresses one by one. This mapping process is intended to ensure that the processor can accurately locate and access the graph structure data stored in the first device memory of the computing fast link device, thereby allocating the graph structure data to the correct computing fast link device for processing and training.

[0181] In one embodiment, the preset ratio is determined according to the type of graph structure data, the data size of the first subgraph structure data, and the data size of the node identifier corresponding to the first subgraph structure data.

[0182] In this embodiment, when determining the preset ratio, the type of graph structure data needs to be taken into account, and different types of graph structure data may require different memory allocation ratios. In addition, the data volume of the first subgraph structure data also needs to be taken into account, because when the data volume is large, more memory space may be required to store it. Finally, the data volume of the node identifier corresponding to the first subgraph structure data also needs to be considered, because the data volume of the node identifier will also affect the memory allocation ratio. Adjustments are made according to actual conditions to ensure that the device memory of the computing fast link device can reasonably store and process the graph structure data, thereby ensuring the efficiency and accuracy of the graph neural network model training.

[0183] In one embodiment, the processor is further configured to divide each first subgraph structure data into M third subgraph structure data, and store each node identifier in the M third subgraph structure data corresponding to each first subgraph structure data in a first device memory of a fast link computing device; when the fast link computing device obtains fourth subgraph structure data, obtain a node identifier of each node in the fourth subgraph structure data stored in the first device memory, obtain a node feature vector of each node in the fourth subgraph structure data based on the node identifier of each node in the fourth subgraph structure data, and send the node feature vector to a graphics processor, where M is an integer greater than one;

[0184] The first device memory of the computing fast link device is specifically used to store each node identifier in the M third subgraph structure data corresponding to the first device memory;

[0185] a computing unit of the fast link computing device, specifically configured to sequentially sample neighbor nodes for each node in each of the third subgraph structure data according to each node identifier in the third subgraph structure data, obtain sampled fourth subgraph structure data, and write the node identifier of each node in the fourth subgraph structure data into the memory of the first device;

[0186] The graphics processor is specifically used to store the graph neural network model, and sequentially receive the fourth subgraph structure data sent by the computing fast link device and the node feature vector corresponding to each node in the fourth subgraph structure data sent by the processor, train the graph neural network model according to the fourth subgraph structure data and the node feature vector, and update the model parameters of the graph neural network model stored in itself after the training is completed.

[0187] This embodiment provides a further partitioning method for first subgraph structure data, namely, partitioning each first subgraph structure data into M third subgraph structure data, and storing the node identifiers in the M third subgraph structure data corresponding to each first subgraph structure data in the first device memory of a fast link computing device. This partitioning method allows for a more detailed description of node relationships in a graph neural network, while also enabling more efficient utilization of computing resources during model training, thereby improving training speed. In one embodiment, the first device memory of the fast link computing device is specifically used to store the node identifiers in the M third subgraph structure data corresponding to itself. A computing unit of the fast link computing device sequentially samples neighboring nodes for each node in each third subgraph structure data based on the node identifiers in the third subgraph structure data, obtaining sampled fourth subgraph structure data, and writing the node identifiers of each node in the fourth subgraph structure data into the first device memory. During this process, the computing unit first determines the nodes requiring neighbor node sampling based on the node identifiers in the third subgraph structure data, then reads the identifiers of the corresponding nodes from the first device memory, reads the neighboring nodes of the corresponding nodes from the second device memory, performs sampling, and transmits the resulting fourth subgraph structure data to a graphics processor. After receiving the fourth subgraph structure data and the node feature vectors corresponding to each node in the fourth subgraph structure data sent by the processor, the graphics processor trains the graph neural network model based on the fourth subgraph structure data and the node feature vectors, and updates the model parameters of the graph neural network model stored in its own storage after training is completed. In this way, efficient training and optimization of the graph neural network model can be achieved.

[0188] Based on the above embodiment, a specific process is shown in FIG5 . The stages of a non-uniform memory access domain are generally divided into three stages, namely, a preprocessing stage, a data loading stage, and a model training stage. In the preprocessing stage, the processor reads the input graph structure data and stores the graph structure data in the host memory. The processor divides the graph structure data to obtain N first subgraph structure data and stores them in the second device memory. The processor further divides the first subgraph structure data and stores the node data of the divided third subgraph structure data in the first device memory. In the data loading stage, the computing unit in the computing fast link device samples neighbor nodes of the third subgraph structure data based on the data in the first device memory and the second device memory (obtaining the sampled fourth subgraph structure data, not shown in the figure). The processor performs a node feature vector query based on the division of the first subgraph node data and the pre-stored node feature vector of the graph structure data to query the node feature vector (not shown in the figure) included in the sampled fourth subgraph structure data. The sampled fourth subgraph structure data and the corresponding node feature vector are sent to the graphics processor. During the model training phase, the graphics processor trains the graph neural network model based on the sampled fourth subgraph structure data and the corresponding node feature vectors, performs forward calculations, backpropagation to calculate gradients, and updates model parameters; then enters the processing of the next third subgraph structure data. Figure 6 further explains the process of graph structure data processing. Specifically, the node feature vectors of the graph structure data are stored in the host memory. The processor communicates with the computing fast link device and the graphics processor via PCIe (Peripheral Component Interconnect Express, a high-speed serial communication interconnection standard). The computing resources of the computing fast link device parse the memory access request of the computing fast link device, obtain the data in the first device memory and the second device memory, and extract the subgraph structure and sample the neighboring nodes. The node feature vectors in the host memory are queried through the memory controller, and the sampled fourth subgraph structure data (not shown in the figure) and the queried node feature vectors are transmitted to the graphics processor. The graphics processor obtains the vectorized representation of the new node through forward propagation, and then updates the model parameters through loss function calculation and backward propagation.

[0189] In one embodiment, the computing fast link device further includes a memory buffer coupled to the computing unit;

[0190] The processor is further configured to, after dividing the graph structure data into N first subgraph structure data, sort the N first subgraph structure data according to the size of the degree distribution;

[0191] The memory buffer is used to store the data of nodes and edges in the first subgraph structure data with the largest degree distribution, so as to provide the data to the computing unit.

[0192] In this embodiment, the computing fast link device also includes a memory buffer connected to the computing unit. The function of this memory buffer is to store the data of the nodes and edges in the first subgraph structure data with the largest degree distribution. The purpose of doing this is to provide it to the computing unit so that the computing unit can sample the neighboring nodes of the nodes in the first subgraph structure data more quickly, obtain the sampled second subgraph structure data, and send the second subgraph structure data to the graphics processor, while writing the node identifier in the second subgraph structure data into the first device memory. Such a design can improve the efficiency of the computing fast link device, because the memory buffer can cache the first subgraph structure data with the largest degree distribution, reducing the frequent access of the computing unit to the memory, thereby speeding up the calculation. At the same time, the memory buffer can also improve the parallel processing capability of the entire system, because the computing unit can access the memory buffer independently without affecting the calculation of other parts.

[0193] In one embodiment, the memory buffer is specifically used to store data of some nodes and some edges in the first subgraph structure data with the largest degree distribution when the data of nodes and edges in the first subgraph structure data with the largest degree distribution is larger than its own memory capacity.

[0194] In this embodiment, when the size of the first subgraph structure data with the largest degree distribution exceeds the capacity of the memory buffer, the memory buffer uses a strategy to select the node and edge data to be stored. For example, according to a certain rule, the data of the node with the highest degree and the related edges are selected, or the data related to other important nodes is selected. In this way, the memory buffer can store the most important node and edge data within a limited capacity to provide it to the computing unit for use. This ensures that the computing unit can quickly access and process the first subgraph structure data with the largest degree distribution to perform neighbor node sampling and generate the second subgraph structure data.

[0195] In a second aspect, as shown in FIG7 , the present application further provides a training method for a graph neural network model, which is applied to a computing fast link device. The training method for the graph neural network model includes:

[0196] S11: Obtaining first sub-graph structure data corresponding to the first sub-graph structure data after the graph structure data is divided and sent by the processor;

[0197] S12: Sampling neighbor nodes of the nodes in the first subgraph structure data to obtain sampled second subgraph structure data;

[0198] S13: Sending the second subgraph structure data to the graphics processor, and triggering the processor to send the node feature vector corresponding to each node in the second subgraph structure data to the graphics processor;

[0199] S14: Trigger the graphics processor to train the graph neural network model based on the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data.

[0200] For an introduction to the training method of the graph neural network model, please refer to the above embodiment, and this application will not go into details here.

[0201] In a third aspect, as shown in FIG8 , the present application further provides a training system for a graph neural network model, which is applied to a computing fast link device. The training system for the graph neural network model includes:

[0202] An acquiring unit 81 is configured to acquire first sub-graph structure data corresponding to itself after the graph structure data is divided and sent by the processor;

[0203] A sampling unit 82 is configured to sample neighboring nodes of nodes in the first subgraph structure data to obtain sampled second subgraph structure data;

[0204] a sending unit 83 configured to send the second subgraph structure data to the graphics processor, and trigger the processor to send the node feature vector corresponding to each node in the second subgraph structure data to the graphics processor;

[0205] The training unit 84 is used to trigger the graphics processor to train the graph neural network model based on the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data.

[0206] For an introduction to the training system of the graph neural network model, please refer to the above embodiment, and this application will not go into details here.

[0207] Fourthly, the present application also provides a server, including a training architecture for the graph neural network model as described above.

[0208] For an introduction to the server, please refer to the above embodiment, and this application will not go into details here.

[0209] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0210] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training architecture for a graph neural network model, characterized in that: Applicable to servers, including: N processors, where N is an integer greater than one; One or more graphics processors connected to each processor in a one-to-one correspondence, and each graphics processor is connected to each other; Each processor is configured to send a node feature vector corresponding to each node in each second subgraph structure data of the graph structure data input by the user to a correspondingly connected graphics processor; Each graphics processor is configured to store a graph neural network model, and upon receiving the second subgraph structure data and a node feature vector corresponding to each node in the second subgraph structure data, train the graph neural network model according to the second subgraph structure data and the node feature vector to obtain gradient information; a master graphics processor, configured to receive gradient information sent by each graphics processor, aggregate and optimize the received gradient information to obtain optimized gradient information, and send the optimized gradient information to each graphics processor, wherein the master graphics processor is any graphics processor among the multiple graphics processors connected to each processor, or a graphics processor not connected to each processor; Each graphics processor is also used to update the model parameters of the stored graph neural network model based on the received optimization gradient information.

2. The training architecture of the graph neural network model according to claim 1, characterized in that: The training architecture also includes: N computing fast link devices are connected to N processors in a one-to-one correspondence; Each processor is further configured to distribute corresponding first subgraph structure data from the N first subgraph structure data divided from the graph structure data to a corresponding connected fast computing link device, and when the corresponding connected fast computing link device obtains second subgraph structure data, send a node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor; The computing fast link device is used to store the first subgraph structure data corresponding to itself, sample neighbor nodes of the nodes in the first subgraph structure data corresponding to itself, obtain the sampled second subgraph structure data, and send the second subgraph structure data to the corresponding connected graphics processor.

3. The training architecture of the graph neural network model according to claim 2, characterized in that: Also includes: N host memories, connected to the N processors in a one-to-one correspondence, for storing node feature vectors corresponding to each node in the graph structure data; The processor is further configured to store a node feature vector corresponding to each node in the graph structure data into the host memory; When the corresponding connected computing fast link device obtains the second subgraph structure data, sending the node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor includes: When the corresponding connected computing fast link device obtains the second subgraph structure data, determining the node identifier of each node in the second subgraph structure data; and A node feature vector corresponding to the node identifier is read from the host memory, and the node feature vector is sent to a correspondingly connected graphics processor.

4. The training architecture of the graph neural network model according to claim 1, wherein: The plurality of graphics processors connected to each processor include the master graphics processor and the slave graphics processor; The slave graphics processor is further configured to obtain gradient information of a first slave model parameter after training the graph neural network model according to the second subgraph structure data and the node feature vector, and send the gradient information of the first slave model parameter to the master graphics processor; The main graphics processor is further configured to process the graph neural network according to the second subgraph structure data and the node feature vector. After training the network model, gradient information of the first master model parameters is obtained, the gradient information of the first slave model parameters sent by each of the slave graphics processors is received, and the gradient information of the first master model parameters and the gradient information of each of the first slave model parameters are aggregated and optimized to obtain optimized gradient information.

5. The training architecture of the graph neural network model according to claim 4, wherein: The step of aggregating and optimizing the gradient information of the first master model parameters and the gradient information of each of the first slave model parameters to obtain optimized gradient information includes: Average processing is performed on the gradient information of the first master model parameter and the gradient information of each of the first slave model parameters to obtain average gradient information, and the optimized gradient information includes the average gradient information.

6. The training architecture of the graph neural network model according to claim 4, characterized in that: The master graphics processor and the slave graphics processor are determined by: determining the master graphics processor and the slave graphics processor according to a user's configuration instruction, or, after each training is completed, randomly selecting a graphics processor to be determined as the master graphics processor, and determining another graphics processor other than the master graphics processor to be determined as the slave graphics processor.

7. The training architecture of the graph neural network model according to claim 2, wherein: The processor is further configured to divide each first subgraph structure data into M third subgraph structure data, and store the M third subgraph structure data corresponding to each first subgraph structure data in a computing fast link device, and when the computing fast link device obtains fourth subgraph structure data, send a node feature vector corresponding to each node in the fourth subgraph structure data to the graphics processor, where M is an integer greater than one; The computing fast link device is specifically configured to store M third subgraph structure data corresponding to itself, and sequentially sample neighboring nodes for each node in each of the third subgraph structure data to obtain sampled fourth subgraph structure data, and send the fourth subgraph structure data to the graphics processor; The graphics processor is specifically used to store the graph neural network model, and sequentially receive the fourth subgraph structure data sent by the computing fast link device and the node feature vector corresponding to each node in the fourth subgraph structure data sent by the processor, and train the graph neural network model according to the fourth subgraph structure data and the node feature vector to obtain gradient information.

8. The training architecture of the graph neural network model according to claim 7, wherein: The plurality of graphics processors connected to each processor include the master graphics processor and the slave graphics processor; The slave graphics processor is further configured to train a graph neural network model based on the Mth fourth subgraph structure data sent by the computing fast link device and the node feature vector corresponding to each node in the Mth fourth subgraph structure data sent by the processor, obtain gradient information of a second slave model parameter, and send the gradient information of the second slave model parameter to the master graphics processor; The master graphics processor is further used to train the graph neural network model based on the node feature vectors corresponding to each node in the Mth fourth subgraph structure data sent by the computing fast link device and the Mth fourth subgraph structure data sent by the processor to obtain the gradient information of the second master model parameters, receive the gradient information of the second slave model parameters sent by the slave graphics processor, and obtain optimized gradient information based on the gradient information of the second master model parameters and the gradient information of each second slave model parameter by aggregating and optimizing them.

9. The training architecture of the graph neural network model according to claim 2, wherein: The first subgraph structure data corresponding to the N first subgraph structure data divided from the graph structure data is obtained by the following steps: Obtaining graph structure data input by a user, and determining the number of first nodes in the graph structure data; and The graph structure data is divided into N first sub-graph structure data according to the first number of nodes, and the difference in the number of nodes between any two first sub-graph structure data is no greater than a first preset number.

10. The training architecture of the graph neural network model according to claim 7, wherein: Dividing each of the first subgraph structure data into M third subgraph structure data includes: Obtaining the second node number and edge number of the target node included in the first subgraph structure data; and Dividing the first subgraph structure data into M pieces of the third subgraph structure data according to the number of the second nodes and the number of the edges; The difference in the number of target nodes between any two of the third subgraph structure data is not greater than the second preset number, and the difference in the number of edges is not greater than the third preset number.

11. The training architecture of the graph neural network model according to claim 10, wherein: Dividing the first subgraph structure data into M pieces of the third subgraph structure data according to the number of the second nodes and the number of the edges, including: Dividing the target node into M groups evenly according to the number of the second nodes to obtain M groups of target child nodes; Dividing the edges into M groups evenly according to the number of target subnodes and edges in each group, and obtaining M groups of edge data corresponding one-to-one to the M groups of target subnodes; and The M groups of the target sub-nodes and the edge data corresponding to themselves are determined as the M third sub-graph structure data.

12. The training architecture of the graph neural network model according to claim 11, wherein: The processor is further configured to obtain a sample number of first neighbor nodes configured for each target node in the first subgraph structure data, divide the sample number of the first neighbor nodes into M equal parts to obtain M groups of sample numbers of second neighbor nodes, and distribute the sample numbers of the second neighbor nodes to the M groups of target subnodes in a one-to-one correspondence; Sampling neighbor nodes for each node in each of the third subgraph structure data to obtain sampled fourth subgraph structure data includes: Neighbor sampling is performed on each target node according to the number of second neighbor node samples corresponding to itself to obtain the sampled fourth subgraph structure data.

13. The training architecture of the graph neural network model according to claim 2, wherein: The processor is also used to determine the edge nodes in each of the first subgraph structure data and the edge neighbor nodes in other first subgraph structures connected to the edge nodes, and store the edge neighbor nodes and the connection relationship between the edge neighbor nodes and the edge nodes in the computing fast link device corresponding to the first subgraph structure where the edge nodes are located.

14. The training architecture of the graph neural network model according to any one of claims 2 to 13, wherein: The computing fast link device includes a computing unit, a private second device memory, and a processor-managed first device memory, wherein a partial physical address of the processor is mapped one-to-one with a physical address of the first device memory; Allocating corresponding first subgraph structure data among the N first subgraph structure data divided from the graph structure data to corresponding connected computing fast link devices, including: Allocating corresponding first subgraph structure data among the N first subgraph structure data to a second device memory of a corresponding connected computing fast link device; The second device memory of the computing fast link device is used to store the first subgraph structure data corresponding to itself; the computing unit is configured to sample neighboring nodes of nodes in the first subgraph structure data according to the first subgraph structure data in the memory of the second device to obtain sampled second subgraph structure data, send the second subgraph structure data to the graphics processor, and write node identifiers in the second subgraph structure data into the memory of the first device; The first device memory is used to store node identifiers in the second subgraph structure data; When the corresponding connected computing fast link device obtains the second subgraph structure data, sending the node feature vector corresponding to each node in the second subgraph structure data to the corresponding connected graphics processor includes: Obtain the node identifiers in the memory of the first device, determine the node feature vectors corresponding to the node identifiers according to the node identifiers, and send the node feature vectors to the graphics processor.

15. The training architecture of the graph neural network model according to claim 14, wherein: The processor is further configured to, when initializing the computing fast link device, divide the device memory of the computing fast link device into a first device memory and a second device memory according to the device memory capacity of the computing fast link device and a preset ratio, and map physical addresses in the first device memory to part of its own physical addresses one by one.

16. The training architecture of the graph neural network model according to claim 15, wherein: The preset ratio is determined according to the type of the graph structure data, the data size of the first subgraph structure data, and the data size of the node identifier corresponding to the first subgraph structure data.

17. The training architecture of the graph neural network model according to claim 14, wherein: The processor is further configured to divide each first subgraph structure data into M third subgraph structure data, and store each node identifier in the M third subgraph structure data corresponding to each first subgraph structure data in a first device memory of a fast link computing device; When the computing fast link device obtains the fourth subgraph structure data, obtaining a node identifier of each node in the fourth subgraph structure data stored in the memory of the first device, obtaining a node feature vector of each node in the fourth subgraph structure data according to the node identifier of each node in the fourth subgraph structure data, and sending the node feature vector to the graphics processor, where M is an integer greater than one; The first device memory of the fast link computing device is specifically used to store each node identifier in the M third subgraph structure data corresponding to itself; The computing unit of the fast link computing device is specifically configured to sequentially sample neighbor nodes for each node in the third subgraph structure data according to each node identifier in the third subgraph structure data to obtain sampled fourth subgraph structure data, and write the node identifier of each node in the fourth subgraph structure data into the first device memory; The graphics processor is specifically used to store the graph neural network model, and sequentially receive the fourth subgraph structure data sent by the computing fast link device and the node feature vector corresponding to each node in the fourth subgraph structure data sent by the processor, train the graph neural network model according to the fourth subgraph structure data and the node feature vector, and update the model parameters of the graph neural network model stored in itself after the training is completed.

18. The training architecture of the graph neural network model according to claim 14, wherein: The computing fast link device further includes a memory buffer connected to the computing unit; The processor is further configured to, after dividing the graph structure data into N first sub-graph structure data, sort the N first sub-graph structure data according to the magnitude of the degree distribution; The memory buffer is used to store the data of nodes and edges in the first subgraph structure data with the largest degree distribution, so as to provide the data to the computing unit.

19. The training architecture of the graph neural network model according to claim 18, wherein: The memory buffer is specifically used to store data of some nodes and some edges in the first subgraph structure data with the largest degree distribution when the data of nodes and edges in the first subgraph structure data with the largest degree distribution is larger than its own memory capacity.

20. A training method for a graph neural network model, characterized in that: Applied to computing fast link equipment, the training method of the graph neural network model includes: Obtaining the first subgraph structure data corresponding to itself after the graph structure data is divided and sent by the processor; Sampling neighbor nodes of nodes in the first subgraph structure data to obtain sampled second subgraph structure data; Sending the second subgraph structure data to a graphics processor, and triggering the processor to send a node feature vector corresponding to each node in the second subgraph structure data to the graphics processor; and The graphics processor is triggered to train the graph neural network model based on the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data.

21. A training system for a graph neural network model, characterized in that: Applied to computing fast link equipment, the training system of the graph neural network model includes: an acquiring unit, configured to acquire first subgraph structure data corresponding to itself after the graph structure data is divided and sent by the processor; a sampling unit, configured to sample neighboring nodes of nodes in the first subgraph structure data to obtain sampled second subgraph structure data; a sending unit, configured to send the second subgraph structure data to a graphics processor, and trigger the processor to send a node feature vector corresponding to each node in the second subgraph structure data to the graphics processor; and A training unit is used to trigger the graphics processor to train the graph neural network model based on the second subgraph structure data and the node feature vector corresponding to each node in the second subgraph structure data.

22. A server, characterized in that: A training architecture comprising a graph neural network model as described in any one of claims 1-19.

Images