Graph convolutional neural network training and graph computing method, device, equipment and system

Abstract

The embodiments of the present application provide a graph convolutional neural network training and graph operation method, apparatus, device and system, in which a full graph is cut to obtain multiple subgraphs, and for each subgraph, the degree of each node in the subgraph in the full graph is obtained, and each subgraph is distributed to each training device for training, and the degree of each node in the subgraph is issued at the same time as the subgraph is issued. When each training device uses a local graph convolutional neural network to perform feedforward calculations on the subgraph, since the calculation is based on the degree of the full graph, it can ensure that the obtained node output features are consistent with the node output features of the full graph. In the process of training the graph convolutional neural network, it is equivalent to training the graph convolutional neural network based on the full graph, and since the amount of data of the subgraph is much smaller than the amount of data of the full graph, the distributed training method of multiple training devices is used to achieve the purpose of training a graph-to-graph convolutional neural network model based on a large order of magnitude.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a graph convolutional neural network training and graph operation method, device, equipment and system. Background Art

[0002] In the field of machine learning technology, in order to enable machines to understand knowledge in certain professional fields and irregular data structures between entities, graphs in graph theory are often used to describe related knowledge and / or entities, such as social networks, knowledge graphs, chemical molecular structures, etc. The graph includes multiple nodes, and the nodes represent knowledge and / or entities. The lines between nodes are called edges, which are used to represent the relationship between two nodes.

[0003] The implementation of various GNN (Graph Neural Network) algorithms in the industry, represented by GCN (Graph Convolution Network), GAT (Graph Attention Network), and GraphSage (Graph Sampling and Aggregation Algorithm), has promoted the further development of artificial intelligence and greatly broadened the application scope of deep learning in graph theory.

[0004] In today's big data era, with the explosive growth of knowledge and entity scale, the number of nodes and edges in the graph is very large. The current GCN model training is based on full-graph training, that is, a single training device is used to train the GCN model based on the full graph. However, the computing resources of the current training equipment are limited. A single training device is prone to overflow during the GCN model training process, resulting in a single training device being unable to meet the requirements for training the GNN model based on large-scale graphs. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a graph convolutional neural network training and graph operation method, device, equipment and system to realize the training of graph convolutional neural network models based on large-scale graphs.

[0006] The specific technical solutions are as follows:

[0007] In a first aspect, an embodiment of the present application provides a graph convolutional neural network training method, the method comprising:

[0008] Obtain a test data set and the number of training devices, where the test data set contains a full graph, the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, indicating that the two nodes are associated;

[0009] Based on the preset graph cutting algorithm and the number of training devices, the full graph is cut to obtain multiple sub-graphs;

[0010] For each subgraph, obtain the degree of each node in the subgraph in the whole graph, where the degree represents the number of edges of the node;

[0011] Sending each subgraph and the degree of each node in the subgraph to each training device respectively, so that each training device uses a local graph convolutional neural network based on the degree of each node in the input subgraph to perform feedforward calculation on the subgraph, obtain the output features of the nodes in the subgraph, and calculate the gradient according to the output features of the nodes and the label information of the nodes;

[0012] Collect the gradients obtained by each training device, and calculate the integrated gradient based on the gradients obtained by each training device;

[0013] Distribute the integrated gradient to each training device so that each training device updates the model parameters of the local graph convolutional neural network model according to the integrated gradient;

[0014] Return to the step of sending each subgraph and the degree of each node in the subgraph to each training device respectively, until the training is terminated when the training termination condition is met.

[0015] In a second aspect, an embodiment of the present application provides a graph computing method, the method comprising:

[0016] Obtain the full graph to be calculated and the number of computing devices, where the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, indicating that the two nodes are associated;

[0017] Based on the preset graph cutting algorithm and the number of computing devices, the full graph is cut to obtain multiple sub-graphs;

[0018] For each subgraph, obtain the degree of each node in the subgraph in the whole graph, where the degree represents the number of edges of the node;

[0019] Each subgraph and the degree of each node in the subgraph are sent to each computing device respectively, so that each computing device uses a local graph convolutional neural network to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph to obtain the output features of the nodes in the subgraph.

[0020] In a third aspect, an embodiment of the present application provides a graph convolutional neural network training device, the device comprising:

[0021] The first acquisition module is used to acquire a test data set and the number of training devices, wherein the test data set includes a full graph, the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, indicating that the two nodes are associated;

[0022] A first subgraph cutting module is used to cut the full graph based on a preset graph cutting algorithm and the number of training devices to obtain multiple subgraphs;

[0023] A first determination module is used to obtain, for each subgraph, the degree of each node in the subgraph in the full graph, wherein the degree represents the number of edges of the node;

[0024] The first sending module is used to send each subgraph and the degree of each node in the subgraph to each training device, so that each training device uses a local graph convolutional neural network based on the degree of each node in the input subgraph to perform feedforward calculation on the subgraph, obtain the output features of the nodes in the subgraph, and calculate the gradient according to the output features of the nodes and the label information of the nodes;

[0025] A gradient integration module is used to collect the gradients obtained by each training device and calculate the integrated gradient according to the gradients obtained by each training device;

[0026] The first sending module is also used to distribute the integrated gradient to each training device so that each training device updates the model parameters of the local graph convolutional neural network model according to the integrated gradient.

[0027] In a fourth aspect, an embodiment of the present application provides a graph computing device, the device comprising:

[0028] The second acquisition module is used to acquire the full graph to be calculated and the number of computing devices, wherein the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, indicating that the two nodes are associated;

[0029] The second subgraph cutting module is used to cut the full graph based on a preset graph cutting algorithm and the number of computing devices to obtain multiple subgraphs;

[0030] A second determination module is used to obtain, for each subgraph, the degree of each node in the subgraph in the whole graph, wherein the degree represents the number of edges of the node;

[0031] The second sending module is used to send each subgraph and the degree of each node in the subgraph to each computing device, so that each computing device can perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph and use the local graph convolutional neural network to obtain the output features of the nodes in the subgraph.

[0032] In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0033] Memory, used to store computer programs;

[0034] The processor is used to implement the method provided in the first aspect or the second aspect of the embodiment of the present application when executing the computer program stored in the memory.

[0035] In a sixth aspect, an embodiment of the present application provides a machine-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect or the second aspect of the embodiment of the present application is implemented.

[0036] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method as provided in the first aspect or the second aspect of the embodiment of the present application.

[0037] In an eighth aspect, an embodiment of the present application provides a distributed graph convolutional neural network training system, the system comprising a management device and a plurality of training devices;

[0038] A management device is used to obtain a test data set and the number of training devices, wherein the test data set includes a full graph, the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, which represents the association between the two nodes; based on a preset graph cutting algorithm and the number of training devices, the full graph is cut to obtain multiple subgraphs; for each subgraph, the degree of each node in the subgraph in the full graph is obtained, wherein the degree represents the number of edges of the node; each subgraph and the degree of each node in the subgraph obtained are sent to each training device respectively;

[0039] Each training device is used to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph using a local graph convolutional neural network to obtain output features of the nodes in the subgraph, and calculate the gradient according to the output features of the nodes and the label information of the nodes; and feed the calculated gradient back to the management device;

[0040] The management device is also used to collect the gradients obtained by each training device, and calculate the integrated gradient according to the gradients obtained by each training device; and distribute the integrated gradient to each training device;

[0041] Each training device is also used to update the model parameters of the local graph convolutional neural network model based on the integrated gradient.

[0042] In a ninth aspect, an embodiment of the present application provides a distributed graph computing system, the system comprising a management device and a plurality of computing devices;

[0043] A management device is used to obtain a full graph to be calculated and the number of computing devices, wherein the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, which represents the association between the two nodes; based on a preset graph cutting algorithm and the number of computing devices, the full graph is cut to obtain multiple subgraphs; for each subgraph, the degree of each node in the subgraph in the full graph is obtained, wherein the degree represents the number of edges of the node; and each subgraph and the obtained degree of each node in the subgraph are sent to each computing device respectively;

[0044] Each computing device is used to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph using a local graph convolutional neural network to obtain output features of the nodes in the subgraph.

[0045] In the graph convolutional neural network training and graph operation method, device, electronic device and system provided by the embodiments of the present application, a full graph contained in a test data set is cut to obtain multiple subgraphs, and for each subgraph, the degree of each node in the subgraph in the full graph is obtained, and then each subgraph is distributed to each training device for training, and at the same time as the subgraph is sent down, the degree of each node in the subgraph obtained above is sent down. In this way, when each training device uses the local graph convolutional neural network to perform feedforward calculation on the subgraph, since the calculation is based on the degree of the full graph, it can ensure that the obtained node output features are consistent with the node output features of the full graph. In the process of training the graph convolutional neural network, it is equivalent to training the graph convolutional neural network based on the full graph, and since the data amount of the subgraph is much smaller than the data amount of the full graph, the computing resources of the training device are easily satisfied. Therefore, the distributed training method of multiple training devices is used to achieve the purpose of training the graph convolutional neural network model based on a large number of graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of a graph convolutional neural network training method according to an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the flow of message transmission in the feedforward phase of an embodiment of the present application;

[0049] Figure 3 It is a sub-graph after the graph of the embodiment of the present application is segmented;

[0050] Figure 4 A schematic diagram of the flow of sub-graph feed-forward calculation of a graph convolutional neural network according to an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the gradient integration process of an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of a graph convolutional neural network training device according to an embodiment of the present application;

[0053] Figure 7A schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0054] Figure 8 A schematic diagram of the structure of a distributed graph convolutional neural network training system according to an embodiment of the present application;

[0055] Fig. 9 A schematic diagram of a flow chart of a graph computing method according to an embodiment of the present application;

[0056] Fig.10 A schematic diagram of the structure of a graph computing device according to an embodiment of the present application;

[0057] Fig.11 A schematic diagram of the structure of an electronic device according to another embodiment of the present application;

[0058] Fig.12 A schematic diagram of the structure of a distributed graph computing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0060] In order to facilitate understanding of the embodiments of the present application, the terms in the embodiments of the present application are first explained below.

[0061] (1) Deep Learning

[0062] Deep learning is a research direction in the field of machine learning. Deep learning combines low-level features to form more abstract high-level representation attribute categories or features to learn the distributed feature representation of data. Deep learning mainly trains models composed of multi-layer neural networks. The training process includes three main processes: forward phase, backward phase, and update phase, which are defined as follows:

[0063] The forward stage refers to inputting the initial features of the training sample into the neural network model to obtain an output value;

[0064] The backward phase refers to calculating the gradient based on the output value obtained in the forward phase, the label of the training sample, and the related loss function;

[0065] In the update phase, the model parameters of the neural network model are updated according to the calculated gradients.

[0066] (2) Graphs and subgraphs

[0067] Graph: A data structure consisting of a set of nodes and a set of edges that can connect two nodes. This data structure can be used to describe various practical scenarios such as social networks, article citations, and transportation.

[0068] Subgraph: A concept in graph theory, referring to a graph whose node set and edge set are subsets of the node set and edge set of a certain graph respectively.

[0069] (3) Graph cutting

[0070] A concept in graph theory, that is, cutting a graph into multiple subgraphs based on nodes or edges and using a graph cutting algorithm. Graph cutting does not affect the original topological relationship between nodes and edges in the subgraphs. Depending on different requirements, the graph cutting algorithm can be different.

[0071] (4) Subring and degree

[0072] Sub-ring: Since the GCN algorithm needs to add edges pointing to each node in the graph in advance, this edge is called a self-ring.

[0073] Degree: A concept in graph theory. The degree of a node refers to the number of edges of the node.

[0074] (5) GCN algorithm

[0075] Taking GCN with two network layers as an example, the calculation process of the forward phase of the GCN algorithm can be summarized as shown in formula (1):

[0076]

[0077] Among them, X is the initial feature of all nodes, with a dimension of (N×C), where N is the number of nodes in the graph and C is the initial feature dimension of each node. Z is the output result after the forward phase calculation of the GCN algorithm; Softmax and ReLU are both activation functions; W 0 , W 1 They are the weight parameters of the first and second network layers of GCN, is the Fourier transform matrix of the graph, which has the form shown in formula (2):

[0078]

[0079] in, is the degree matrix of each node, which is a diagonal matrix of dimension (N×N). is the adjacency matrix of each node, with dimension (N×N), and The wavy lines above represent the degree matrix and adjacency matrix obtained by adding self-loops to the original graph.

[0080] The above is the process of the GCN algorithm. That is, each time a network layer is passed, the current features of all nodes are first compared with After performing matrix operations (also called graph Fourier transforms), new features are obtained, and then matrix operations are performed on the new features and the weight parameters W of the current layer. After the activation function, the output is used as the input feature of the next network layer. As the number of layers increases, the above process is repeated until the final output layer is reached. Finally, a loss function is constructed based on its output, and backward and update operations are performed to update the model parameters in the GCN model.

[0081] In order to realize the training of graph convolutional neural network models based on large-scale graphs, the embodiments of the present application provide a graph convolutional neural network training method, device, electronic device, storage medium and system. Below, the graph convolutional neural network training method provided by the embodiments of the present application is first introduced.

[0082] The execution subject of the graph convolutional neural network training method provided in the embodiment of the present application may be an electronic device with specific core processing capabilities, such as a management device in a distributed system. The method for implementing the graph convolutional neural network training method provided in the embodiment of the present application may be at least one of software, hardware circuits, and logic circuits set in the above-mentioned execution subject.

[0083] like Figure 1 As shown, the graph convolutional neural network training method provided in the embodiment of the present application may include the following steps.

[0084] S101, obtaining a test data set and the number of training devices, wherein the test data set includes a full graph, the full graph includes multiple nodes, and a line between two nodes constitutes an edge, indicating that the two nodes are associated.

[0085] S102, based on a preset graph cutting algorithm and the number of training devices, the full graph is cut to obtain multiple sub-graphs.

[0086] S103, for each subgraph, obtaining the degree of each node in the subgraph in the whole graph, wherein the degree represents the number of edges of the node.

[0087] S104, sending each subgraph and the degree of each node in the subgraph to each training device respectively, so that each training device uses a local graph convolutional neural network to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph, obtains the output features of the nodes in the subgraph, and calculates the gradient according to the output features of the nodes and the label information of the nodes.

[0088] S105, collecting the gradients obtained by each training device, and calculating the integrated gradient according to the gradients obtained by each training device.

[0089] S106, distribute the integrated gradient to each training device, so that each training device updates the model parameters of the local graph convolutional neural network model according to the integrated gradient.

[0090] Return to execute steps S104 to S106 until the training ends when the training end condition is met.

[0091] By applying the scheme of the embodiment of the present application, the full graph contained in the test data set is cut to obtain multiple subgraphs, and for each subgraph, the degree of each node in the subgraph in the full graph is obtained, and then each subgraph is distributed to each training device for training, and at the same time as the subgraph is issued, the degree of each node in the subgraph obtained above is issued. In this way, when each training device uses the local graph convolutional neural network to perform feedforward calculation on the subgraph, since it is calculated based on the degree of the full graph, it can ensure that the obtained node output features are consistent with the node output features of the full graph. In the process of training the graph convolutional neural network, it is equivalent to training the graph convolutional neural network based on the full graph, and since the data volume of the subgraph is much smaller than the data volume of the full graph, the computing resources of the training device are easily satisfied. Therefore, the distributed training method of multiple training devices is used to achieve the purpose of training the graph convolutional neural network model based on a large number of graphs. The resource advantages of multiple training devices for parallel training are brought into play, and the problems of resource overflow and long training time of a single training device can be solved, thereby significantly accelerating the training process of the graph convolutional neural network model.

[0092] Based on formula (1), the matrix operation based on the entire graph is split into a series of sum operations based on nodes, and formula (3) is obtained. It can be proved that formula (3) is completely equivalent to formula (1).

[0093]

[0094] Among them, k represents the kth network layer in GCN, and the subscript i represents the number of a node. represents the output feature of the node after the transformation of the kth network layer. j represents all neighbor nodes of node i. Since GCN adds self-loops, these neighbors also include node i itself.

[0095] deg(i) represents the degree of node i, and the reciprocal of the square root of deg(i) is called norm. That is:

[0096]

[0097] According to formula (3), the calculation process of the forward phase from the k-1th layer to the kth layer can be equivalently described as a message passing process, such as Figure 2 shown.

[0098] First, the message is constructed, and each node sends its current characteristics Multiply it by its own norm(i) to get Message(i) for subsequent aggregation. Then message passing is performed. For a node i, it receives Message(j) sent by all neighbors j, such as Figure 2 The A node shown in , its surrounding neighbors (i.e. Figure 2 Nodes B, C, and D in the network will pass the newly generated Message to node A, and the message passing process is the same to any other node. Then, after aggregation, node i sums up all the received Messages (j) and multiplies it by its own norm (i), and the aggregation is completed. Of course, the summation here is only one way of aggregation. The specific aggregation can also be achieved by taking the average, taking the maximum value, etc., which will not be listed here one by one. Finally, the aggregated message is compared with the parameter W of the neural network model. k-1 After multiplication, we get the features of the kth layer Figure 2 The example in shows that node A sums the features of nodes B, C, and D as its new feature.

[0099] In summary, the forward phase of the GCN algorithm in formula (1) can be implemented using Message Passing, while the backward phase and update phase are no different from the general deep learning process. In addition, when implementing the GCN algorithm from the perspective of Message Passing in formula (3), the GCN operation no longer needs to consider the matrix operation of the entire graph. Instead, it can be trained from the granularity of the node: for a specific node, as long as the neighbors of the node are fully sampled and the corresponding degree is the degree of the entire graph, even if Message Passing is performed in the subgraph, it can be guaranteed that the final output features of the point are completely consistent with the output features obtained by the full graph operation. This shows that subgraph training of graph convolutional neural networks is feasible.

[0100] The training device in the embodiment of the present application can be a training device equipped with a high-performance computing unit represented by a GPU (Graphics Processing Unit), and the specific training process is executed by the GPU installed therein. In order to enable each training device to train the graph convolutional neural network in parallel as much as possible, the subgraphs are generally divided according to the number of training devices, that is, the number of divided subgraphs is as equal as possible to the number of training devices. Therefore, after obtaining a test data set containing the full graph, the full graph can be cut based on a preset graph cutting algorithm and the number of training devices to obtain multiple subgraphs. Among them, the preset graph cutting algorithm can be an arbitrarily set cutting algorithm, such as cutting multiple nodes with a related relationship into a subgraph, or cutting a subgraph by using a node expansion scanning method, and so on.

[0101] In one implementation of the embodiment of the present application, S102 may specifically be: according to the number of training devices, multiple groups of central nodes are selected from the training nodes of the entire graph; for each group of central nodes, based on the neighbor relationship of the group of central nodes, an outward expansion scan is performed to obtain multi-order neighbor nodes and edges of the group of central nodes, and multiple subgraphs are obtained based on the obtained neighbor nodes and edges of each order.

[0102] The preset graph cutting algorithm adopted in the embodiment of the present application selects multiple groups of central nodes from all training nodes of the whole graph according to the number of training devices. The training nodes are part of the nodes in the whole graph. The number of training nodes generally accounts for more than 60% of the total number of nodes in the whole graph. In general, the training nodes can be evenly divided according to the number of training devices to select multiple groups of central nodes. For example, if the number of training devices is 10 and the number of training nodes is 100, 10 groups of central nodes are selected from the training nodes, and each group of central nodes has 10 training nodes. Starting from the central node, the scan is expanded outward based on the neighbor relationship of the central node. On this basis, the process can continue to expand outward multiple-order neighbor nodes and edges as needed, and finally obtain multiple subgraphs after segmentation based on the obtained neighbor nodes and edges of each order.

[0103] For each group of central nodes, based on the neighbor relationship of the group of central nodes, the scan is expanded outward to obtain multi-order neighbor nodes and edges of the group of central nodes, and the steps of obtaining multiple subgraphs based on the obtained neighbor nodes and edges of each order can be specifically as follows: for each group of central nodes, based on the neighbor relationship of the group of central nodes, according to the number of network layers of the graph convolutional neural network model, the scan is expanded outward N orders to obtain the neighbor nodes and edges of each order of the group of central nodes, and the subgraphs composed of the group of central nodes, neighbor nodes of each order and edges are determined, where N is the number of network layers. In other words, the order of outward expansion is determined by the number of network layers of the graph convolutional neural network. If the number of network layers of the graph convolutional neural network is two, it is necessary to expand the second-order neighbors outward from the central node. If the number of network layers of the graph convolutional neural network is three, it is necessary to expand the third-order neighbors outward from the central node, and so on.

[0104] Taking a graph convolutional neural network model with two network layers as an example, for each group of central nodes, based on the neighbor relationship of the group of central nodes, an outward expansion scan is performed to obtain multi-order neighbor nodes and edges of the group of central nodes, and the steps of obtaining multiple subgraphs according to the obtained each-order neighbor nodes and edges can be specifically as follows: for each group of central nodes, based on the neighbor relationship of the group of central nodes, an outward expansion scan is performed to obtain the first-order neighbor nodes and edges of the group of central nodes, and it is determined that the group of central nodes, first-order neighbor nodes, and edges between the group of central nodes and first-order neighbor nodes constitute a subgraph; based on the neighbor relationship of the first-order neighbor nodes, an outward expansion scan is performed to obtain the second-order neighbor nodes and edges of the group of central nodes, and it is determined that the central node, first-order neighbor nodes, second-order neighbor nodes, edges between the central node and first-order neighbor nodes, and edges between first-order neighbor nodes and second-order neighbor nodes constitute a subgraph.

[0105] like Figure 3 As shown in the figure, the central nodes are nodes 1, 2, and 3. Based on the neighbor relationship of the central node, the scan is expanded outward to identify that nodes 2, 3, 4, and 6 are the first-order neighbor nodes of node 1, nodes 1, 5, and 7 are the first-order neighbor nodes of node 2, and nodes 1, 6, 8, and 9 are the first-order neighbor nodes of node 3. It can be determined that nodes 1, 2, 3, 4, 5, 6, 7, 8, and 9 and the edges between these nodes constitute a subgraph (such as Figure 3 ), and also identify nodes 3, 5, 6, 7, 8, 9, 10, 11, 12, 15 as second-order neighbor nodes of node 1, nodes 3, 6, 4, 13, 14 as second-order neighbor nodes of node 2, and nodes 1, 2, 4, 6, 15, 16, 17, 18 as second-order neighbor nodes of node 3. Then it can be determined that nodes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 and the edges between these nodes constitute a subgraph (such as Figure 3 ).

[0106] If the training device directly performs feedforward calculation on the above subgraph, there are still defects. Since the neighbors of each second-order neighbor node in the subgraph are not collected, the degree of these second-order neighbor nodes is missing in the subgraph. The degree of a node represents the number of edges of the node. Specifically, the number of edges of the node can be directly set to the degree of the node, or the degree of the node can be set by assigning weights. The larger the weight, the more edges the node has. If the degree of each node is judged based on the subgraph, errors will occur. Therefore, before sending each subgraph to each training device, it is necessary to obtain the degree of each node in the subgraph in the full graph for each subgraph. The specific way to obtain the degree of each node can be: when obtaining the full graph, first record the degree of each node in the full graph, and after segmenting each subgraph, read the degree of each node in the subgraph in the full graph from the recorded information.

[0107] In one implementation of the embodiment of the present application, S103 may specifically be: for each subgraph, obtaining the degree of each node in the subgraph from the degree matrix of the full graph, wherein the degree matrix pre-records the degree of each node in the full graph.

[0108] The specific way to record the degree of each node in the full graph is to establish a degree matrix, which records the label of each node in the full graph and the degree of the node. Then, according to the label of each node in the subgraph, the degree of each node can be extracted from the degree matrix of the full graph. At the same time, the original features, labels and other additional information of these collected nodes that will not change will also be extracted and cached with the subgraph. This can prevent repeated sampling of subgraphs in subsequent iterations.

[0109] After obtaining the degree of the subgraph and each node in the subgraph, the subgraph and the degree of each node in the subgraph are sent to the training device. In this way, the training device can use the local graph convolutional neural network to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph to obtain the output features of the nodes in the subgraph. The specific feedforward calculation can be calculated using formula (3).

[0110] Due to the limitations of subgraphs, the neighboring nodes of the outermost nodes in the subgraph will be missing, so the information they receive will also be missing. Therefore, after each network layer, the feature representation of some nodes will deviate from the situation transmitted by the entire graph. However, after the feedforward calculation of multiple network layers, the output features of the training nodes are consistent with the features obtained by feedforward calculation using the entire graph, and only the output of the training nodes in the graph convolutional neural network will constitute the final loss function. Therefore, as long as the final output features of the central node selected from the training nodes are correct, the correctness of the subgraph training of the graph convolutional neural network model can be guaranteed.

[0111] After each training device obtains the output features of the node through feedforward calculation, it can use the loss function to calculate the gradient according to the output features of the node and the label information of the node, and feed the calculated gradient back to the electronic device. The electronic device collects the gradients obtained by each training device and calculates the integrated gradient based on the gradients obtained by each training device. The integrated gradient is then transmitted back to each training device, and each training device synchronously updates the model parameters of the local graph convolutional neural network model based on the integrated gradient.

[0112] S105 may specifically be: aggregating the gradients obtained by each training device through a parameter server or a ring global protocol, and calculating an average value to obtain an integrated gradient.

[0113] In the specific implementation, the gradients obtained by each training device can be aggregated and averaged through PS (Parameter Server) or RingAllreduce (Ring Global Reduction) to obtain the integrated gradient. Among them, both the PS method and the Ring Allreduce method have mature framework support in the current industry, such as TensorFlow, PyTorch, etc.

[0114] At this point, the feedforward, feedback and model parameter update process of one training iteration is completed. Subsequently, the steps S104 to S106 are returned to execute, and the training iteration is continued until the training termination condition is met, where the training termination condition can be that the number of training iterations reaches a preset number, or the difference between the output feature of the node and the label information of the node is less than a preset threshold.

[0115] Extended to any graph convolutional neural network model with N layers (N is greater than 2), the above subgraphing process only needs to expand the sampling scope of neighbor nodes. Generally, for a graph convolutional neural network model with N layers, sampling subgraphs also requires collecting N-order neighbors of the training nodes to construct subgraphs. This is the only way to ensure the correctness of the subgraphing algorithm. Apart from this, the other processes are exactly the same as the above two-layer network process.

[0116] To facilitate understanding, the graph convolutional neural network training method provided in the embodiment of the present application is introduced below with reference to specific examples.

[0117] Figure 4 The following is a flow chart of the GCN subgraph feedforward calculation process. Consider a two-layer GCN model as an example, as follows:

[0118] First, all nodes belonging to the test dataset are evenly divided according to the number of training devices. Figure 4For example, assume that each device is assigned 3 central nodes, namely nodes A, B, and C in the figure.

[0119] Then, taking these three points as the starting point, collect their first-order neighbor nodes. At this time, the central node and the first-order neighbor nodes constitute a subgraph of the original graph. Since two layers of GCN are required for Message Passing, according to formula (3), in order to make the output features of the three central nodes in the graph after Message Passing in two layers of network layers consistent with the output features of Message Passing of the whole graph, it is necessary to ensure that the features of all first-order neighbor nodes after one Message Passing are consistent with the effects after Message Passing of the whole graph. In other words, it is necessary to ensure that all neighbor nodes of the first-order neighbor nodes send messages to them, so the neighbor nodes of all first-order neighbor nodes must also be collected. Based on this, the first-order and second-order neighbor nodes of the three central nodes are obtained, forming a subgraph of the original graph.

[0120] There are still defects in directly performing Message Passing on the above subgraph. Since the neighbors of each second-order neighbor node in the subgraph are not collected, the degree of these second-order neighbor nodes is missing in the subgraph. If the degree of each node is judged based on the subgraph, errors will occur. Therefore, before starting Message Passing, it is necessary to extract the degree of each node from the degree matrix of the whole graph based on the identification information of each node in the subgraph, and then calculate the corresponding norm according to formula (4). At the same time, the original features, labels and other additional information of these collected nodes that will not change also need to be extracted together and cached with the sampled subgraph, so as to prevent repeated sampling of subgraphs in subsequent iterations.

[0121] Then, the Message Passing process described by formula (3) is performed in the subgraph, as Figure 4 As shown in the figure. The arrows in the figure indicate the direction of message transmission.

[0122] Due to the limitations of the subgraph, the neighbor nodes of the outermost nodes in the subgraph will be missing, so the messages they receive will also be missing. Therefore, after each network layer, the feature representation of some nodes will deviate from the situation of full-graph transmission. However, the output features of the three central nodes are consistent with the features obtained by full-graph message passing, and only the output features of the central node in GCN will constitute the final loss function. Therefore, as long as the final output features of the central node are correct, the correctness of GCN subgraph training is guaranteed.

[0123] The above steps are the feedforward calculation process of a single subgraph. Figure 5It shows that all central nodes are evenly divided according to the above method, subgraphs are obtained by sampling, and then distributed to different training devices, that is, each training device has a copy of GCN, and each training device independently collects subgraphs according to the assigned central node. After that, all subgraphs perform MessagePassing in parallel, so as to obtain the output features of each group of training nodes in parallel on different devices.

[0124] Finally, each device calculates the gradient separately, and then aggregates and averages the gradients obtained by each training device through the PS method or Ring Allreduce method to obtain the integrated gradient, which is then transmitted back to each training device. Each training device synchronously updates the model parameters of its own model copy based on the integrated gradient. Both the PS method and the RingAllreduce method are supported by mature frameworks in the current industry, such as TensorFlow, PyTorch, etc. Figure 5 The architecture of PS is used as an illustrative example of gradient integration.

[0125] So far, we have given a complete process of feedforward, backpropagation and model parameter update for a round of GCN subgraph training iteration. The above steps of feedforward, backpropagation and model parameter update are repeated and iterated, and the subgraph and the degree of each node in the subgraph only need to be obtained before the start of the first round of training, and then saved through cache for reuse.

[0126] Corresponding to Figure 1 The method embodiment shown in the figure, the embodiment of the present application provides a graph convolutional neural network training device, such as Figure 6 As shown, the device may include:

[0127] A first acquisition module 610 is used to acquire a test data set and the number of training devices, wherein the test data set includes a full graph, the full graph includes multiple nodes, and a line between two nodes constitutes an edge, indicating that the two nodes are associated;

[0128] A first subgraph cutting module 620, configured to cut the full graph based on a preset graph cutting algorithm and the number of training devices to obtain a plurality of subgraphs;

[0129] A first determination module 630 is used to obtain, for each subgraph, the degree of each node in the subgraph in the whole graph, where the degree represents the number of edges of the node;

[0130] A first sending module 640 is used to send each subgraph and the degree of each node in the subgraph to each training device, so that each training device uses a local graph convolutional neural network to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph, obtain the output features of the nodes in the subgraph, and calculate the gradient according to the output features of the nodes and the label information of the nodes;

[0131] A gradient integration module 650 is used to collect the gradients obtained by each training device and calculate the integrated gradient according to the gradients obtained by each training device;

[0132] The first sending module 640 is also used to distribute the integrated gradient to each training device so that each training device updates the model parameters of the local graph convolutional neural network model according to the integrated gradient.

[0133] Optionally, the first subgraph cutting module 620 can be specifically used to select multiple groups of central nodes from the training nodes of the entire graph according to the number of training devices; for each group of central nodes, based on the neighbor relationship of the group of central nodes, an outward expansion scan is performed to obtain multi-order neighbor nodes and edges of the group of central nodes, and multiple subgraphs are obtained based on the obtained neighbor nodes and edges of each order.

[0134] Optionally, the first subgraph cutting module 620 can be specifically used to expand outward and scan N orders for each group of central nodes based on the neighbor relationship of the group of central nodes and according to the number of network layers of the graph convolutional neural network model, to obtain the neighbor nodes and edges of each order of the group of central nodes, and determine that the group of central nodes, neighbor nodes of each order and edges constitute a subgraph, where N is the number of network layers.

[0135] Optionally, the first determination module 630 may be specifically configured to obtain, for each subgraph, the degree of each node in the subgraph from the degree matrix of the full graph, wherein the degree matrix pre-records the degree of each node in the full graph.

[0136] Optionally, the gradient integration module 650 can be specifically used to aggregate the gradients obtained by each training device through a parameter server or a ring global reduction, and calculate the average value to obtain an integrated gradient.

[0137] By applying the scheme of the embodiment of the present application, the full graph contained in the test data set is cut to obtain multiple subgraphs, and for each subgraph, the degree of each node in the subgraph in the full graph is obtained, and then each subgraph is distributed to each training device for training, and at the same time as the subgraph is issued, the degree of each node in the subgraph obtained above is issued. In this way, when each training device uses the local graph convolutional neural network to perform feedforward calculation on the subgraph, since it is calculated based on the degree of the full graph, it can ensure that the obtained node output features are consistent with the node output features of the full graph. In the process of training the graph convolutional neural network, it is equivalent to training the graph convolutional neural network based on the full graph, and since the data volume of the subgraph is much smaller than the data volume of the full graph, the computing resources of the training device are easily satisfied. Therefore, the distributed training method of multiple training devices is used to achieve the purpose of training the graph convolutional neural network model based on a large number of graphs. The resource advantages of multiple training devices for parallel training are brought into play, and the problems of resource overflow and long training time of a single training device can be solved, thereby significantly accelerating the training process of the graph convolutional neural network model.

[0138] The present application also provides an electronic device, such as Figure 7 As shown, it includes a processor 701 and a memory 702; wherein the memory 702 is used to store computer programs; the processor 701 is used to implement the above-mentioned graph convolutional neural network training method when executing the computer program stored in the memory 702.

[0139] In addition, an embodiment of the present application also provides a machine-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned graph convolutional neural network training method is implemented.

[0140] An embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned graph convolutional neural network training method.

[0141] The present application also provides a distributed graph convolutional neural network training system. Figure 8 As shown, the system includes a management device 810 and a plurality of training devices 820;

[0142] The management device 810 is used to obtain a test data set and the number of training devices, wherein the test data set includes a full graph, the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, which represents the association between the two nodes; based on a preset graph cutting algorithm and the number of training devices, the full graph is cut to obtain multiple subgraphs; for each subgraph, the degree of each node in the subgraph in the full graph is obtained, wherein the degree represents the number of edges of the node; each subgraph and the degree of each node in the subgraph obtained are sent to each training device 820 respectively;

[0143] Each training device 820 is used to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph using a local graph convolutional neural network to obtain output features of the nodes in the subgraph, and calculate the gradient according to the output features of the nodes and the label information of the nodes; and feed the calculated gradient back to the management device 810;

[0144] The management device 810 is further used to collect the gradients obtained by each training device 820, and calculate the integrated gradient according to the gradients obtained by each training device 820; and distribute the integrated gradient to each training device 820;

[0145] Each training device 820 is also used to update the model parameters of the local graph convolutional neural network model based on the integrated gradient.

[0146] By applying the scheme of the embodiment of the present application, the full graph contained in the test data set is cut to obtain multiple subgraphs, and for each subgraph, the degree of each node in the subgraph in the full graph is obtained, and then each subgraph is distributed to each training device for training, and at the same time as the subgraph is issued, the degree of each node in the subgraph obtained above is issued. In this way, when each training device uses the local graph convolutional neural network to perform feedforward calculation on the subgraph, since it is calculated based on the degree of the full graph, it can ensure that the obtained node output features are consistent with the node output features of the full graph. In the process of training the graph convolutional neural network, it is equivalent to training the graph convolutional neural network based on the full graph, and since the data volume of the subgraph is much smaller than the data volume of the full graph, the computing resources of the training device are easily satisfied. Therefore, the distributed training method of multiple training devices is used to achieve the purpose of training the graph convolutional neural network model based on a large number of graphs. The resource advantages of multiple training devices for parallel training are brought into play, and the problems of resource overflow and long training time of a single training device can be solved, thereby significantly accelerating the training process of the graph convolutional neural network model.

[0147] For the graph convolutional neural network training device, electronic device, machine-readable storage medium, computer program product and distributed graph convolutional neural network training system embodiments, since the method contents involved are basically similar to the aforementioned graph convolutional neural network training method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0148] Since the graph operation process, that is, the graph convolutional neural network model's reasoning process on the graph is essentially a feedforward calculation process of the above-mentioned graph convolutional neural network model after the training is completed, not only the training process but also the graph operation process can use the same steps as above to perform sub-graph operations.

[0149] In order to implement sub-graph computing, the present application embodiment provides a graph computing method, device, electronic device, storage medium and system. Below, the graph computing method provided by the present application embodiment is first introduced.

[0150] The execution subject of the graph operation method provided in the embodiment of the present application may be an electronic device with specific core processing capabilities, such as a management device in a distributed system. The method for implementing the graph operation method provided in the embodiment of the present application may be at least one of software, hardware circuits, and logic circuits set in the above-mentioned execution subject.

[0151] like Fig. 9 As shown, the graph operation method provided in the embodiment of the present application may include the following steps.

[0152] S901, obtaining a full graph to be calculated and the number of computing devices, wherein the full graph includes multiple nodes, and a line between two nodes constitutes an edge, indicating that the two nodes are associated.

[0153] S902, based on a preset graph cutting algorithm and the number of computing devices, the entire graph is cut to obtain a plurality of sub-graphs.

[0154] S903, for each subgraph, obtaining the degree of each node in the subgraph in the entire graph, where the degree represents the number of edges of the node.

[0155] S904, sending each subgraph and the degree of each node in the subgraph to each computing device respectively, so that each computing device uses a local graph convolutional neural network to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph to obtain the output features of the nodes in the subgraph.

[0156] By applying the solution of the embodiment of the present application, the whole graph is cut to obtain multiple subgraphs, and for each subgraph, the degree of each node in the subgraph in the whole graph is obtained, and then each subgraph is distributed to each computing device for calculation, and at the same time as the subgraph is sent down, the degree of each node in the subgraph obtained above is also sent down. In this way, when each computing device uses the local graph convolutional neural network to perform feedforward calculation on the subgraph, since the calculation is based on the degree of the whole graph, it can ensure that the obtained node output features are consistent with the node output features of the whole graph, thereby achieving the purpose of calculating large-scale graphs.

[0157] The computing device in the embodiment of the present application can be a computing device equipped with a high-performance computing unit represented by a GPU, and the specific computing process is executed by the GPU installed therein. After obtaining the number of computing devices, the subgraphs can be divided according to the number of computing devices, that is, the number of divided subgraphs is as equal as possible to the number of computing devices. Therefore, after obtaining the full graph to be calculated, the full graph can be cut based on the preset graph cutting algorithm and the number of computing devices to obtain multiple subgraphs. Among them, the preset graph cutting algorithm can be an arbitrarily set cutting algorithm, such as cutting N nodes with a related relationship into a subgraph, or cutting a subgraph by using a node expansion scanning method, etc.

[0158] In one implementation of an embodiment of the present application, S902 may specifically be: according to the number of computing devices, multiple groups of central nodes are selected from the training nodes of the entire graph; for each group of central nodes, based on the neighbor relationships of the group of central nodes, an outward expansion scan is performed to obtain multi-order neighbor nodes and edges of the group of central nodes, and multiple subgraphs are obtained based on the obtained neighbor nodes and edges of each order.

[0159] The preset graph cutting algorithm adopted in the embodiment of the present application selects multiple groups of central nodes from all training nodes of the whole graph according to the number of computing devices. The training nodes are part of the nodes in the whole graph, and the number of training nodes generally accounts for more than 60% of the total number of nodes in the whole graph. In general, the training nodes can be evenly divided according to the number of computing devices to select multiple groups of central nodes. For example, if the number of computing devices is 10 and the number of training nodes is 100, 10 groups of central nodes are selected from the training nodes, and each group of central nodes has 10 training nodes. Starting from the central node, the scan is expanded outward based on the neighbor relationship of the central node. On this basis, the process can continue to expand outward multiple-order neighbor nodes and edges as needed, and finally obtain multiple subgraphs after segmentation based on the obtained neighbor nodes and edges of each order.

[0160] For each group of central nodes, based on the neighbor relationship of the group of central nodes, the scan is expanded outward to obtain multi-order neighbor nodes and edges of the group of central nodes, and the steps of obtaining multiple subgraphs based on the obtained neighbor nodes and edges of each order can be specifically as follows: for each group of central nodes, based on the neighbor relationship of the group of central nodes, according to the number of network layers of the graph convolutional neural network model, the scan is expanded outward N orders to obtain the neighbor nodes and edges of each order of the group of central nodes, and the subgraphs composed of the group of central nodes, neighbor nodes of each order and edges are determined, where N is the number of network layers. In other words, the order of outward expansion is determined by the number of network layers of the graph convolutional neural network. If the number of network layers of the graph convolutional neural network is two, it is necessary to expand the second-order neighbors outward from the central node. If the number of network layers of the graph convolutional neural network is three, it is necessary to expand the third-order neighbors outward from the central node, and so on.

[0161] Taking a graph convolutional neural network model with two network layers as an example, for each group of central nodes, based on the neighbor relationship of the group of central nodes, an outward expansion scan is performed to obtain multi-order neighbor nodes and edges of the group of central nodes, and the steps of obtaining multiple subgraphs according to the obtained each-order neighbor nodes and edges can be specifically as follows: for each group of central nodes, based on the neighbor relationship of the group of central nodes, an outward expansion scan is performed to obtain the first-order neighbor nodes and edges of the group of central nodes, and it is determined that the group of central nodes, first-order neighbor nodes, and edges between the group of central nodes and first-order neighbor nodes constitute a subgraph; based on the neighbor relationship of the first-order neighbor nodes, an outward expansion scan is performed to obtain the second-order neighbor nodes and edges of the group of central nodes, and it is determined that the central node, first-order neighbor nodes, second-order neighbor nodes, edges between the central node and first-order neighbor nodes, and edges between first-order neighbor nodes and second-order neighbor nodes constitute a subgraph.

[0162] If the computing device directly performs feedforward calculation on the above subgraph, there are still defects. Since the neighbors of each second-order neighbor node in the subgraph are not collected, the degree of these second-order neighbor nodes is missing in the subgraph. The degree of a node represents the number of edges of the node. Specifically, the number of edges of the node can be directly set to the degree of the node, or the degree of the node can be set by assigning weights. The larger the weight, the more edges the node has. If the degree of each node is judged based on the subgraph, errors will occur. Therefore, before sending each subgraph to each computing device, it is necessary to obtain the degree of each node in the subgraph in the full graph for each subgraph. The specific way to obtain the degree of each node can be: when obtaining the full graph, first record the degree of each node in the full graph, and after segmenting each subgraph, read the degree of each node in the subgraph in the full graph from the recorded information.

[0163] In one implementation of the embodiment of the present application, S903 may specifically be: for each subgraph, obtaining the degree of each node in the subgraph from the degree matrix of the full graph, wherein the degree matrix pre-records the degree of each node in the full graph.

[0164] The specific way to record the degree of each node in the full graph is to establish a degree matrix, which records the label of each node in the full graph and the degree of the node. Then, according to the label of each node in the subgraph, the degree of each node can be extracted from the degree matrix of the full graph. At the same time, the original features, labels and other additional information of these collected nodes that will not change will also be extracted and cached with the subgraph. This can prevent repeated sampling of subgraphs in subsequent iterations.

[0165] After obtaining the degree of the subgraph and each node in the subgraph, the subgraph and the degree of each node in the subgraph are sent to the computing device. In this way, the computing device can use the local graph convolutional neural network to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph to obtain the output features of the nodes in the subgraph. The specific feedforward calculation can be calculated using formula (3).

[0166] Corresponding to Fig. 9 The method embodiment shown in the figure, the embodiment of the present application provides a graph computing device, such as Fig.10 As shown, the device may include:

[0167] The second acquisition module 1010 is used to acquire the full graph to be calculated and the number of computing devices, wherein the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, indicating that the two nodes are associated;

[0168] The second subgraph cutting module 1020 is used to cut the full graph based on a preset graph cutting algorithm and the number of computing devices to obtain multiple subgraphs;

[0169] The second determination module 1030 is used to obtain, for each subgraph, the degree of each node in the subgraph in the whole graph, where the degree represents the number of edges of the node;

[0170] The second sending module 1040 is used to send each subgraph and the degree of each node in the subgraph to each computing device, so that each computing device can perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph and use the local graph convolutional neural network to obtain the output features of the nodes in the subgraph.

[0171] Optionally, the second subgraph cutting module 1020 can be specifically used to select multiple groups of central nodes from the training nodes of the entire graph according to the number of computing devices; for each group of central nodes, based on the neighbor relationship of the group of central nodes, an outward expansion scan is performed to obtain multi-order neighbor nodes and edges of the group of central nodes, and multiple subgraphs are obtained based on the obtained neighbor nodes and edges of each order.

[0172] Optionally, the second subgraph cutting module 1020 can be specifically used to expand outward and scan N orders for each group of central nodes based on the neighbor relationship of the group of central nodes and according to the number of network layers of the graph convolutional neural network model, to obtain the neighbor nodes and edges of each order of the group of central nodes, and determine that the group of central nodes, neighbor nodes of each order and edges constitute a subgraph, where N is the number of network layers.

[0173] Optionally, the second determination module 1030 may be specifically configured to obtain, for each subgraph, the degree of each node in the subgraph from the degree matrix of the full graph, wherein the degree matrix pre-records the degree of each node in the full graph.

[0174] By applying the solution of the embodiment of the present application, the whole graph is cut to obtain multiple subgraphs, and for each subgraph, the degree of each node in the subgraph in the whole graph is obtained, and then each subgraph is distributed to each computing device for calculation, and at the same time as the subgraph is sent down, the degree of each node in the subgraph obtained above is also sent down. In this way, when each computing device uses the local graph convolutional neural network to perform feedforward calculation on the subgraph, since the calculation is based on the degree of the whole graph, it can ensure that the obtained node output features are consistent with the node output features of the whole graph, thereby achieving the purpose of calculating large-scale graphs.

[0175] The present application also provides an electronic device, such as Fig.11 As shown, it includes a processor 1101 and a memory 1102; wherein the memory 1102 is used to store computer programs; the processor 1101 is used to implement the above-mentioned graph operation method when executing the computer program stored in the memory 1102.

[0176] The memory may include a RAM (Random Access Memory), or may include a NVM (Non-volatile Memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the processor.

[0177] The above-mentioned processor can be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0178] In addition, an embodiment of the present application further provides a machine-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the above-mentioned graph operation method is implemented.

[0179] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the above-mentioned graph operation method.

[0180] The present application also provides a distributed graph computing system. Fig.12As shown, the system includes a management device 1210 and a plurality of computing devices 1220;

[0181] The management device 1210 is used to obtain a full graph to be calculated and the number of computing devices, wherein the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, which represents the association between the two nodes; based on a preset graph cutting algorithm and the number of computing devices, the full graph is cut to obtain multiple subgraphs; for each subgraph, the degree of each node in the subgraph in the full graph is obtained, wherein the degree represents the number of edges of the node; and each subgraph and the obtained degree of each node in the subgraph are sent to each computing device 1220 respectively;

[0182] Each computing device 1220 is used to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph using a local graph convolutional neural network to obtain output features of the nodes in the subgraph.

[0183] By applying the solution of the embodiment of the present application, the whole graph is cut to obtain multiple subgraphs, and for each subgraph, the degree of each node in the subgraph in the whole graph is obtained, and then each subgraph is distributed to each computing device for calculation, and at the same time as the subgraph is sent down, the degree of each node in the subgraph obtained above is also sent down. In this way, when each computing device uses the local graph convolutional neural network to perform feedforward calculation on the subgraph, since the calculation is based on the degree of the whole graph, it can ensure that the obtained node output features are consistent with the node output features of the whole graph, thereby achieving the purpose of calculating large-scale graphs.

[0184] For the graph computing device, electronic device, machine-readable storage medium, computer program product and distributed graph computing system embodiments, since the method contents involved are basically similar to the aforementioned graph computing method embodiments, the description is relatively simple and the relevant parts can be referred to the partial description of the method embodiments.

[0185] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, DSL (Digital Subscriber Line, digital subscriber line)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD (Digital Versatile Disc)), or a semiconductor medium (such as an SSD (Solid State Disk)).

[0186] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0187] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, machine-readable storage medium, computer program product and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0188] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A graph convolutional neural network training method, characterized in that: A management device applied to a distributed system, wherein the distributed system includes the management device and a plurality of training devices, and the method includes: Obtaining a test data set and the number of training devices, wherein the test data set includes a full graph, the full graph includes multiple nodes, and a line between two nodes constitutes an edge, indicating that the two nodes are associated; Based on a preset graph cutting algorithm and the number of training devices, the full graph is cut to obtain a plurality of sub-graphs; For each subgraph, obtain the degree of each node in the subgraph in the full graph, where the degree represents the number of edges of the node; Sending each subgraph and the degree of each node in the subgraph to each training device respectively, so that each training device uses a local graph convolutional neural network based on the degree of each node in the input subgraph to perform feedforward calculation on the subgraph, obtain the output features of the nodes in the subgraph, and calculate the gradient according to the output features of the nodes and the label information of the nodes; Collecting the gradients obtained by the training devices, and calculating the integrated gradient according to the gradients obtained by the training devices; Distributing the integrated gradient to each of the training devices, so that each of the training devices updates the model parameters of the local graph convolutional neural network model according to the integrated gradient; Return to the step of sending each subgraph and the degree of each node in the subgraph to each training device, until the training is terminated when the training termination condition is met; The step of cutting the full graph to obtain a plurality of sub-graphs based on a preset graph cutting algorithm and the number of training devices includes: According to the number of training devices, selecting multiple groups of central nodes from the training nodes of the full graph; For each group of central nodes, based on the neighbor relationship of the group of central nodes, according to the number of network layers of the graph convolutional neural network model, the scan is expanded outward N orders to obtain the neighbor nodes and edges of each order of the group of central nodes, and determine the subgraph composed of the group of central nodes, neighbor nodes of each order and edges, where N is the number of network layers.

2. The method according to claim 1, characterized in that The step of obtaining, for each subgraph, the degree of each node in the subgraph in the full graph comprises: For each subgraph, the degree of each node in the subgraph is obtained from the degree matrix of the full graph, wherein the degree of each node in the full graph is pre-recorded in the degree matrix.

3. The method according to claim 1, characterized in that The step of collecting the gradients obtained by the training devices and calculating the integrated gradient according to the gradients obtained by the training devices comprises: The gradients obtained by the training devices are aggregated by means of a parameter server or a ring global reduction, and the average value is calculated to obtain an integrated gradient.

4. A graph operation method, characterized in that: A management device applied to a distributed system, wherein the distributed system includes the management device and a plurality of computing devices, and the method includes: Obtaining a full graph to be calculated and the number of computing devices, wherein the full graph includes multiple nodes, and a line between two nodes constitutes an edge, indicating that the two nodes are associated; Based on a preset graph cutting algorithm and the number of computing devices, the full graph is cut to obtain a plurality of sub-graphs; For each subgraph, obtain the degree of each node in the subgraph in the full graph, where the degree represents the number of edges of the node; Sending each subgraph and the degree of each node in the subgraph to each computing device respectively, so that each computing device uses a local graph convolutional neural network to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph to obtain output features of the nodes in the subgraph; The step of cutting the full graph to obtain a plurality of sub-graphs based on a preset graph cutting algorithm and the number of computing devices includes: Selecting multiple groups of central nodes from the training nodes of the full graph according to the number of computing devices; For each group of central nodes, based on the neighbor relationship of the group of central nodes, according to the number of network layers of the graph convolutional neural network model, the scan is expanded outward N orders to obtain the neighbor nodes and edges of each order of the group of central nodes, and determine the subgraph composed of the group of central nodes, neighbor nodes of each order and edges, where N is the number of network layers.

5. The method according to claim 4, characterized in that The step of obtaining, for each subgraph, the degree of each node in the subgraph in the full graph comprises: For each subgraph, the degree of each node in the subgraph is obtained from the degree matrix of the full graph, wherein the degree of each node in the full graph is pre-recorded in the degree matrix.

6. A graph convolutional neural network training device, characterized in that: A management device applied to a distributed system, wherein the distributed system includes the management device and a plurality of training devices, and the apparatus includes: A first acquisition module is used to acquire a test data set and the number of training devices, wherein the test data set includes a full graph, the full graph includes a plurality of nodes, and a line between two nodes constitutes an edge, indicating that the two nodes are associated; A first subgraph cutting module, configured to cut the full graph based on a preset graph cutting algorithm and the number of training devices to obtain a plurality of subgraphs; A first determination module is used to obtain, for each subgraph, the degree of each node in the subgraph in the full graph, wherein the degree represents the number of edges of the node; A first sending module is used to send each subgraph and the degree of each node in the subgraph to each training device, so that each training device uses a local graph convolutional neural network based on the degree of each node in the input subgraph to perform feedforward calculation on the subgraph, obtain the output features of the nodes in the subgraph, and calculate the gradient according to the output features of the nodes and the label information of the nodes; A gradient integration module, used to collect the gradients obtained by the training devices and calculate the integrated gradient according to the gradients obtained by the training devices; The first sending module is further used to distribute the integrated gradient to each of the training devices, so that each of the training devices updates the model parameters of the local graph convolutional neural network model according to the integrated gradient; Among them, the first subgraph cutting module is specifically used to select multiple groups of central nodes from the training nodes of the full graph according to the number of training devices; for each group of central nodes, based on the neighbor relationship of the group of central nodes, according to the number of network layers of the graph convolutional neural network model, expand outward and scan N orders to obtain the neighbor nodes and edges of each order of the group of central nodes, and determine that the group of central nodes, neighbor nodes of each order and edges constitute a subgraph, where N is the number of network layers.

7. A graph computing device, characterized in that: A management device applied to a distributed system, wherein the distributed system includes the management device and a plurality of computing devices, and the apparatus includes: A second acquisition module is used to acquire a full graph to be calculated and the number of computing devices, wherein the full graph includes multiple nodes, and a line between two nodes constitutes an edge, indicating that the two nodes are associated; A second subgraph cutting module, configured to cut the full graph based on a preset graph cutting algorithm and the number of computing devices to obtain a plurality of subgraphs; A second determination module is used to obtain, for each subgraph, the degree of each node in the subgraph in the full graph, wherein the degree represents the number of edges of the node; A second sending module is used to send each subgraph and the degree of each node in the subgraph to each computing device, so that each computing device uses a local graph convolutional neural network to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph to obtain output features of the nodes in the subgraph; Among them, the second subgraph cutting module is specifically used to select multiple groups of central nodes from the training nodes of the full graph according to the number of computing devices; for each group of central nodes, based on the neighbor relationship of the group of central nodes, according to the number of network layers of the graph convolutional neural network model, expand outward and scan N orders to obtain the neighbor nodes and edges of each order of the group of central nodes, and determine that the group of central nodes, neighbor nodes of each order and edges constitute a subgraph, where N is the number of network layers.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor, when used to execute the computer program stored in the memory, implements the method described in any one of claims 1 to 3 or the method described in any one of claims 4 to 5.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 3 or any one of claims 4 to 5 is implemented.

10. A distributed graph convolutional neural network training system, characterized in that: The system includes a management device and a plurality of training devices; The management device is used to obtain a test data set and the number of training devices, wherein the test data set includes a full graph, the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, which represents the association between the two nodes; based on a preset graph cutting algorithm and the number of training devices, the full graph is cut to obtain multiple subgraphs; for each subgraph, the degree of each node in the subgraph in the full graph is obtained, wherein the degree represents the number of edges of the node; each subgraph and the obtained degree of each node in the subgraph are sent to each training device respectively; Each training device is used to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph using a local graph convolutional neural network to obtain output features of the nodes in the subgraph, and calculate the gradient according to the output features of the nodes and the label information of the nodes; and feed the calculated gradient back to the management device; The management device is further used to collect the gradients obtained by the training devices, calculate the integrated gradient according to the gradients obtained by the training devices, and distribute the integrated gradient to the training devices; Each training device is further used to update the model parameters of the local graph convolutional neural network model according to the integrated gradient; The management device is specifically used to select multiple groups of central nodes from the training nodes of the full graph according to the number of training devices; for each group of central nodes, based on the neighbor relationship of the group of central nodes, according to the number of network layers of the graph convolutional neural network model, expand outward and scan N orders to obtain neighbor nodes and edges of each order of the group of central nodes, and determine that the group of central nodes, neighbor nodes of each order and edges constitute a subgraph, where N is the number of network layers.

11. A distributed graph computing system, characterized in that: The system includes a management device and a plurality of computing devices; The management device is used to obtain a full graph to be calculated and the number of computing devices, wherein the full graph includes multiple nodes, and the connection between two nodes constitutes an edge, which represents the association between the two nodes; based on a preset graph cutting algorithm and the number of computing devices, the full graph is cut to obtain multiple subgraphs; for each subgraph, the degree of each node in the subgraph in the full graph is obtained, wherein the degree represents the number of edges of the node; and each subgraph and the obtained degree of each node in the subgraph are sent to each computing device respectively; Each computing device is used to perform feedforward calculation on the subgraph based on the degree of each node in the input subgraph using a local graph convolutional neural network to obtain output features of the nodes in the subgraph; The management device is specifically used to select multiple groups of central nodes from the training nodes of the full graph according to the number of computing devices; for each group of central nodes, based on the neighbor relationship of the group of central nodes, according to the number of network layers of the graph convolutional neural network model, expand outward and scan N orders to obtain neighbor nodes and edges of each order of the group of central nodes, and determine that the group of central nodes, neighbor nodes of each order and edges constitute a subgraph, where N is the number of network layers.

Images