Graph neural network training method and device, electronic equipment and storage medium

Abstract

The invention discloses a graph neural network training method and device, electronic equipment and a storage medium, and belongs to the technical field of graph neural networks. The method comprises the following steps: analyzing a graph neural network training task to obtain network parameters and hardware resource limiting conditions of an original graph neural network; dividing the training process of the original graph neural network into at least one training optimization stage, and determining training parameters corresponding to each training optimization stage; determining a performance prediction result of the graph neural network based on the initial parameter combination value of the training parameter, the network parameter of the original graph neural network and a hardware resource limiting condition, and searching a target parameter combination value of the training parameter in a parameter combination value space of the training parameter according to the performance prediction result; and training the original image neural network based on the target parameter combination value of the training parameter to obtain a target image neural network. Through the method, the adaptive capability of the graph neural network under diversified requirements can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of graph neural network technology, and in particular to a graph neural network training method, device, electronic device and storage medium. Background Art

[0002] Graph Neural Networks (GNNs) is a deep learning model that processes graph-structured data. Graphs consist of nodes (or vertices) and edges connecting these nodes, and can effectively represent various relationships, such as user relationships in social networks, routes in traffic networks, etc. In machine learning and deep learning, by training GNNs with training data sets, the model can learn the features and patterns in the data and apply them to subsequent prediction or classification tasks.

[0003] However, existing graph neural network training methods often only focus on training speed, but do not give enough consideration to other operating indicators during the training process, resulting in limited adaptability in different demand scenarios. Summary of the invention

[0004] The embodiments of the present application provide a graph neural network training method, device, electronic device and storage medium to at least solve the problem of insufficient adaptability of existing graph neural network training methods under different demand scenarios.

[0005] In order to solve the above technical problems, this application is implemented as follows: In a first aspect, an embodiment of the present application provides a graph neural network training method, comprising: obtaining network parameters and hardware resource constraints of the original graph neural network by parsing the graph neural network training task; dividing the training process of the original graph neural network into at least one training optimization stage, and determining the training parameters corresponding to each of the training optimization stages; determining a performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network, and the hardware resource constraints, and searching for a target parameter combination value of the training parameters in the parameter combination value space of the training parameters based on the performance prediction result; and training the original graph neural network based on the target parameter combination value of the training parameters to obtain a target graph neural network.

[0006] In a second aspect, an embodiment of the present application provides a graph neural network training device, comprising: a task analysis module, used to obtain network parameters and hardware resource constraints of the original graph neural network by analyzing the graph neural network training task; a parameter determination module, used to divide the training process of the original graph neural network into at least one training optimization stage, and determine the training parameters corresponding to each of the training optimization stages; a performance prediction module, used to determine the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network and the hardware resource constraints, and search for the target parameter combination values ​​of the training parameters in the parameter combination value space of the training parameters according to the performance prediction results; a network training module, used to train the original graph neural network based on the target parameter combination values ​​of the training parameters to obtain a target graph neural network.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect above are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the method described in the first aspect above.

[0010] In an embodiment of the present application, the network parameters and hardware resource constraints of the original graph neural network are obtained by parsing the graph neural network training task; the training process of the original graph neural network is divided into at least one training optimization stage, and the training parameters corresponding to each training optimization stage are determined; based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network, and the hardware resource constraints, the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters are determined, and the target parameter combination values ​​of the training parameters are searched in the parameter combination value space of the training parameters according to the performance prediction results; based on the target parameter combination values ​​of the training parameters, the original graph neural network is trained to obtain the target graph neural network. In this way, by dividing the training process of the original graph neural network into at least one training optimization stage, determining the training parameters corresponding to each training optimization stage, and searching the target parameter combination values ​​from the parameter combination value space of the training parameters based on the graph neural network training task, efficient optimization of the graph neural network training process can be achieved, while improving the adaptability of the graph neural network under diversified needs.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 A schematic diagram of the process of the graph neural network training method provided in an embodiment of the present application is shown; Figure 2 An architectural diagram of a graph neural network training method provided in an embodiment of the present application is shown; Figure 3 A schematic diagram of the process of parameter space search provided by an embodiment of the present application is shown; Figure 4 A schematic diagram of the structure of a graph neural network training device provided in an embodiment of the present application is shown; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0014] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0015] Figure 1 A flow chart of the graph neural network training method provided in an embodiment of the present application is shown. The execution subject of the method may be a terminal device or a server, wherein the terminal device may be a device such as a personal computer, or a mobile terminal device such as a mobile phone or a tablet computer, and the terminal device may be a terminal device used by a user. The server may be an independent server, or a server cluster composed of multiple servers, and the server may be a background server for a certain business, or a background server for a certain platform or application (for example, a cloud computing platform, an application system in a specific field, etc.). In the embodiment of the present application, the execution subject is taken as an example to illustrate. For the case of a terminal device, it can be processed according to the following relevant content, which will not be repeated here. As shown in the figure, the graph neural network training method 100 may include the following steps: Step 101: Analyze the graph neural network training task to obtain the network parameters and hardware resource constraints of the original graph neural network.

[0016] Among them, the graph neural network training task includes the training task scenario, graph neural network type, data set for training and hardware resource configuration information.

[0017] In a specific implementation, in response to a graph neural network training task input by a user, the graph neural network task is parsed to obtain network parameters such as the number of network layers and graph basic data of the original graph neural network and hardware resource constraints. For example, the network parameters of the original graph neural network are determined according to the graph neural network type and the data set for training. The graph neural network types include graph convolutional network (GCN), graph attention network (GAT), heterogeneous graph network (HGN), knowledge graph, homogeneous graph model, dynamic graph model, etc.; the network parameters of the original graph neural network are statically analyzed, for example, the original graph neural network is subjected to node number statistics, neighborhood analysis, feature dimension statistics, and clustering coefficient calculation, etc., to obtain the training complexity of the original graph neural network, and the hardware resource constraints are determined in combination with the training task scenario and hardware resource configuration information. Among them, the training task scenario includes but is not limited to social network analysis, recommendation system, knowledge graph reasoning, etc.

[0018] Step 102: Divide the training process of the original graph neural network into at least one training optimization stage, and determine the training parameters corresponding to each of the training optimization stages.

[0019] Among them, the training optimization stage can include a preprocessing optimization stage, a sampling optimization stage, a data transmission optimization stage, and a model calculation optimization stage.

[0020] In a specific implementation, the training process of the original graph neural network can be divided into at least one training optimization stage as needed. For example, the training process of the original graph neural network can be divided into a sampling optimization stage, a data transmission optimization stage, and a model calculation optimization stage. Each training optimization stage corresponds to different training parameters. For example, the sampling optimization stage corresponds to the number of sampling hops, sampling bias, batch data size, etc.; the data transmission optimization stage corresponds to the cache ratio, etc.; the model calculation optimization stage corresponds to the subgraph size of the batch data, the node feature vector dimension, etc., so as to determine the training parameters corresponding to each training optimization stage.

[0021] Step 103: Based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network and hardware resource constraints, determine the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters, and search for the target parameter combination values ​​of the training parameters in the parameter combination value space of the training parameters according to the performance prediction results.

[0022] In a specific implementation, the training parameters corresponding to each training optimization stage determined in the above step 102 can be obtained, and initial parameter values ​​can be set for each training parameter to obtain initial parameter combination values ​​of the training parameters; based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network, and the hardware resource constraints, the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters are determined. Here, a training performance simulation model can be pre-constructed, and the graph neural network corresponding to the initial parameter combination values ​​of the training parameters can be predicted through the training performance simulation model to obtain performance prediction results; then, based on the performance prediction results, an optimization algorithm is used in the parameter combination value space of the training parameters to search and obtain the optimal target parameter value. Here, the optimization algorithm is such as a genetic algorithm, a particle swarm optimization, and the like.

[0023] Considering the huge optional training parameter space, if the traditional manual training parameter tuning method is used, either a good optimization effect cannot be achieved and it will fall into the local optimum; or it will cost a lot of parameter tuning overhead to find the most suitable parameter strategy among many candidate parameters. More importantly, the relevant schemes can usually only be tuned for a certain performance indicator. For example, by increasing the storage overhead to cache some node features, the data transmission delay can be greatly reduced, and the optimal training can be achieved through careful adjustment of the cache ratio.

[0024] The optimization process in the embodiment of the present application uses four strategies to automatically optimize the code without manual intervention. The principles are as follows: According to the Pareto optimization theory, if a specific training parameter setting a performs no worse than a specific training parameter setting b in any training performance indicator, and a performs better than b in a certain training performance indicator, then a is called Pareto dominant over b, or a is better than b, and a strongly Pareto dominates b. If a training performance indicator corresponding to solution A is better than a training performance indicator corresponding to solution B, but another training performance indicator corresponding to solution A is worse than a training performance indicator corresponding to solution B, then solution A is said to be indistinguishable from solution B, also known as solution A can Pareto dominate solution B.

[0025] If in the parameter space, there is a training parameter setting , Each corresponding training performance indicator is superior to any other parameter setting, then it is called is the absolute optimal training parameter setting. Given a number of performance indicators, these performance indicators are set as the objective function. The purpose of parameter space exploration is to find all parameter settings that are not inferior to all other parameter settings, and form a Pareto frontier based on this. The above boundary will be output as the result of parameter space exploration.

[0026] Step 104: Based on the target parameter combination value of the training parameters, the original graph neural network is trained to obtain a target graph neural network.

[0027] In a specific implementation, the original graph neural network is trained based on the target parameter values ​​of the determined sampling parameters to obtain the target graph neural network for subsequent use and deployment.

[0028] In this way, by dividing the training process of the original graph neural network into at least one training optimization stage, determining the training parameters corresponding to each training optimization stage, and searching for the target parameter combination values ​​from the parameter combination value space of the training parameters based on the graph neural network training task, it is possible to achieve efficient optimization of the graph neural network training process and improve the adaptability of the graph neural network under diversified needs.

[0029] In a possible implementation, the graph neural network training task includes a training task scenario, a graph neural network type, a data set for executing training, and hardware resource configuration information; in the above step 101, the network parameters and hardware resource constraints of the original graph neural network are obtained by parsing the graph neural network training task, including: Determine the network parameters of the original graph neural network based on the graph neural network type and the data set for training. Obtain the training complexity of the original graph neural network by performing static analysis on the network parameters of the original graph neural network. Determine the hardware resource constraint conditions based on the training complexity of the original graph neural network, the training task scenario and the hardware resource configuration information.

[0030] In the present application embodiment, Figure 2 As shown in , the user input graph neural network training task includes task scenario, graph neural network type, data set for training and hardware resource configuration information (such as hardware computing platform); for the task scenario, perform task requirement analysis to specify the emphasis of the current training task on the three performance indicators of training accuracy, training speed and storage overhead; determine the network parameters of the original graph neural network according to the graph neural network type and the data set for training; perform model evaluation on the network parameters of the original graph neural network model, that is, statically analyze the computational complexity of the original graph neural network model and obtain the expected training load based on the computational complexity; perform graph data evaluation on the data set for training and carry out graph data evaluation from the perspective of graph analysis. Here, graph data evaluation can include node number statistics, neighborhood analysis, feature dimension statistics and clustering coefficient calculation, etc. The graph data evaluation results can reflect the training overhead to a certain extent; determine the training complexity according to the expected training load, graph data evaluation results, etc.; and then determine the hardware resource constraints according to the training complexity of the original graph neural network, the training task scenario and the hardware computing platform, where the hardware resource constraints can include available storage space, hardware computing power, etc.

[0031] In this way, through the quantitative analysis of the above items, on the one hand, it can be used as input data for performance prediction to evaluate training performance; on the other hand, it can be used as constraints and exploration targets for parameter space exploration. For example, for given training parameters, when the predicted expected storage overhead exceeds the maximum available storage capacity allowed by the hardware, it can be considered to exceed the parameter space exploration constraints.

[0032] In a possible implementation, the training optimization parameter stage includes a graph sampling optimization stage, a data transmission optimization stage, and a network model calculation optimization stage; the training parameters include sampling parameters, data cache parameters, and model training parameters; In the above-mentioned step 102, the training parameters corresponding to each of the training optimization stages are determined, including: determining the sampling parameters according to the graph sampling strategy corresponding to the graph sampling optimization stage; determining the data cache parameters according to the data transmission strategy corresponding to the data transmission optimization stage; determining the model training parameters according to the model calculation optimization strategy corresponding to the network model calculation optimization stage.

[0033] In specific implementation, the training process of the original graph neural network can be divided into a sampling optimization stage, a data transmission optimization stage, and a model calculation optimization stage. The sampling parameters are determined according to the graph sampling strategy corresponding to the graph sampling optimization stage; wherein the graph sampling strategy may include sampling algorithms such as FastGCN, GraphSAINT, and GraphSAGE, and the sampling parameters may include the number of sampling hops, sampling bias, batch data size, etc. According to the data transmission strategy corresponding to the data transmission optimization stage, the data cache parameters are determined, wherein the data transmission strategy includes a cache update strategy, and the corresponding data cache parameters include a cache ratio, etc.; according to the model calculation optimization strategy corresponding to the network model calculation optimization stage, the model training parameters are determined, and the model training parameters include the subgraph size of the batch data, the node feature vector dimension, etc.

[0034] Exemplarily, for the graph sampling optimization stage, although the existing sampling optimization strategies vary, the behavior of the sampler can generally be abstracted as expanding a subgraph from a given target vertex set. In an embodiment of the present application, some training vertices can be extracted from the original graph neural network, and the feature subgraph can be expanded through the training vertices. In this way, by uniformly defining the sampler, the vertex set is iteratively expanded with a certain probability or rule, and a feature subgraph is generated based on this. Among them, the neighborhood nodes of the feature node can be determined by the following formula:

[0035] in, For the l The feature subgraph output by the layer, As an indicator function, according to the probability function Decide whether to select a node Neighborhood Nodes in u .

[0036] For the data transmission optimization stage, the computing platform can be abstracted into the host side and the device side. The host side includes the CPU, memory, etc., and the device side includes the graphics card, video memory, etc. Regardless of the specific implementation of the transmission strategy, the training process of the graph neural network needs to ensure that the data required to perform the calculation is already on the device side. Here, the transmission strategy usually uses the idle memory resources on the device side as a cache to reduce redundant data transmission. Although the existing transmission strategies have slight differences in cache update strategies, they can be uniformly abstracted into the following steps: (1) The device cache is initialized based on the available storage resources on the device; Given a feature subgraph, the device cache compares each node in the feature subgraph with the pre-stored target nodes to determine which part of the feature subgraph has been cached, and the remaining part is transmitted from the host side through the data link between the host side and the device side, that is, determining the cache nodes and transmission nodes in the feature subgraph.

[0037] (2) After completing the transfer operation of all necessary data, training can be performed on small batches of data; (3) The device cache is updated according to the cache update strategy, and a portion of the transmission configuration settings are uniformly listed in the device cache. Specifically, different transmission strategies can be distinguished by appropriately configuring these settings.

[0038] For the network model calculation optimization phase, its time complexity can be uniformly analyzed based on aggregation and update functions. And the space complexity Based on complexity analysis and combined with the hardware resource configuration information of the device-side computing platform, the theoretical time overhead and storage overhead on the target device can be comprehensively obtained.

[0039] In a possible implementation, in the above step 103, based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network and the hardware resource constraints, determining the performance prediction result of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters includes: According to the network parameters of the original graph neural network and the initial parameter values ​​of the sampling parameters, training vertices are extracted from the original graph neural network to generate a feature subgraph; according to the subgraph scale of the feature subgraph, the initial parameter values ​​of the data cache parameters and the initial values ​​of the model training parameters, the initial parameter combination values ​​of the training parameters are determined; the network parameters of the original graph neural network, hardware resource constraints and the initial parameter combination values ​​of the training parameters are input into a pre-constructed training performance simulation model, and the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters are determined through the training performance simulation model.

[0040] The performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters determined by the training performance simulation model includes: Determine at least one performance estimate of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters through the training performance simulation model; and determine the performance prediction result of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters based on the at least one performance estimate.

[0041] The performance estimate includes device storage overhead, time overhead and training accuracy; the at least one performance estimate of the graph neural network corresponding to the initial parameter combination value of the training parameter determined by the training performance simulation model includes: The subgraph scale of the characteristic subgraph and the initial parameter combination values ​​of the training parameters are input into a pre-constructed storage gray box model, and the device storage overhead is determined through the storage model; the initial parameter combination values ​​of the training parameters and the hardware resource constraint conditions are input into a pre-constructed time gray box model, and the time overhead is determined through the time gray box model; the training accuracy is determined according to the subgraph scale and the average degree of the characteristic subgraph.

[0042] In an exemplary embodiment, in order to avoid the huge time overhead caused by the actual measurement of training parameters, a proxy model simulation method is used to achieve efficient and accurate training performance simulation. To this end, the embodiment of the present application proposes to use a performance predictor based on a "gray box model" for performance prediction. By combining complexity theory analysis, hardware computing power evaluation, and deep learning methods, the performance predictor has both the prediction accuracy of the "black box model" and the interpretability of the "white box model". To achieve the above goals, the training performance of the graph neural network is fully modeled from three aspects: training accuracy, training speed, and storage overhead. Taking the training storage overhead as an example, the storage gray box model is as follows: make Indicates the storage overhead on the device side, such as video memory overhead. It mainly consists of three parts: model running overhead , data cache overhead , other runtime overhead .

[0043] ; Among them, model operation overhead, data cache overhead, and other runtime overhead can be formalized as: ; ; ; Model running overhead , that is, the space complexity Proportional to the model parameter |Φ|.

[0044] Data Cache Overhead Cache ratio needs to be considered r ,Number of nodes in the whole graph| V |, graph feature dimension n attr , these three factors together determine the memory usage of the graph nodes in the GPU. The number of nodes can be expressed as , f cache It is a learnable function that will be handed over to the "black box model" for learning to obtain the final representation.

[0045] Other runtime overhead It mainly refers to the intermediate computational overhead of the model's forward propagation and backward propagation, which is mainly related to the size of the subgraphs sampled in each batch of data (mini-batch). The average size of the subgraph can be used express, is a learnable function.

[0046] Among them, the model parameter quantity |Φ| and the cache ratio can be determined according to the subgraph scale of the feature subgraph and the initial parameter combination value of the training parameter. r ,Number of nodes in the whole graph| V |,Graph feature dimensions n attr wait.

[0047] Exemplarily, the temporal grey box model is as follows: The time cost of training graph neural networks T , training accuracy Formally: ; ; ; ; ; ; Among them, n iter is the iteration round, t sample is the sampling time, t trans is the transmission time, t replace is the cache replacement time, t compute To calculate the time, r is the cache ratio (the ratio of the original full image), For the i The point set of the feature subgraph of batches, Device is the hardware parameter of the device. Host are the host hardware parameters, Hit is the cache hit rate, Nattr is the node feature vector dimension (feature vector is also called graph embedding information), B 0 How many nodes are expanded for the characteristic subgraph growth, Deg is the average degree of the graph (deg, including in-degree and out-degree), G For a given graph neural network.

[0048] Among them, the iteration round n can be determined according to the initial parameter combination value of the training parameters iter , Cache ratior , No. i The point set of the feature subgraph of the batch , node feature vector dimension Nattr 、How many nodes B are expanded by the characteristic subgraph growth 0 , the average degree of the graph Deg; determine the device-side hardware parameters based on hardware resource constraints Device , Host hardware parameters Host , cache hit rate Hit wait.

[0049] Here, there are many learnable functions or parameters, but due to the small number of existing real data sets, it is difficult to meet the needs of training learnable functions. The embodiment of the present application introduces a "gray box model" to collect the performance under different training parameters in known training tasks, learn the hidden rules, and complete the training of the proxy model. It can better realize the simulation of training accuracy, training speed, and storage overhead, and can improve the accuracy of performance prediction results.

[0050] In an exemplary embodiment, if Figure 3 As shown in , by parsing the graph neural network training task given by the user, immutable parameters such as hardware computing resources, graph data set information, and network model information are obtained, and they are input into a pre-built performance predictor; at the same time, by optimizing the training process of the graph neural network, configurable parameters such as batch data size, data cache size, and neighborhood sampling hop count are obtained, and an initialization search set is constructed according to the configurable parameters. The parameter space searcher generates candidate parameter input values ​​according to the initialization search set and enters the performance predictor; performance prediction is performed through the performance predictor to obtain performance prediction results; according to the performance prediction results, the target parameter value of the sampling parameter is searched in the parameter value space of the sampling parameter; and then based on the target parameter value of the sampling parameter, the original graph neural network is trained to obtain the target graph neural network.

[0051] In a possible implementation, the step of determining, based on the at least one performance estimate, a performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameter includes: According to the graph neural network training task, determine the weight corresponding to the at least one performance estimate value; based on the at least one performance estimate value and its corresponding weight, determine the performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters.

[0052] In an embodiment of the present application, the emphasis of the graph neural network training task given by the user on performance estimates such as training accuracy, training speed, and storage overhead can be analyzed to determine the weights corresponding to the device storage overhead, training speed, and training accuracy; based on the device storage overhead, training speed, training accuracy, and their corresponding weights, the performance prediction results of the graph neural network corresponding to the initial parameter values ​​of the sampling parameters can be determined.

[0053] The embodiment of the present application provides a graph neural network training method, which parses the graph neural network training task to obtain the original graph neural network and hardware resource information; extracts feature nodes from the original graph neural network according to preset sampling parameters to generate a feature subgraph, and determines the performance prediction result of the graph neural network corresponding to the initial parameter value of the sampling parameter according to the cache nodes and transmission nodes in the feature subgraph, and the hardware resource information; searches for the target parameter value of the sampling parameter in the parameter value space of the sampling parameter according to the performance prediction result; and trains the original graph neural network based on the target parameter value of the sampling parameter to obtain the target graph neural network. In this way, by dividing the training process of the original graph neural network into at least one training optimization stage, determining the training parameters corresponding to each training optimization stage, and searching for the target parameter combination value from the parameter combination value space of the training parameters based on the graph neural network training task, it is possible to achieve efficient optimization of the graph neural network training process, while improving the adaptability of the graph neural network under diversified demands. Figure 4 The structure diagram of the graph neural network training device provided in the embodiment of the present application is shown. The graph neural network training device can implement the following Figure 1 All or part of the contents of the illustrated embodiment, the neural network training device 400 includes: The task parsing module 410 is used to obtain the network parameters and hardware resource constraints of the original graph neural network by parsing the graph neural network training task; A parameter determination module 420, used to divide the training process of the original graph neural network into at least one training optimization stage, and determine the training parameters corresponding to each of the training optimization stages; A performance prediction module 430 is used to determine the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network and the hardware resource constraints, and search for the target parameter combination values ​​of the training parameters in the parameter combination value space of the training parameters according to the performance prediction results; The network training module 440 is used to train the original graph neural network based on the target parameter combination value of the training parameters to obtain a target graph neural network.

[0054] In a possible implementation, the graph neural network training task includes a training task scenario, a graph neural network type, a data set for executing training, and hardware resource configuration information; The task parsing module 410, when used to parse the graph neural network training task to obtain the network parameters and hardware resource constraints of the original graph neural network, is specifically used to: Determining network parameters of the original graph neural network according to the graph neural network type and the data set on which training is performed; By performing static analysis on the network parameters of the original graph neural network, the training complexity of the original graph neural network is obtained; Determine the hardware resource constraint conditions based on the training complexity, training task scenario and hardware resource information of the original graph neural network.

[0055] In a possible implementation, the training optimization parameter stage includes a graph sampling optimization stage, a data transmission optimization stage, and a network model calculation optimization stage; the training parameters include sampling parameters, data cache parameters, and model training parameters; The parameter determination module 420, when used to determine the training parameters corresponding to each of the training optimization stages, is specifically used to: Determining the sampling parameters according to the graph sampling strategy corresponding to the graph sampling optimization stage; Determining the data cache parameters according to the data transmission strategy corresponding to the data transmission optimization stage; The model training parameters are determined according to the model calculation optimization strategy corresponding to the network model calculation optimization stage.

[0056] In a possible implementation, the performance prediction module 430, when used to determine the performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters based on the initial parameter combination value of the training parameters, the network parameters of the original graph neural network, and the hardware resource constraint condition, is specifically used to: Extracting training vertices from the original graph neural network to generate a feature subgraph according to the network parameters of the original graph neural network and the initial parameter values ​​of the sampling parameters; Determining an initial parameter combination value of the training parameter according to the subgraph scale of the feature subgraph, the initial parameter value of the data cache parameter and the initial value of the model training parameter; The network parameters of the original graph neural network, the hardware resource constraints and the initial parameter combination values ​​of the training parameters are input into a pre-constructed training performance simulation model, and the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters are determined by the training performance simulation model.

[0057] In a possible implementation, the performance prediction module 430, when used to determine the performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameter through the training performance simulation model, is specifically used to: Determine at least one performance estimate of the graph neural network corresponding to the initial parameter combination value of the training parameters through the training performance simulation model; Determine a performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters based on the at least one performance estimate value.

[0058] In a possible implementation, the performance estimate includes device storage overhead, time overhead, and training accuracy; the performance prediction module 430, when used to determine at least one performance estimate of the graph neural network corresponding to the initial parameter combination value of the training parameter through the training performance simulation model, is specifically used to: Inputting the subgraph scale of the feature subgraph and the initial parameter combination value of the training parameter into a pre-built storage gray box model, and determining the device storage overhead through the storage model; Inputting the initial parameter combination value of the training parameter and the hardware resource constraint condition into a pre-built time gray box model, and determining the time cost through the time gray box model; The training accuracy is determined according to the subgraph scale and the average degree of the feature subgraph.

[0059] In a possible implementation, the performance prediction module 430, when used to determine the performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameter according to the at least one performance estimate, is specifically used to: Determining a weight corresponding to the at least one performance estimate value according to the graph neural network training task; Based on the at least one performance estimate and its corresponding weight, determine the performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters.

[0060] The embodiment of the present application provides a graph neural network training device, including a task parsing module, a parameter determination module, a performance prediction module and a network training module; the task parsing module obtains the network parameters and hardware resource constraints of the original graph neural network by parsing the graph neural network training task; the parameter determination module divides the training process of the original graph neural network into at least one training optimization stage, and determines the training parameters corresponding to each of the training optimization stages; the performance prediction module determines the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network and the hardware resource constraints, and searches for the target parameter combination values ​​of the training parameters in the parameter combination value space of the training parameters according to the performance prediction results; the network training module trains the original graph neural network based on the target parameter combination values ​​of the training parameters to obtain the target graph neural network. In this way, by dividing the training process of the original graph neural network into at least one training optimization stage, determining the training parameters corresponding to each training optimization stage, and searching for the target parameter combination values ​​from the parameter combination value space of the training parameters based on the graph neural network training task, it is possible to achieve efficient optimization of the graph neural network training process, while improving the adaptability of the graph neural network under diversified needs.

[0061] Figure 5 The hardware structure diagram of the electronic device provided by the embodiment of the present application is shown. Referring to the figure, at the hardware level, the electronic device 500 includes a processor 510, and optionally, includes an internal bus 520, a network interface 530, and a memory 540. Among them, the memory 540 may include a memory 541, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory 542 (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device 500 may also include hardware required for other services.

[0062] The processor 510, the network interface 530 and the memory may be interconnected via an internal bus 520, which may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0063] The memory 540 stores programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions. The memory 540 may include a memory 541 and a non-volatile memory 542 , and provide instructions and data to the processor 510 .

[0064] The processor 510 reads the corresponding computer program from the non-volatile memory 542 into the memory and then runs it, forming a device for locating the target user at the logical level. The processor 510 executes the program stored in the memory and specifically performs: Figure 1 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0065] The above application Figure 1 The method disclosed in the illustrated embodiment can be applied to the processor 510 or implemented by the processor 510. The processor 510 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 510. The above processor 510 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0066] The computer device can also execute the methods described in the foregoing method embodiments, and realize the functions and beneficial effects of the methods described in the foregoing method embodiments, which will not be repeated here.

[0067] Of course, in addition to software implementation, the electronic device 500 of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0068] The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable medium stores one or more programs, and when the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes Figure 1 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0069] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0070] Furthermore, an embodiment of the present application also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the following process is implemented: Figure 1 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of each method described in the foregoing method embodiments, which will not be described in detail here.

[0071] The embodiments of the present application can be applied to various electronic device collaboration or interconnection scenarios, including: collaboration and interconnection between mobile phones and laptops / tablet computers; collaboration and interconnection between mobile terminals and smart TVs / displays; collaboration and interconnection between mobile phones or tablet computers and in-car entertainment systems; collaboration and interconnection between mobile terminals and smart conference systems, etc., so as to meet the needs of users in various scenarios such as smart homes, smart offices, and smart travel.

[0072] In short, the above is only a preferred embodiment of the present application and does not limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0073] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0074] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0075] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0076] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A graph neural network training method, characterized in that: include: By parsing the graph neural network training task, the network parameters and hardware resource constraints of the original graph neural network are obtained; Dividing the training process of the original graph neural network into at least one training optimization stage, and determining the training parameters corresponding to each of the training optimization stages; Based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network and the hardware resource constraints, determine the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters, and search for the target parameter combination values ​​of the training parameters in the parameter combination value space of the training parameters according to the performance prediction results; Based on the target parameter combination values ​​of the training parameters, the original graph neural network is trained to obtain a target graph neural network.

2. The method according to claim 1, characterized in that The graph neural network training task includes the training task scenario, graph neural network type, data set for training execution, and hardware resource configuration information; The network parameters and hardware resource constraints of the original graph neural network are obtained by parsing the graph neural network training task, including: Determining network parameters of the original graph neural network according to the graph neural network type and the data set on which training is performed; By performing static analysis on the network parameters of the original graph neural network, the training complexity of the original graph neural network is obtained; Determine the hardware resource constraint conditions based on the training complexity, training task scenario and hardware resource information of the original graph neural network.

3. The method according to claim 1, characterized in that The training optimization parameter stage includes a graph sampling optimization stage, a data transmission optimization stage, and a network model calculation optimization stage; The training parameters include sampling parameters, data cache parameters and model training parameters; The determining of the training parameters corresponding to each of the training optimization stages includes: Determining the sampling parameters according to the graph sampling strategy corresponding to the graph sampling optimization stage; Determining the data cache parameters according to the data transmission strategy corresponding to the data transmission optimization stage; The model training parameters are determined according to the model calculation optimization strategy corresponding to the network model calculation optimization stage.

4. The method according to claim 3, characterized in that: The step of determining the performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters based on the initial parameter combination value of the training parameters, the network parameters of the original graph neural network, and the hardware resource constraint condition comprises: Extracting training vertices from the original graph neural network to generate a feature subgraph according to the network parameters of the original graph neural network and the initial parameter values ​​of the sampling parameters; Determining an initial parameter combination value of the training parameter according to the subgraph scale of the feature subgraph, the initial parameter value of the data cache parameter and the initial value of the model training parameter; The network parameters of the original graph neural network, the hardware resource constraints and the initial parameter combination values ​​of the training parameters are input into a pre-constructed training performance simulation model, and the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters are determined by the training performance simulation model.

5. The method according to claim 4, characterized in that The performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameter determined by the training performance simulation model includes: Determine at least one performance estimate of the graph neural network corresponding to the initial parameter combination value of the training parameters through the training performance simulation model; Based on the at least one performance estimate value, a performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters is determined.

6. The method according to claim 5, characterized in that The performance estimate includes device storage overhead, time overhead and training accuracy; The determining, by the training performance simulation model, at least one performance estimate of the graph neural network corresponding to the initial parameter combination value of the training parameter comprises: Inputting the subgraph scale of the feature subgraph and the initial parameter combination value of the training parameter into a pre-built storage gray box model, and determining the device storage overhead through the storage model; Inputting the initial parameter combination value of the training parameter and the hardware resource constraint condition into a pre-built time gray box model, and determining the time overhead through the time gray box model; The training accuracy is determined according to the subgraph scale and the average degree of the feature subgraph.

7. The method according to claim 5, characterized in that The step of determining, based on the at least one performance estimate, a performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameter comprises: Determining a weight corresponding to the at least one performance estimate value according to the graph neural network training task; Based on the at least one performance estimate and its corresponding weight, a performance prediction result of the graph neural network corresponding to the initial parameter combination value of the training parameters is determined.

8. A graph neural network training device, characterized in that: include: The task parsing module is used to parse the graph neural network training task to obtain the network parameters and hardware resource constraints of the original graph neural network; A parameter determination module, used to divide the training process of the original graph neural network into at least one training optimization stage, and determine the training parameters corresponding to each of the training optimization stages; A performance prediction module, used to determine the performance prediction results of the graph neural network corresponding to the initial parameter combination values ​​of the training parameters based on the initial parameter combination values ​​of the training parameters, the network parameters of the original graph neural network and hardware resource constraints, and search for the target parameter combination values ​​of the training parameters in the parameter combination value space of the training parameters according to the performance prediction results; The network training module is used to train the original graph neural network based on the target parameter combination value of the training parameters to obtain a target graph neural network.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images