Throughput prediction method, device, electronic device and readable medium

Abstract

The embodiments of the present invention provide a throughput prediction method, device, electronic device and readable medium. The method includes: obtaining a graph corresponding to a distributed file system; dividing the graph into at least two subgraphs; extracting feature information associated with throughput in the subgraphs respectively; merging the feature information of the subgraphs to obtain feature information associated with throughput of the graph; and predicting the throughput of the distributed file system based on the graph feature information. Thus, it is possible to divide the graph corresponding to the distributed file system into multiple subgraphs, extract features from the subgraphs respectively, and then merge the subgraph features to predict throughput. While achieving a more accurate throughput prediction, by dividing the subgraphs and extracting features respectively, the efficiency of feature extraction in the throughput prediction process is improved, and the pressure in the inference calculation process is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a throughput prediction method, a throughput prediction device, an electronic device and a computer-readable medium. Background Art

[0002] In the prior art, the application field of distributed file systems is very wide. They can be mainly used to solve the large-scale data storage, management and access problems that cannot be handled by single file systems. For example, in the fields of big data processing and analysis, cloud computing and virtualization environment, backup and disaster recovery, Internet of Things (IoT) and edge computing, distributed file systems play a very important role.

[0003] For a distributed file system, its performance indicators may include throughput, response time, storage efficiency, scalability, fault recovery capability, etc. Among them, throughput directly affects the ability of the distributed file system to process data. If the throughput of the distributed file system is less than the amount of data that the distributed file system needs to process, the distributed file system may not be able to run in a good state. Summary of the invention

[0004] The embodiment of the present invention provides a throughput prediction method, device, electronic device and computer-readable storage medium to accurately predict the throughput of a distributed file system and keep the distributed file system running in a good state.

[0005] An embodiment of the present invention discloses a method for predicting throughput, the method comprising:

[0006] Obtain a graph corresponding to a distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes;

[0007] dividing the graph into at least two subgraphs;

[0008] Extracting feature information associated with throughput from the subgraphs respectively;

[0009] Merging the feature information of the subgraphs to obtain feature information of the graph associated with throughput;

[0010] Based on the characteristic information of the graph, the throughput of the distributed file system is predicted.

[0011] Optionally, the step of obtaining a graph corresponding to the distributed file system includes:

[0012] Obtaining throughput association information of storage nodes in a distributed file system, connection relationships between the storage nodes, and transmission performance information between the storage nodes;

[0013] Based on the throughput association information of the storage nodes in the distributed file system, the connection relationship between the storage nodes, and the transmission performance information between the storage nodes, a graph corresponding to the distributed system is constructed; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent the storage nodes in the distributed file system, and the edges represent the connection relationship between the storage nodes; the graph nodes also record the throughput association information of the storage nodes, and the edges also record the transmission performance information between the storage nodes.

[0014] Optionally, the step of dividing the graph into at least two subgraphs comprises:

[0015] The graph is partitioned into at least two subgraphs based on the number of edges between the subgraphs.

[0016] Optionally, the step of respectively extracting feature information associated with throughput in the subgraphs includes:

[0017] The subgraphs are respectively input into a preset graph neural network to obtain feature information associated with throughput output by the graph neural network.

[0018] Optionally, the step of merging the feature information of the subgraphs to obtain feature information of the graph associated with throughput includes:

[0019] Synchronize the features corresponding to the same graph nodes in the feature information of different subgraphs to obtain the synchronized feature information;

[0020] The synchronously processed feature information of the subgraphs is merged to obtain feature information of the graph associated with the throughput.

[0021] Optionally, the step of synchronizing the features corresponding to the same graph nodes in the feature information of the different subgraphs. The step of obtaining the synchronized feature information comprises:

[0022] For any of the subgraphs, searching for a boundary node in the subgraph as a target synchronization node, wherein the number of edges of the boundary node in the subgraph is less than the number of edges of the boundary node in the graph;

[0023] Searching for at least two subgraphs containing the target synchronization node as target synchronization subgraphs;

[0024] The feature information of the target synchronization subgraph corresponding to the feature of the target synchronization node is synchronized to obtain the synchronized feature information.

[0025] Optionally, the step of predicting the throughput of the distributed file system based on the characteristic information of the graph includes:

[0026] The feature information of the graph is input into a preset convolutional neural network to obtain the throughput of the distributed file system predicted by the convolutional neural network.

[0027] The embodiment of the present invention further provides a throughput prediction device, the device comprising:

[0028] A graph acquisition module, used to acquire a graph corresponding to the distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes;

[0029] A subgraph division module, used for dividing the graph into at least two subgraphs;

[0030] A feature extraction module, used to extract feature information associated with throughput in the subgraphs respectively;

[0031] A feature merging module, used for merging the feature information of the subgraphs to obtain feature information of the graph associated with the throughput;

[0032] The throughput analysis module is used to predict the throughput of the distributed file system based on the characteristic information of the graph.

[0033] Optionally, the graph acquisition module includes:

[0034] A system information acquisition submodule, used to acquire throughput association information of storage nodes in a distributed file system, connection relationships between the storage nodes, and transmission performance information between the storage nodes;

[0035] A graph acquisition submodule is used to construct a graph corresponding to the distributed system based on the throughput association information of the storage nodes in the distributed file system, the connection relationship between the storage nodes, and the transmission performance information between the storage nodes; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent the storage nodes in the distributed file system, and the edges represent the connection relationship between the storage nodes; the graph nodes also record the throughput association information of the storage nodes, and the edges also record the transmission performance information between the storage nodes.

[0036] Optionally, the subgraph partitioning module includes:

[0037] The subgraph partitioning submodule is used to partition the graph into at least two subgraphs based on the number of edges between the subgraphs.

[0038] Optionally, the feature extraction module includes:

[0039] The feature extraction submodule is used to input the subgraphs into a preset graph neural network respectively to obtain feature information associated with the throughput output by the graph neural network.

[0040] Optionally, the feature merging module includes:

[0041] A feature synchronization submodule is used to synchronize features corresponding to the same graph node in feature information of different subgraphs to obtain synchronized feature information;

[0042] The feature merging submodule is used to merge the synchronously processed feature information of the subgraphs to obtain feature information of the graph associated with the throughput.

[0043] Optionally, the feature synchronization submodule includes:

[0044] A node search unit, configured to search, for any of the subgraphs, a boundary node in the subgraph as a target synchronization node, wherein the number of edges of the boundary node in the subgraph is less than the number of edges of the boundary node in the graph;

[0045] A subgraph searching unit, used for searching at least two subgraphs where the target synchronization node exists as target synchronization subgraphs;

[0046] The feature synchronization unit is used to synchronize the feature information of the target synchronization subgraph with the feature corresponding to the target synchronization node to obtain the synchronized feature information.

[0047] Optionally, the throughput analysis module includes:

[0048] The throughput analysis submodule is used to input the feature information of the graph into a preset convolutional neural network to obtain the throughput of the distributed file system predicted by the convolutional neural network.

[0049] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0050] The memory is used to store computer programs;

[0051] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0052] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0053] The embodiments of the present invention include the following advantages:

[0054] Through the throughput prediction method provided by the embodiment of the present invention, a graph corresponding to a distributed file system is obtained; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent the connection relationship between the storage nodes; the graph is divided into at least two subgraphs; feature information associated with throughput in the subgraphs is extracted respectively; the feature information of the subgraphs is merged to obtain feature information associated with throughput of the graph; and the throughput of the distributed file system is predicted based on the feature information of the graph. Thus, it is possible to divide the graph corresponding to the distributed file system into a plurality of subgraphs, extract features from the subgraphs respectively, and then merge the subgraph features to predict throughput. While achieving a more accurate throughput prediction, the efficiency of feature extraction in the throughput prediction process is improved by extracting features separately by dividing the subgraphs, and the pressure in the inference calculation process is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of a method for predicting throughput provided in an embodiment of the present invention;

[0056] Figure 2 is an overall flow chart of a throughput prediction method provided in an embodiment of the present invention;

[0057] Figure 3 is a schematic diagram of constructing a graph provided in an embodiment of the present invention;

[0058] Figure 4 is a schematic diagram of subgraph division provided in an embodiment of the present invention;

[0059] Figure 5 is a schematic diagram of a process of extracting features from a subgraph provided in an embodiment of the present invention;

[0060] Figure 6 is a schematic diagram of feature synchronization provided in an embodiment of the present invention;

[0061] Figure 7 is a schematic diagram of a process of predicting throughput using a convolutional neural network provided in an embodiment of the present invention;

[0062] Figure 8 is a structural block diagram of a throughput prediction device provided in an embodiment of the present invention;

[0063] Fig. 9 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0064] Fig.10 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Reference Figure 1 , shows a flowchart of a method for predicting throughput provided in an embodiment of the present invention, which may specifically include the following steps:

[0067] Step 101, obtaining a graph corresponding to a distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes;

[0068] Specifically, the distributed file system may include at least one storage node, and the storage nodes may be connected to each other in communication to transmit data between the storage nodes. Thus, the storage nodes may be used as graph nodes in a graph, and the connection relationships between the storage nodes may be used as edges in the graph to obtain a graph corresponding to the distributed file system.

[0069] Step 102, dividing the graph into at least two subgraphs;

[0070] In order to improve the reasoning efficiency of throughput prediction, a parallel reasoning method can be used to process the graph of the distributed file system. Thus, the graph can be divided into at least two subgraphs.

[0071] The subgraph may include at least one storage node and may also include connection relationships between storage nodes. There may be repeated storage nodes and / or repeated edges between subgraphs, which is not limited in the present invention.

[0072] Step 103, extracting characteristic information associated with throughput from the subgraphs respectively;

[0073] In a specific implementation, parallel reasoning may be used to perform feature extraction on the subgraphs respectively to obtain feature information associated with throughput. Specifically, data associated with throughput may be recorded in the graph nodes and edges in the subgraphs, and feature extraction may be performed on the data associated with throughput in the graph nodes and edges in the subgraphs to obtain feature information associated with throughput in the subgraphs.

[0074] Step 104, merging the characteristic information of the subgraphs to obtain characteristic information of the graph associated with throughput;

[0075] After the feature extraction of the subgraph is completed, the feature information of the subgraph can be merged again to obtain feature information related to throughput corresponding to the complete graph of the distributed file system.

[0076] Step 105: predict the throughput of the distributed file system based on the characteristic information of the graph.

[0077] When the characteristic information of the graph is obtained, the throughput of the distributed file system can be predicted based on the information associated with the throughput contained in the characteristic information of the graph to obtain the predicted throughput. Therefore, the amount of data that the distributed file system needs to process can be adjusted based on the predicted throughput, so that the distributed file system can run in a better state.

[0078] Through the throughput prediction method provided by the embodiment of the present invention, a graph corresponding to a distributed file system is obtained; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent the connection relationship between the storage nodes; the graph is divided into at least two subgraphs; feature information associated with throughput in the subgraphs is extracted respectively; the feature information of the subgraphs is merged to obtain feature information associated with throughput of the graph; and the throughput of the distributed file system is predicted based on the feature information of the graph. Thus, it is possible to divide the graph corresponding to the distributed file system into a plurality of subgraphs, extract features from the subgraphs respectively, and then merge the subgraph features to predict throughput. While achieving a more accurate throughput prediction, the efficiency of feature extraction in the throughput prediction process is improved by extracting features separately by dividing the subgraphs, and the pressure in the inference calculation process is reduced.

[0079] In one embodiment of the present invention, the step of obtaining a graph corresponding to the distributed file system includes:

[0080] S11, obtaining throughput association information of storage nodes in a distributed file system, connection relationships between the storage nodes, and transmission performance information between the storage nodes;

[0081] S12, based on the throughput association information of the storage nodes in the distributed file system, the connection relationship between the storage nodes, and the transmission performance information between the storage nodes, construct a graph corresponding to the distributed system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent the storage nodes in the distributed file system, and the edges represent the connection relationship between the storage nodes; the graph nodes also record the throughput association information of the storage nodes, and the edges also record the transmission performance information between the storage nodes.

[0082] In a specific implementation, in the process of constructing a graph corresponding to a distributed file system, the throughput correlation information of storage nodes in the distributed file system can be written into the graph. Thus, in the subsequent process of performing throughput prediction based on the graph, the throughput correlation information can be obtained from the graph to perform throughput prediction.

[0083] In the process of constructing the graph corresponding to the distributed file system, the transmission performance between the storage nodes in the distributed file system can also affect the overall data throughput of the distributed file system. Therefore, the transmission performance between the storage nodes in the distributed file system can also be written into the graph. Specifically, the transmission performance between the corresponding storage nodes can be written into the edges between the graph nodes in the graph.

[0084] Thus, the graph corresponding to the constructed distributed system may include several graph nodes and edges for connecting graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent the connection relationship between storage nodes; the graph nodes also record the throughput association information of the storage nodes, and the edges also record the transmission performance information between the storage nodes. The connection relationship may be a logical connection relationship or a physical connection relationship.

[0085] As a specific example of the present invention, the throughput-related information of the storage node recorded in the graph node may be the storage node memory, CPU occupancy, real-time load of the storage node, etc. The transmission performance information between the storage nodes recorded in the edge may be information such as bandwidth and communication frequency. In order to facilitate data processing, the graph nodes in the graph corresponding to the distributed system may also be numbered, and the information of the graph nodes and edges may be stored in feature matrices respectively.

[0086] In the specific implementation, after obtaining the graph node information and edge information, obviously abnormal data can be screened out to avoid introducing too much noise data during the training process. Then the graph node features and edge features are normalized separately. The node features include node memory, CPU occupancy, and real-time load. If normalization is not performed, the node memory data may be hundreds or thousands, while the CPU occupancy is a decimal not greater than 1. Such training results will be highly dependent on the node memory and almost ignore the CPU occupancy. Therefore, normalization is required in the preprocessing stage to de-dimensionalize the data.

[0087] In one embodiment of the present invention, the step of dividing the graph into at least two subgraphs includes:

[0088] S21, dividing the graph into at least two subgraphs based on the number of edges between the subgraphs.

[0089] Specifically, in the process of dividing the graph into at least two subgraphs, the subgraph division may be performed based on the number of edges between the subgraphs. In order to reduce the computing resource consumption of a single subgraph in the subsequent subgraph feature extraction process, the subgraph division may be performed based on the minimum number of edges between the subgraphs and the roughly same number of nodes between the subgraphs.

[0090] Specifically, in order to ensure that the dependency information of graph nodes, that is, the connection relationship between graph nodes and other graph nodes is not lost, redundant nodes can be introduced during the subgraph segmentation process, that is, the same graph nodes can exist between subgraphs.

[0091] In one embodiment of the present invention, the step of respectively extracting characteristic information associated with throughput in the subgraphs includes:

[0092] S31, input the subgraphs into a preset graph neural network respectively to obtain feature information associated with the throughput output by the graph neural network.

[0093] In a specific implementation, the subgraph can be processed by a graph neural network (GNN) to extract feature information associated with throughput.

[0094] Specifically, in the graph neural network model, each graph node is a multi-layer perceptron (MLP). Each multi-layer perceptron will update the features of the graph node based on the features of the graph node, the features of the adjacent graph nodes, and the features of the adjacent edges. This process is called feature aggregation. Generally speaking, it can be divided into two steps. The first step is to input the features of the graph nodes at both ends of the edge and the features of the edge into the edge model to complete the update of the edge features. The update formula is as follows:

[0095] e ij′=f(e ij ,v i ,v j )

[0096] Among them, e ij The eigenvector representing the edge, v i The feature vector of the graph node representing one end of the edge, v j The feature vector of the graph node representing the other end of the edge.

[0097] Then, for any graph node, the updated edge features and neighboring graph node features are input into the multi-layer perceptron corresponding to the graph node to update the features of the current graph node. The update formula is as follows:

[0098] v i ′ =f(v j ,e ij ′)

[0099] v j The feature vector representing the node in the neighbor graph, e ij ′ represents the updated feature vector of the edge.

[0100] In an embodiment of the present invention, the step of combining the characteristic information of the subgraphs to obtain the characteristic information of the graph associated with the throughput includes:

[0101] S41, synchronizing the features corresponding to the same graph nodes in the feature information of different subgraphs to obtain the synchronized feature information;

[0102] In the specific implementation, since there may be redundant nodes between subgraphs, that is, the same graph nodes may exist between subgraphs, and the same graph nodes may be missing edges in different subgraphs, in order to ensure that the graph nodes have complete feature information, the features corresponding to the same graph nodes in the feature information of different subgraphs can be synchronized to obtain the feature information of the graph associated with the throughput.

[0103] S42: Merge the synchronously processed feature information of the subgraphs to obtain feature information of the graph associated with throughput.

[0104] After the synchronization process is completed, the graph nodes can have complete feature information in the feature information of the subgraph. At this time, the synchronized feature information of the subgraph can be merged to obtain the feature information associated with the throughput of the complete graph.

[0105] In one embodiment of the present invention, the step of synchronizing the features corresponding to the same graph nodes in the feature information of the different subgraphs to obtain the synchronized feature information comprises:

[0106] S51, for any of the subgraphs, searching for a boundary node in the subgraph as a target synchronization node, wherein the number of edges of the boundary node in the subgraph is less than the number of edges of the boundary node in the graph;

[0107] In a specific implementation, for any subgraph, the boundary nodes in the subgraph can be searched, and the boundary nodes can be nodes located at the edge of the subgraph. Since the boundary nodes are located at the edge of the subgraph, their connection relationship with other graph nodes may be lost due to the segmentation of the graph. Therefore, for the subgraph, it can be considered that the boundary nodes may have lost edges, that is, the number of edges of the boundary nodes in the subgraph is less than the number of edges of the boundary nodes in the graph. Therefore, the boundary nodes can be used as the target synchronization nodes that need to be synchronized, and feature synchronization can be performed on the boundaries.

[0108] S52, searching for at least two subgraphs containing the target synchronization node as target synchronization subgraphs;

[0109] At least two subgraphs having target synchronization nodes may be searched and used as target synchronization subgraphs that need to be synchronized with each other.

[0110] S53, synchronizing the features corresponding to the target synchronization node in the feature information of the target synchronization subgraph to obtain synchronized feature information.

[0111] Subsequently, the features corresponding to the target synchronization node in the feature information of the synchronization target synchronization subgraph can be synchronized to other subgraphs based on MPI (Message Passing Interface, a standard communication protocol for parallel computing) communication to obtain the synchronized feature information.

[0112] In one embodiment of the present invention, the step of predicting the throughput of the distributed file system based on the characteristic information of the graph includes:

[0113] S61, inputting the feature information of the graph into a preset convolutional neural network to obtain the throughput of the distributed file system predicted by the convolutional neural network.

[0114] Afterwards, a convolutional neural network can be trained to output the throughput predicted by the distributed file system based on the feature information of the input graph. Thus, after obtaining the feature information of the graph, it can be input into the convolutional neural network to obtain the throughput of the distributed file system predicted by the convolutional neural network.

[0115] During the training process of the convolutional neural network, several training samples can be obtained. The training samples can include the feature information of the graph corresponding to the distributed file system and the actual throughput corresponding to the feature information of the graph. The feature information of the graph in the training sample can be input into the convolutional neural network to obtain the throughput output by the convolutional neural network. The mean square error between the throughput output by the convolutional neural network and the actual throughput is calculated to obtain the loss value. The loss is then back-propagated to adjust the model parameters, and the training is completed through multiple iterations.

[0116] As a specific example of the present invention, Figure 2 : is an overall flow chart of a throughput prediction method provided in an embodiment of the present invention. The overall steps of the throughput prediction method may include:

[0117] (1) obtaining a graph corresponding to the distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes;

[0118] (2) Divide the graph into four sub-graphs (sub-graphs) Figure 1 ~ son Figure 4 );

[0119] (3) extracting feature information associated with throughput from the subgraphs through a graph neural network (GNN);

[0120] (4) Synchronization and aggregation: merging the feature information of the subgraphs, synchronizing the feature information of each subgraph, and obtaining feature information of the graph related to throughput;

[0121] (5) Based on the feature information of the graph, a convolutional neural network (CNN) is used to predict the throughput of the distributed file system to obtain an inference result.

[0122] As a specific example of the present invention, Figure 3 It is a schematic diagram of constructing a graph provided in an embodiment of the present invention.

[0123] (1) Obtaining throughput association information of storage nodes in a distributed file system, connection relationships between the storage nodes, and transmission performance information between the storage nodes;

[0124] (2) Obtain a graph corresponding to the distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes; the graph nodes also record throughput association information of the storage nodes, and the edges also record transmission performance information between the storage nodes.

[0125] (3) The information of the graph nodes and edges is stored in feature matrices respectively.

[0126] As a specific example of the present invention, Figure 4 It is a schematic diagram of subgraph division provided in an embodiment of the present invention.

[0127] (1) Implement graph data segmentation based on the metis graph segmentation algorithm. When segmenting a subgraph, in order to prevent the dependency information of each graph node from being lost, it is necessary to segment the subgraph with redundancy. For example, graph nodes A, B, and C can be divided into a subgraph. In order to avoid the loss of dependency information of graph nodes, in addition to taking graph nodes A, B, and C and the connected edges out of the original graph, all of their direct neighbor nodes, such as graph node D, must also be taken out. This ensures that the information of nodes A, B, and C will not be lost, and the inference results will be more accurate.

[0128] As a specific example of the present invention, Figure 5 It is a schematic diagram of the process of extracting features from a subgraph provided in an embodiment of the present invention.

[0129] Each subgraph performs inference calculations separately. In the model of graph neural network (GNN), each graph node is a multi-layer perceptron (MLP). Each multi-layer perceptron will update the features of the graph node based on the features of the node, the features of the adjacent graph nodes, and the features of the adjacent edges to obtain the node features of the subgraph. This process is called feature aggregation. Generally speaking, it is divided into two steps. The first step is to input the node features at both ends of the edge and the features of the edge into the edge model to complete the update of the edge features. Then, the updated edge features and the features of the neighboring graph nodes are input into the model corresponding to the graph node to update the features of the current graph node.

[0130] As a specific example of the present invention, Figure 6 It is a schematic diagram of feature synchronization provided in an embodiment of the present invention.

[0131] After each subgraph is split, only the internal nodes have complete dependency information, and the updated features are complete, while the dependencies of the boundary nodes generated after splitting are incomplete. The feature update of such nodes needs to be done in other subgraphs that also contain the node and the node exists in other subgraphs. For example, Figure 4 The node D in the current subgraph has incomplete dependency information. To update the features of such nodes, it is necessary to communicate with the subgraph with complete dependency information to synchronize the features of the node. To this end, MPI communication is used to synchronize the node features of each subgraph. Only after synchronization is completed can the next round of iterative reasoning calculation be performed.

[0132] As a specific example of the present invention, Figure 7 It is a schematic diagram of the process of predicting throughput using a convolutional neural network provided in an embodiment of the present invention.

[0133] The features of each subgraph after reasoning can be summarized to obtain the complete node features, and the entire node features can be input into the convolutional neural network model for calculation to obtain the predicted throughput value.

[0134] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0135] Reference Figure 8 , shows a structural block diagram of a throughput prediction device provided in an embodiment of the present invention, which may specifically include the following modules:

[0136] A graph acquisition module 801 is used to acquire a graph corresponding to a distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes;

[0137] A subgraph division module 802, configured to divide the graph into at least two subgraphs;

[0138] A feature extraction module 803, used to extract feature information associated with throughput in the subgraphs respectively;

[0139] A feature merging module 804 is used to merge the feature information of the subgraphs to obtain feature information of the graph associated with the throughput;

[0140] The throughput analysis module 805 is used to predict the throughput of the distributed file system based on the characteristic information of the graph.

[0141] Optionally, the graph acquisition module includes:

[0142] A system information acquisition submodule, used to acquire throughput association information of storage nodes in a distributed file system, connection relationships between the storage nodes, and transmission performance information between the storage nodes;

[0143] A graph acquisition submodule is used to construct a graph corresponding to the distributed system based on the throughput association information of the storage nodes in the distributed file system, the connection relationship between the storage nodes, and the transmission performance information between the storage nodes; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent the storage nodes in the distributed file system, and the edges represent the connection relationship between the storage nodes; the graph nodes also record the throughput association information of the storage nodes, and the edges also record the transmission performance information between the storage nodes.

[0144] Optionally, the subgraph partitioning module includes:

[0145] The subgraph partitioning submodule is used to partition the graph into at least two subgraphs based on the number of edges between the subgraphs.

[0146] Optionally, the feature extraction module includes:

[0147] The feature extraction submodule is used to input the subgraphs into a preset graph neural network respectively to obtain feature information associated with the throughput output by the graph neural network.

[0148] Optionally, the feature merging module includes:

[0149] A feature synchronization submodule is used to synchronize features corresponding to the same graph node in feature information of different subgraphs to obtain synchronized feature information;

[0150] The feature merging submodule is used to merge the synchronously processed feature information of the subgraphs to obtain feature information of the graph associated with the throughput.

[0151] Optionally, the feature synchronization submodule includes:

[0152] A node search unit, configured to search, for any of the subgraphs, a boundary node in the subgraph as a target synchronization node, wherein the number of edges of the boundary node in the subgraph is less than the number of edges of the boundary node in the graph;

[0153] A subgraph searching unit, used for searching at least two subgraphs where the target synchronization node exists as target synchronization subgraphs;

[0154] The feature synchronization unit is used to synchronize the feature information of the target synchronization subgraph with the feature corresponding to the target synchronization node to obtain the synchronized feature information.

[0155] Optionally, the throughput analysis module includes:

[0156] The throughput analysis submodule is used to input the feature information of the graph into a preset convolutional neural network to obtain the throughput of the distributed file system predicted by the convolutional neural network.

[0157] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] In addition, an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0159] Memory 903, used for storing computer programs;

[0160] The processor 901 is used to execute the program stored in the memory 903 to implement the following steps:

[0161] Obtain a graph corresponding to a distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes;

[0162] dividing the graph into at least two subgraphs;

[0163] Extracting feature information associated with throughput from the subgraphs respectively;

[0164] Merging the feature information of the subgraphs to obtain feature information of the graph associated with throughput;

[0165] Based on the characteristic information of the graph, the throughput of the distributed file system is predicted.

[0166] Optionally, the step of obtaining a graph corresponding to the distributed file system includes:

[0167] Obtaining throughput association information of storage nodes in a distributed file system, connection relationships between the storage nodes, and transmission performance information between the storage nodes;

[0168] Based on the throughput association information of the storage nodes in the distributed file system, the connection relationship between the storage nodes, and the transmission performance information between the storage nodes, a graph corresponding to the distributed system is constructed; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent the storage nodes in the distributed file system, and the edges represent the connection relationship between the storage nodes; the graph nodes also record the throughput association information of the storage nodes, and the edges also record the transmission performance information between the storage nodes.

[0169] Optionally, the step of dividing the graph into at least two subgraphs comprises:

[0170] The graph is partitioned into at least two subgraphs based on the number of edges between the subgraphs.

[0171] Optionally, the step of respectively extracting feature information associated with throughput in the subgraphs includes:

[0172] The subgraphs are respectively input into a preset graph neural network to obtain feature information associated with throughput output by the graph neural network.

[0173] Optionally, the step of merging the feature information of the subgraphs to obtain feature information of the graph associated with throughput includes:

[0174] Synchronize the features corresponding to the same graph nodes in the feature information of different subgraphs to obtain the synchronized feature information;

[0175] The synchronously processed feature information of the subgraphs is merged to obtain feature information of the graph associated with the throughput.

[0176] Optionally, the step of synchronizing the features corresponding to the same graph nodes in the feature information of the different subgraphs. The step of obtaining the synchronized feature information comprises:

[0177] For any of the subgraphs, searching for a boundary node in the subgraph as a target synchronization node, wherein the number of edges of the boundary node in the subgraph is less than the number of edges of the boundary node in the graph;

[0178] Searching for at least two subgraphs containing the target synchronization node as target synchronization subgraphs;

[0179] The feature information of the target synchronization subgraph corresponding to the feature of the target synchronization node is synchronized to obtain the synchronized feature information.

[0180] Optionally, the step of predicting the throughput of the distributed file system based on the characteristic information of the graph includes:

[0181] The feature information of the graph is input into a preset convolutional neural network to obtain the throughput of the distributed file system predicted by the convolutional neural network.

[0182] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0183] The communication interface is used for communication between the above terminal and other devices.

[0184] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0185] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0186] like Fig.10 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1001 is further provided, in which instructions are stored. When the computer-readable storage medium 1001 is executed on a computer, the computer executes the throughput prediction method described in the above embodiment.

[0187] In yet another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer executes the throughput prediction method described in the above embodiment.

[0188] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0189] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0190] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0191] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A throughput prediction method, characterized in that: The method comprises: Obtain a graph corresponding to a distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes; dividing the graph into at least two subgraphs; Extracting feature information associated with throughput from the subgraphs respectively; Merging the feature information of the subgraphs to obtain feature information of the graph associated with throughput; Based on the characteristic information of the graph, the throughput of the distributed file system is predicted.

2. The method according to claim 1, characterized in that: The step of obtaining a graph corresponding to the distributed file system includes: Obtaining throughput association information of storage nodes in a distributed file system, connection relationships between the storage nodes, and transmission performance information between the storage nodes; Based on the throughput association information of the storage nodes in the distributed file system, the connection relationship between the storage nodes, and the transmission performance information between the storage nodes, a graph corresponding to the distributed system is constructed; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent the storage nodes in the distributed file system, and the edges represent the connection relationship between the storage nodes; the graph nodes also record the throughput association information of the storage nodes, and the edges also record the transmission performance information between the storage nodes.

3. The method according to claim 1, characterized in that The step of dividing the graph into at least two subgraphs comprises: The graph is partitioned into at least two subgraphs based on the number of edges between the subgraphs.

4. The method according to claim 1, characterized in that: The step of respectively extracting feature information associated with throughput in the subgraphs comprises: The subgraphs are respectively input into a preset graph neural network to obtain feature information associated with throughput output by the graph neural network.

5. The method according to claim 1, characterized in that The step of combining the characteristic information of the subgraphs to obtain characteristic information of the graph associated with throughput includes: Synchronizing features corresponding to the same graph nodes in feature information of different subgraphs to obtain synchronized feature information; The synchronously processed feature information of the subgraphs is merged to obtain feature information of the graph associated with the throughput.

6. The method according to claim 5, characterized in that The step of synchronizing the features corresponding to the same graph nodes in the feature information of different subgraphs to obtain the synchronized feature information comprises: For any of the subgraphs, searching for a boundary node in the subgraph as a target synchronization node, wherein the number of edges of the boundary node in the subgraph is less than the number of edges of the boundary node in the graph; Searching for at least two subgraphs containing the target synchronization node as target synchronization subgraphs; The feature information of the target synchronization subgraph corresponding to the feature of the target synchronization node is synchronized to obtain the synchronized feature information.

7. The method according to claim 1, characterized in that The step of predicting the throughput of the distributed file system based on the characteristic information of the graph includes: The feature information of the graph is input into a preset convolutional neural network to obtain the throughput of the distributed file system predicted by the convolutional neural network.

8. A throughput prediction device, characterized in that: The device comprises: A graph acquisition module, used to acquire a graph corresponding to the distributed file system; the graph includes a plurality of graph nodes and edges for connecting the graph nodes; the graph nodes represent storage nodes in the distributed file system, and the edges represent connection relationships between the storage nodes; A subgraph division module, used for dividing the graph into at least two subgraphs; A feature extraction module, used to extract feature information associated with throughput in the subgraphs respectively; A feature merging module, used for merging the feature information of the subgraphs to obtain feature information of the graph associated with the throughput; The throughput analysis module is used to predict the throughput of the distributed file system based on the characteristic information of the graph.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images