A high performance computing cluster management system

Abstract

The present invention relates to the field of computing cluster management, and in particular to a high-performance computing cluster management system, comprising: a plurality of cluster modules, which are used to send and receive cluster data broadcast by a host in a computing cluster; a monitoring module, which is used to generate corresponding response data, a machine learning module, which is used to pre-process the response data; and an allocation module, which is used to adjust the computing cluster according to the learning results of the machine learning module. By setting a response time threshold and a response time monitoring model in the machine learning module, it is determined whether the computing cluster needs to be adjusted according to the response time of the computing cluster, and the response time monitoring model is used to visualize the response time between computing clusters, which effectively improves the accuracy of response time identification between computing clusters and improves the operation efficiency of the computing cluster, thereby reducing the complexity of computing cluster management.

Description

Technical Field

[0001] The present invention relates to the field of computing cluster management, and in particular to a high-performance computing cluster management system. Background Art

[0002] Computers are globally collaborative network devices used to transmit, accelerate, display, calculate, and store data information on the Internet infrastructure. In order to ensure the security and reliability of computer operation, each computer room is deployed with a computing cluster management system.

[0003] Currently, when the headquarters obtains data from each computing cluster management system, staff need to go to each data center to manually transmit the data from each computing cluster management system.

[0004] This method makes it impossible for the headquarters to automatically obtain data from each data center in real time, making it inefficient for the headquarters to obtain data from the computing cluster management system of each data center computer room.

[0005] Chinese patent application publication number: CN111373373B discloses a method for processing a state update request in a distributed data processing system having multiple processing nodes, including maintaining multiple counters, which include: a working counter indicating a current time interval; a replication counter indicating a time interval in which all requests associated with the time interval are replicated at several processing nodes in the multiple processing nodes; and a persistence counter indicating a time interval in the multiple time intervals in which all requests associated with the time interval are stored in a persistent storage device. These counters are used to manage the processing of these state update requests.

[0006] Chinese patent application number: CN111295643A discloses a method for managing data in a distributed data processing system including multiple processing nodes, including storing data units in data storage devices associated with multiple different durability levels. The method includes maintaining indicators, which include a first indicator associated with a first durability level and a second indicator associated with a second durability level. The first indicator is maintained to reflect such a time interval, at which time interval, all data unit sets associated with the time interval are stored at the first durability level. The second indicator is maintained to reflect such a time interval, at which time interval, all data unit sets associated with the time interval are stored at the second durability level. The first indicator and the second indicator are used to manage the processing of data units in the distributed data processing system.

[0007] However, the above method has the following problems: the computing cluster cannot be regulated in real time, and the computing cluster operation efficiency is low. Summary of the invention

[0008] To this end, the present invention provides a high-performance computing cluster management system to overcome the problems in the prior art that the computing cluster cannot be regulated in real time and the computing cluster operation efficiency is low.

[0009] To achieve the above object, the present invention provides a high performance computing cluster management system, comprising:

[0010] Several cluster modules, which are used to send and receive cluster data broadcast by hosts in the computing cluster;

[0011] A monitoring module, which is connected to the cluster module, is used to monitor the response time of receiving the cluster data in the cluster module and generate corresponding response data,

[0012] Wherein, the monitoring module is provided with a monitor for collecting and generating corresponding response data;

[0013] A machine learning module, which is connected to the monitoring module, is used to pre-process the response data, generate corresponding pre-processed data, and select several indicator features of the pre-processed data for learning;

[0014] The machine learning module is provided with a response time threshold, the preprocessed data is fitted according to the response data, and a number of data with a sampling rate of a standard sampling rate are generated, and corresponding indicator features are generated;

[0015] An allocation module is connected to the machine learning module and the cluster module, and is used to adjust the computing cluster according to the learning result of the machine learning module.

[0016] Furthermore, the cluster module is set as a first type of cluster and a second type of cluster, wherein:

[0017] For the first type of cluster, it is provided with a receiving host and several slaves for receiving cluster data broadcasted by other cluster hosts;

[0018] The second type of cluster is connected to the first type of cluster and is provided with a sending host and a plurality of slaves for collecting slave data, and the sending host broadcasts to the first type of cluster.

[0019] The second type of cluster is a cluster formed when the receiving host determines the sending host.

[0020] Furthermore, after receiving the online message of the sending host, the receiving host determines the sending host as a host in the second type of cluster.

[0021] Furthermore, the sending host is provided with an interval time, and a first message is broadcast to the receiving host every interval time. After the receiving host receives the first message, the cluster data stored in several slaves of the first type of cluster is updated to the cluster data carried by the first message.

[0022] The first message carries cluster data of the second type of cluster, and the cluster data includes cache usage information of the master and slaves in the second type of cluster.

[0023] Furthermore, the response time is the time from when the receiving host receives the first message to when the cluster data is updated. The monitor monitors the response time and generates corresponding response data, which is input into the machine learning module.

[0024] Furthermore, the machine learning module generates corresponding indicator features according to the response data, fits the response data, and generates a number of processed data with a sampling rate of the standard sampling rate, divides the obtained processed data according to the standard sampling rate, and generates the corresponding pre-processed data.

[0025] The standard sampling rate is related to the minimum recognition capability of the learning module.

[0026] Furthermore, the response time monitoring model is provided in the machine learning module, and the preprocessed data is passed through the response time monitoring model.

[0027] Furthermore, the response time threshold is set in the machine learning module. If the response time exceeds the response time threshold, the allocation module sends a second message to the receiving host.

[0028] Further, if the response time does not exceed the response time threshold, the allocation module does not adjust the cluster module.

[0029] Furthermore, when the receiving host receives the second message, it re-marks some slaves of the first type of cluster as slaves of the second type of cluster.

[0030] Compared with the prior art, the present invention utilizes the method of setting up several cluster modules, monitoring modules, machine learning modules, and allocation modules. By setting a response time threshold and a response time monitoring model in the machine learning module, it determines whether a computing cluster needs to be adjusted according to the response time of the computing cluster, and uses the response time monitoring model to visualize the response time between computing clusters. While effectively improving the accuracy of response time identification between computing clusters, it also improves the operating efficiency of the computing cluster, thereby further reducing the complexity of computing cluster management.

[0031] Furthermore, by setting up a host in the computing cluster, data can be collected and updated on the slaves through the host, thereby improving the operation efficiency of the computing cluster and reducing the complexity of computing cluster management.

[0032] Furthermore, by setting an interval time in the second computer to broadcast the first message to the first computer, the error in response time caused by different broadcast interval time is avoided, the accuracy of response time collection is improved, and thus the accuracy of machine learning results is improved.

[0033] Furthermore, by updating the cluster data stored in several slaves of the first cluster with the cluster data carried by the first message, the computing cluster data can be updated in real time, thereby improving the operation efficiency of the computing cluster and further reducing the complexity of computing cluster management.

[0034] Furthermore, by using the machine learning module to preprocess the response data to form corresponding preprocessed data, the recognition efficiency of the response data is improved, so that effective features of the response data can be quickly selected, thereby further improving the accuracy of the machine learning results.

[0035] Furthermore, by setting the corresponding response time monitoring model in the machine learning module and learning the preprocessed data, the recognition efficiency of the response data is improved, thereby further reducing the complexity of computing cluster management.

[0036] Furthermore, by setting a response time threshold in the machine learning module, it is possible to quickly distinguish whether the cluster module needs to be adjusted, thereby improving the operating efficiency of the computing cluster and further reducing the complexity of computing cluster management.

[0037] Furthermore, by adjusting the cluster module through the allocation module, the management of the computing cluster can be made simpler, thereby further reducing the complexity of the computing cluster management. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the structure of the high performance computing cluster management system of the present invention;

[0039] Figure 2 The figure is a flow chart of the operation of the response time monitoring model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0042] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0043] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0044] See also Figure 1 As shown, it is a schematic diagram of the structure of the high-performance computing cluster management system of the present invention, including:

[0045] Several cluster modules, which are used to send and receive cluster data broadcast by hosts in the computing cluster;

[0046] The monitoring module is connected to the cluster module to monitor the response time of receiving cluster data in the cluster module and generate corresponding response data.

[0047] Among them, the monitoring module is provided with a monitor to collect and generate corresponding response data;

[0048] A machine learning module, which is connected to the monitoring module, is used to preprocess the response data, generate corresponding preprocessed data, and select several indicator features of the preprocessed data for learning;

[0049] Among them, the machine learning module is provided with a response time threshold, the preprocessing data is fitted according to the response data, and a number of data with a standard sampling rate are generated, and corresponding indicator features are generated;

[0050] The allocation module is connected to the machine learning module and the cluster module, and is used to adjust the computing cluster according to the learning results of the machine learning module.

[0051] In a specific implementation, the response time of multiple computing clusters is obtained, and the average response time of the multiple computing clusters is calculated to serve as a response time threshold for measuring the length of the computing cluster response time.

[0052] By setting up several cluster modules, monitoring modules, machine learning modules, and allocation modules, and by setting the response time threshold and response time monitoring model in the machine learning module, it is determined whether the computing cluster needs to be adjusted according to the response time of the computing cluster, and the response time monitoring model is used to visualize the response time between computing clusters. This effectively improves the accuracy of response time identification between computing clusters and improves the operational efficiency of the computing cluster, thereby further reducing the complexity of computing cluster management.

[0053] Specifically, the cluster module is set as the first type of cluster and the second type of cluster, where:

[0054] For the first type of cluster, it is provided with a receiving host and several slaves for receiving cluster data broadcasted by other cluster hosts;

[0055] The second type of cluster is connected to the first type of cluster and is provided with a sending host and several slaves for collecting slave data, which is broadcasted by the sending host to the first type of cluster.

[0056] The second type of cluster is a cluster formed when the receiving host determines the sending host.

[0057] By setting up a host in a computing cluster, data can be collected and updated from the slaves through the host, thereby improving the operational efficiency of the computing cluster and reducing the complexity of computing cluster management.

[0058] Specifically, after receiving the online message of the sending host, the receiving host determines the sending host as a host in the second type of cluster.

[0059] Embodiment 1: After the receiving host receives the online message of the sending host, if the receiving host is a host in the first type of cluster, the receiving host determines a host in the second type of cluster, wherein the second type of cluster is a cluster formed after the sending host is added to the first type of cluster.

[0060] Specifically, the sending host is provided with an interval time t, and the first message is broadcast to the receiving host every interval time t. After receiving the first message, the receiving host updates the cluster data stored in several slaves of the first type of cluster to the cluster data carried by the first message.

[0061] The first message carries cluster data of the second type of cluster, and the cluster data includes cache usage information of the master and slave in the second type of cluster.

[0062] The optimal t setting is in the range of 0.4s-0.6s, which provides the best response effect between computing clusters. Any value of t can be selected within this range.

[0063] By setting the interval duration in the sending host to broadcast the first message to the receiving host, the error in response time caused by different broadcast interval durations is avoided, the accuracy of response time collection is improved, and thus the accuracy of machine learning results is improved.

[0064] Specifically, the response time is the time from when the receiving host receives the first message to when the cluster data is updated. The monitor monitors the response time and generates corresponding response data, which is input into the machine learning module.

[0065] By updating the cluster data stored in several slave machines of the first cluster with the cluster data carried by the first message, the computing cluster data can be updated in real time, thereby improving the operation efficiency of the computing cluster and further reducing the complexity of computing cluster management.

[0066] Specifically, the machine learning module generates corresponding indicator features according to the response data, fits the response data, and generates a number of processed data with a sampling rate of the standard sampling rate, and divides the obtained processed data according to the standard sampling rate to generate corresponding pre-processed data.

[0067] Among them, the standard sampling rate is related to the minimum recognition ability of the learning module.

[0068] In implementation, the sampling rate is generally set between 0.3s and 0.5s;

[0069] Preferably, setting the sampling rate to 0.4s has a better recognition effect on the response time, and for the system described in this application, the corresponding processing effect is optimal.

[0070] By using the machine learning module to preprocess the response data, corresponding preprocessed data is formed, which improves the recognition efficiency of the response data, so that effective features of the response data can be quickly selected, thereby further improving the accuracy of the machine learning results.

[0071] Specifically, a response time monitoring model is provided in the machine learning module, and the preprocessed data is passed through the response time monitoring model.

[0072] See also Figure 2 As shown, it is an operation flow chart of the response time monitoring model according to an embodiment of the present invention, and the preprocessed data passes through the response time monitoring model.

[0073] The response time monitoring model is composed of a series of convolutional layers, attention layers, pooling layers, and fully connected layers;

[0074] Among them, if the size of the convolution kernel in the convolution layer is 2, 3, 4, 5, and there are 128 convolution kernels of each size, then after the convolution layer, 4×128 one-dimensional vectors will be obtained. After passing through the pooling layer, 4x128 values ​​will be obtained. After splicing together, a 512×1 vector is finally obtained. The obtained vector is fully connected in the fully connected layer to obtain the final learning result.

[0075] The response time monitoring model is generated based on the response time database training, including:

[0076] The preprocessed data in the response time monitoring model is divided into a training data set and a test data set;

[0077] Insert an attention layer between the convolution layer and the pooling layer. The number of attention layers can be determined according to demand. All weights in the attention layer are randomly assigned to prepare for training.

[0078] The preprocessed data features obtained from the convolution layer are input into the attention layer. The convolution layer before the attention layer contains several convolution kernels. In the attention layer, the input features are randomly grouped, weighted averaged, and new features are obtained. The new features are input into the pooling layer for pooling.

[0079] In the specific implementation, the training process of the response time monitoring model is to iteratively optimize the training data set of the preprocessed data according to the response time database until the loss value converges to the preset value, and test the response time monitoring model through the test data set. When the test results meet the preset test results, the response time monitoring model parameters are saved.

[0080] By setting the corresponding response time monitoring model in the machine learning module and learning the preprocessed data, the recognition efficiency of the response data is improved, thereby further reducing the complexity of computing cluster management.

[0081] Specifically, a response time threshold is set in the machine learning module. If the response time exceeds the response time threshold, the allocation module sends a second message to the receiving host.

[0082] By setting the response time threshold in the machine learning module, it is possible to quickly distinguish whether the cluster module needs to be adjusted, thereby improving the operating efficiency of the computing cluster and further reducing the complexity of computing cluster management.

[0083] Specifically, if the response time does not exceed the response time threshold, the allocation module does not adjust the cluster module.

[0084] Specifically, when the receiving host receives the second message, it re-marks some slaves of the first type of cluster as slaves of the second type of cluster.

[0085] By adjusting the cluster module through the allocation module, the management of the computing cluster can be made easier, thereby further reducing the complexity of computing cluster management.

[0086] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A high performance computing cluster management system, characterized in that: include: Several cluster modules, which are used to send and receive cluster data broadcast by hosts in the computing cluster; A monitoring module, which is connected to the cluster module, is used to monitor the response time of receiving the cluster data in the cluster module and generate corresponding response data, Wherein, the monitoring module is provided with a monitor for collecting and generating corresponding response data; A machine learning module, which is connected to the monitoring module, is used to pre-process the response data, generate corresponding pre-processed data, and select several indicator features of the pre-processed data for learning; The machine learning module is provided with a response time threshold, the preprocessed data is fitted according to the response data, and a number of data with a sampling rate of a standard sampling rate are generated, and corresponding indicator features are generated; an allocation module, connected to the machine learning module and the cluster module, for adjusting the computing cluster according to the learning result of the machine learning module; The cluster module sets the computing clusters into a first type of cluster and a second type of cluster, wherein: For the first type of cluster, it is provided with a receiving host and several slaves for receiving cluster data broadcasted by other cluster hosts; The second type of cluster is connected to the first type of cluster and is provided with a sending host and a plurality of slaves for collecting slave data, and the sending host broadcasts to the first type of cluster. The second type of cluster is a cluster formed when the receiving host determines the sending host; A response time threshold is provided in the machine learning module, and if the response time exceeds the response time threshold, the allocation module sends a second message to the receiving host; When the receiving host receives the second message, it re-marks some slaves of the first type of cluster as slaves of the second type of cluster.

2. The high performance computing cluster management system according to claim 1, characterized in that: After receiving the online message of the sending host, the receiving host determines the sending host as a host in the second type of cluster.

3. The high performance computing cluster management system according to claim 2, characterized in that: The sending host is provided with an interval time, and broadcasts a first message to the receiving host every interval time. After receiving the first message, the receiving host updates the cluster data stored in several slaves of the first type of cluster with the cluster data carried by the first message; The first message carries cluster data of the second type of cluster, and the cluster data includes cache usage information of the master and slaves in the second type of cluster.

4. The high performance computing cluster management system according to claim 3, characterized in that: The response time is the time from when the receiving host receives the first message to when the cluster data is updated. The monitor monitors the response time and generates corresponding response data, which is input into the machine learning module.

5. The high performance computing cluster management system according to claim 4, characterized in that: The machine learning module generates corresponding indicator features according to the response data, fits the response data, and generates a number of processed data with a sampling rate of a standard sampling rate, divides the obtained processed data according to the standard sampling rate, and generates the corresponding pre-processed data, The standard sampling rate is related to the minimum recognition capability of the learning module.

6. The high performance computing cluster management system according to claim 5, characterized in that: The machine learning module is provided with a response time monitoring model for learning the preprocessed data.

7. The high performance computing cluster management system according to claim 6, characterized in that: If the response time does not exceed the response time threshold, the allocation module does not adjust the cluster module.

Images