Cluster resource monitoring method and device, equipment, medium and product

Abstract

The invention discloses a cluster resource monitoring method and device, equipment, a medium and a product. The method comprises the following steps: inputting a historical resource time sequence set of a cluster system into a pre-trained resource monitoring model; wherein the historical resource time sequence set comprises resource use data of a historical time period, and the resource monitoring model comprises a linear time sequence network, a nonlinear time sequence network and a resource output network; inputting the historical resource time sequence set into a linear time sequence network to obtain an output first resource use result of the cluster system in a future time period; inputting the historical resource time sequence set into a nonlinear time sequence network to obtain an output second resource use result of the cluster system in a future time period; and inputting the first resource use result and the second resource use result into the resource output network to obtain an output target resource use result of the cluster system in the future time period, thereby realizing accurate prediction of the use condition of the cluster resources.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a cluster resource monitoring method, device, equipment, medium and product. Background Art

[0002] In today's digital age, cluster systems are widely used in many fields such as data processing and cloud computing. With the continuous growth of business volume and the increasing complexity of application scenarios, it is becoming increasingly critical to accurately grasp the usage of cluster resources.

[0003] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art:

[0004] At present, cluster systems rely on monitoring tools for real-time monitoring and early warning of resources, which makes it difficult to allocate resources in a timely manner to meet changing business needs. It is easy to waste cluster resources or lack cluster resources, affecting the normal operation of business. Summary of the invention

[0005] The embodiments of the present invention provide a cluster resource monitoring method, device, equipment, medium and product to solve the problem that the usage of cluster resources cannot be accurately predicted, provide data support for the timely and balanced allocation of cluster resources, and thus ensure the stable operation of the cluster system.

[0006] According to an embodiment of the present invention, a cluster resource monitoring method is provided, the method comprising:

[0007] Inputting the historical resource time series set of the cluster system into a pre-trained resource monitoring model; wherein the historical resource time series set includes resource usage data of a historical period, and the resource monitoring model includes a linear time series network, a nonlinear time series network and a resource output network;

[0008] Inputting the historical resource time series set into the linear time series network to obtain the output of the first resource usage result of the cluster system in the future period;

[0009] Inputting the historical resource time series set into the nonlinear time series network to obtain output a second resource usage result of the cluster system in a future time period;

[0010] The first resource usage result and the second resource usage result are input into the resource output network to obtain the output target resource usage result of the cluster system in the future period.

[0011] According to another embodiment of the present invention, a cluster resource monitoring device is provided, the device comprising:

[0012] A historical resource time series set input module, used to input the historical resource time series set of the cluster system into a pre-trained resource monitoring model; wherein the historical resource time series set contains resource usage data of a historical period, and the resource monitoring model includes a linear time series network, a nonlinear time series network and a resource output network;

[0013] A first resource usage result output module, used for inputting the historical resource time series set into the linear time series network to obtain an output of a first resource usage result of the cluster system in a future period;

[0014] A second resource usage result output module, used for inputting the historical resource time series set into the nonlinear time series network to obtain an output second resource usage result of the cluster system in a future time period;

[0015] The target resource usage result output module is used to input the first resource usage result and the second resource usage result into the resource output network to obtain the output target resource usage result of the cluster system in the future period.

[0016] According to another embodiment of the present invention, there is provided an electronic device, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the cluster resource monitoring method described in any embodiment of the present invention.

[0020] According to another embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the cluster resource monitoring method described in any embodiment of the present invention when executed.

[0021] According to another embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method for monitoring cluster resources according to any embodiment of the present invention is implemented.

[0022] An embodiment of the above invention has at least the following advantages or beneficial effects:

[0023] By setting a linear timing network, a nonlinear network and a resource output network in the resource monitoring model, the historical resource timing set is input into the linear timing network to obtain the output of the first resource usage result, the historical resource timing set is input into the nonlinear timing network to obtain the output of the second resource usage result, the first resource usage result and the second resource usage result are input into the resource output network to obtain the output target resource usage result of the cluster system in the future period, so that the resource monitoring model has the ability to express multi-scale features, improves the generalization ability of the resource prediction model, solves the problem that the usage of cluster resources cannot be accurately predicted, and provides data support for the timely and balanced allocation of cluster resources, thereby ensuring the stable operation of the cluster system.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flow chart of a cluster resource monitoring method provided by an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of the architecture of a resource monitoring model provided by an embodiment of the present invention;

[0028] Figure 3 A flowchart of another cluster resource monitoring method provided by an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a cluster resource monitoring device provided by an embodiment of the present invention;

[0030] Figure 5 The present invention is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", "initial", "reference", "benchmark", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 This is a flow chart of a cluster resource monitoring method provided by an embodiment of the present invention. This embodiment is applicable to monitoring and predicting the resource usage of a cluster system in a future period. The method can be executed by a cluster resource monitoring device. The cluster resource monitoring device can be implemented in the form of hardware and / or software. The cluster resource monitoring device can be configured in a terminal device. Figure 1 As shown, the method includes:

[0034] S110: Input the historical resource time series set of the cluster system into a pre-trained resource monitoring model.

[0035] Specifically, a cluster system refers to a computing architecture that connects multiple independent terminal devices (usually called nodes) through a high-speed network to form an organic whole to collaboratively complete specific tasks. From a hardware perspective, these nodes can have different configurations, including differences in processors, memory, storage, etc., but they all follow a unified communication protocol and management mechanism to interact. From a software perspective, cluster systems are usually equipped with dedicated cluster management software, which is responsible for key functions such as node resource allocation, task scheduling, fault monitoring and recovery. For large-scale data processing tasks, complex scientific computing projects, and business application scenarios that require high availability and high performance, cluster systems can give full play to the computing resource advantages of each node, and significantly improve computing efficiency and overall system performance by decomposing and assigning tasks to multiple nodes for parallel execution. At the same time, redundant design ensures uninterrupted business operation and effectively responds to emergencies such as node failures.

[0036] Exemplarily, types of cluster systems include, but are not limited to, high performance computing clusters, load balancing clusters, high availability clusters, etc. Cluster systems are not limited here, and can be customized according to actual needs.

[0037] In this embodiment, the historical resource time series set includes resource usage data of the historical time period. Specifically, the historical time period refers to the time period before the time period to which the current moment belongs. In the historical resource time series set, the time intervals between each two adjacent historical time periods may be the same or different. Exemplarily, the time length of the historical time period is 1 day or 12 hours, but is not limited to the example case.

[0038] For example, assuming that the length of the historical period is 1 day and the current time period is November 20, the historical resource time series set includes resource usage data corresponding to the five historical days of November 15, November 16, November 17, November 18 and November 19, and may also include resource usage data corresponding to the five historical days of November 14, November 16, November 17, November 18 and November 19.

[0039] Specifically, the resource usage data represents resource usage of the cluster system, and includes sampling data corresponding to at least one resource usage parameter.

[0040] Exemplarily, the resource usage data includes at least one resource usage parameter such as PVC (Persistent Volume Claim) usage, message traffic, node status, network connection status, node CPU (Central Processing Unit) usage, node memory occupancy rate and disk input / output usage rate, but is not limited to the example scenario. Among them, PVC usage rate refers to the ratio of PVC storage capacity allocated to applications or containers to the total storage capacity of PVC applied for, which reflects the utilization degree of storage resources in the cluster system in terms of persistent storage. Message traffic refers to the number of messages or message data volume transmitted through the message queue per unit time, reflecting the frequency of information interaction and data transmission scale between different components in the cluster system. Node status describes the operation status of each node in the cluster system, usually including status information such as whether it is online, whether it is processing tasks, whether it is idle and whether it is faulty. Network connection status refers to the status of data communication links between nodes in the cluster and between nodes and external networks, usually including connection status, bandwidth capacity, network delay, packet loss rate and bandwidth usage, etc.

[0041] In this embodiment, the resource monitoring model includes a linear timing network, a nonlinear timing network and a resource output network.

[0042] S120: Input the historical resource time series set into the linear time series network to obtain the output of the first resource usage result of the cluster system in the future period.

[0043] Specifically, the linear time series network refers to a neural network that has the ability to process time series data and realizes linear feature expression.

[0044] In a specific embodiment, exemplarily, the linear time series network is a linear regression neural network, a recurrent neural network, a linear time-invariant neural network or a linear state space model, etc., but is not limited to the example case.

[0045] In another specific embodiment, the linear timing network includes a first feature module, an additive attention module, a first long short-term memory module and a first output module; wherein the first feature module is used to extract features from the historical resource timing set to obtain a first timing feature, the additive attention module is used to determine the additive timing feature based on the first timing feature, the first long short-term memory module is used to determine the first long and short-term feature based on the additive timing feature, and the first output module is used to output the first resource usage result of the cluster system in the future period based on the first long and short-term feature.

[0046] Exemplarily, the first feature module is a convolutional neural network, and the first output module is a fully connected layer. The network architecture of the first feature module and the first output module is not limited here.

[0047] Specifically, the first time series feature includes the first parameter representation of each resource usage parameter in each historical period, and the additive attention module is specifically used to map each first parameter representation and query vector in the input first time series feature to a low-dimensional feature space through a linear transformation to obtain a low-dimensional parameter feature and a low-dimensional query vector. Each low-dimensional parameter feature is added to the low-dimensional query vector, and each addition result is processed by a nonlinear activation function and then dot-producted with the weight vector to obtain an attention score. All attention scores are normalized using a softmax function to obtain multiple attention weights. Each second parameter representation is weighted and summed with its corresponding attention weight to obtain an additive time series feature.

[0048] Exemplarily, the first long short-term memory module is a standard LSTM (Long Short-Term Memory), a bidirectional LSTM or a multi-layer LSTM, but is not limited to the example case.

[0049] S130: Input the historical resource time series set into the nonlinear time series network to obtain the output of the second resource usage result of the cluster system in the future period.

[0050] Specifically, the nonlinear time series network refers to a neural network that has the ability to process time series data and realizes nonlinear feature expression.

[0051] In a specific embodiment, exemplarily, the nonlinear time series network is a convolutional neural network, a recursive neural network, an echo state network, etc., but is not limited to the example case.

[0052] In another specific embodiment, the nonlinear time series network includes a second feature module, a multiplicative attention module, a second long short-term memory module and a second output module; wherein the second feature module is used to extract features from the historical resource time series set to obtain a second time series feature, the multiplicative attention module is used to determine the multiplicative time series feature based on the second time series feature, the second long short-term memory module is used to determine the second long and short-term feature based on the multiplicative time series feature, and the second output module is used to output the second resource usage result of the cluster system in the future time period based on the second long and short-term feature.

[0053] Exemplarily, the second feature module is a convolutional neural network, and the second output module is a fully connected layer. The network architecture of the second feature module and the second output module is not limited here.

[0054] Specifically, the second time series feature includes a second parameter representation of each resource usage parameter in each historical period, and a multiplicative attention module, which is specifically used to convert each second parameter representation in the input second time series feature into a key vector, perform dot product processing on the query vector and each key vector to obtain an attention score, and use a softmax function to normalize all attention scores to obtain multiple attention weights. Each second parameter representation is weighted and summed with its corresponding attention weight to obtain a multiplicative time series feature.

[0055] Exemplarily, the second long short-term memory module is a standard LSTM (Long Short-Term Memory), a bidirectional LSTM or a multi-layer LSTM, but is not limited to the example case.

[0056] Specifically, the future time period refers to a time period after the time period to which the current moment belongs. The time length of the future time period may be the same as or different from the time length of the historical time period.

[0057] In a specific embodiment, the number of future time periods is one or more. Exemplarily, the future time period is the next 12 hours and / or the next 7 days, but is not limited to the example scenario.

[0058] Specifically, the first resource usage result and the second resource usage result include prediction trend data corresponding to at least one resource usage parameter, where the resource usage parameter may be a resource usage parameter in a historical resource time series set or may not be a resource usage parameter in a historical resource time series set.

[0059] In a specific embodiment, the first resource usage result and the second resource usage result are predicted trend data corresponding to the PVC usage rate.

[0060] S140: Input the first resource usage result and the second resource usage result into a resource output network to obtain an output target resource usage result of the cluster system in a future period.

[0061] Specifically, the network algorithm in the resource output network is a weighted sum algorithm. Exemplarily, the target resource usage result y is expressed as: y=w1×y1+w2×y2, where w1 and w2 represent learnable model parameters in the resource output network, representing the weight coefficients corresponding to the first resource usage result and the second resource usage result, y1 represents the first resource usage result, and y2 represents the second resource usage result.

[0062] Figure 2A schematic diagram of the architecture of a resource monitoring model provided by an embodiment of the present invention. Specifically, [x1, x2, ..., xn] represents a historical resource time series set, the convolution module, additive attention module, bidirectional LSTM and fully connected layer on the left constitute a linear time series network, the convolution module, multiplicative attention module, bidirectional LSTM and fully connected layer on the right constitute a nonlinear time series network, and "y" represents the target resource usage result.

[0063] On the basis of the above embodiments, specifically, the method also includes: inputting the training resource timing set of the cluster system into an untrained resource monitoring model to obtain an output of predicted resource usage results of the cluster system in a future time period; adjusting the model parameters of the untrained resource monitoring model according to the predicted resource usage results and the actual resource usage results until the training end conditions are met, thereby obtaining a trained resource monitoring model.

[0064] On the basis of the above embodiments, specifically, the method also includes: determining the model performance value of each trained resource monitoring model according to the verification resource timing set of the cluster system and at least two trained resource monitoring models, and screening at least two trained resource monitoring models according to the at least two model performance values ​​to obtain the final trained resource monitoring model.

[0065] Specifically, different untrained resource monitoring models have different hyperparameters, such as hyperparameters including but not limited to learning rate, number of hidden layer units, batch size, etc.

[0066] Exemplarily, the model performance value includes at least one of mean square error, mean absolute error, square root error and coefficient of determination, but is not limited to the example scenario.

[0067] The benefit of this setting is that it ensures the generalization capability of the resource monitoring model, thereby further improving the prediction accuracy of cluster resource usage.

[0068] The technical solution of this embodiment is to set a linear timing network, a nonlinear network and a resource output network in a resource monitoring model, input the historical resource timing set into the linear timing network to obtain the output of the first resource usage result, input the historical resource timing set into the nonlinear timing network to obtain the output of the second resource usage result, input the first resource usage result and the second resource usage result into the resource output network to obtain the output target resource usage result of the cluster system in the future period, so that the resource monitoring model has the ability to express multi-scale features, improves the generalization ability of the resource prediction model, solves the problem that the usage of cluster resources cannot be accurately predicted, provides data support for the timely allocation and balanced allocation of cluster resources, and thus ensures the stable operation of the cluster system.

[0069] Figure 3 This is a flow chart of another cluster resource monitoring method provided by an embodiment of the present invention. This embodiment further refines the "historical resource time series set" in the above embodiment. Figure 3 As shown, the method includes:

[0070] S210: Take the reference time period closest to the current time in the initial resource timing set of the cluster system as the reference time period.

[0071] In this embodiment, the initial resource time series set includes resource usage data corresponding to at least two reference time periods, wherein the reference time period refers to a time period before the time period to which the current moment belongs.

[0072] In a specific embodiment, the method further includes: performing preprocessing and / or feature engineering on the original resource time series set of the cluster system to obtain an initial resource time series set, wherein the original resource time series set includes resource usage data corresponding to at least two reference time periods.

[0073] Exemplarily, preprocessing includes but is not limited to at least one of data cleaning, data formatting, data standardization and filling in missing values, wherein data cleaning is used to remove noise values ​​and outliers in the original resource time series set, and data formatting is used to unify the data format corresponding to the same resource usage parameters in the original resource time series set.

[0074] Specifically, feature engineering refers to a series of processes to extract, transform, and select feature parameters that contribute to the resource monitoring model from the original resource time series set. Feature extraction refers to the use of a specified algorithm to extract new features that can represent the essential characteristics of the data from the original resource time series set, such as converting text in resource usage data into a numerical vector. Feature transformation includes but is not limited to standardization and normalization, and feature selection refers to the screening of at least two resource usage parameters corresponding to the original resource time series set. Feature selection methods include but are not limited to filtering, wrapping, and embedded methods.

[0075] The advantage of this setting is that preprocessing reduces the impact of dimensional differences on the resource monitoring model, and feature engineering processing avoids overfitting of the resource monitoring model, further improving the prediction accuracy of cluster resource usage.

[0076] For example, assuming that the current time is 11:00 on November 20, the initial resource time series set includes resource usage data corresponding to five historical days, namely November 15, November 16, November 17, November 18 and November 19, then the benchmark period is November 19.

[0077] S220: Determine resource similarities according to the resource usage data of the benchmark period and the resource usage data corresponding to each reference period.

[0078] In this embodiment, the reference time period corresponds to the resource similarity one by one.

[0079] In a specific embodiment, resource similarity is determined based on resource usage data of the benchmark period and resource usage data corresponding to each reference period, including: for each reference period, determining resource similarity based on resource usage data of the benchmark period and resource usage data corresponding to the reference period.

[0080] In another specific embodiment, resource similarities are determined based on resource usage data of a benchmark time period and resource usage data corresponding to each reference time period, including: for each reference time period, the resource usage data of the benchmark time period and the resource usage data of the reference time period are divided according to the division time interval to obtain a benchmark segmented resource data set and a reference segmented resource data set; based on the benchmark segmented resource data set and the reference segmented resource data set, at least two segment similarities are determined, and the statistical values ​​corresponding to the at least two segment similarities are used as the resource similarity of the reference time period.

[0081] Exemplarily, the division time interval may be 1 hour or 0.5 hour, and the statistical value may be an average value, a maximum value, a minimum value, a median value, etc., but is not limited to the above example scenarios.

[0082] Specifically, the benchmark segmented resource data set includes segmented resource usage data corresponding to at least two benchmark sub-periods, and the reference segmented resource data set includes segmented resource usage data corresponding to at least two historical sub-periods. Exemplarily, the benchmark segmented resource data set includes sampling data corresponding to PVC usage rates at 8:00-8:30, 8:30-9:00, 9:00-9:30, and 9:30-10:00 on November 19, and the reference segmented resource data set includes sampling data corresponding to PVC usage rates at 8:00-8:30, 8:30-9:00, 9:00-9:30, and 9:30-10:00 on November 17.

[0083] Based on the above embodiments, illustratively, the similarity algorithm includes but is not limited to the Euclidean distance algorithm, the Manhattan distance algorithm, the cosine similarity algorithm, the Pearson correlation coefficient, the Jaccard similarity coefficient or the Hamming distance algorithm, etc. The similarity algorithm used is not limited here, and can be customized according to actual needs.

[0084] The benefit of segmenting resource usage data is that the similarity between resource usage data for the entire period may mask local differences. When the amount of data is large, similarity distortion may occur. However, segmented similarity can highlight the local differences between resource usage data in each segment and reduce the complexity of similarity calculation, thereby ensuring the reliability of the historical resource time series set.

[0085] S230: According to the resource similarity of each reference period, the initial resource time series set is screened to obtain a historical resource time series set.

[0086] In a specific embodiment, the initial resource timing set is screened according to the resource similarity of each reference time period to obtain the historical resource timing set, including: deleting the resource usage data of the reference time period corresponding to the resource similarity less than the similarity threshold from the initial resource timing set to obtain the historical resource timing set.

[0087] In another specific embodiment, the initial resource timing set is screened according to the resource similarity of each reference time period to obtain a historical resource timing set, including: according to the resource similarity of each reference time period, the resource usage data in the initial resource timing set is sorted to obtain a resource sorting set, and the historical resource timing set is determined based on the screening conditions and the resource sorting set.

[0088] Specifically, the filtering condition is the filtering ratio or the filtering quantity. When the sorting is in ascending order, the resource sorting set is cut from back to front according to the filtering condition to obtain the historical resource time series set. When the sorting is in descending order, the resource sorting set is cut from front to back according to the filtering condition to obtain the historical resource time series set.

[0089] In another specific embodiment, the initial resource timing set is screened according to the resource similarity of each reference time period to obtain the historical resource timing set, including: according to the similarity threshold and the resource similarity of each reference time period, the initial resource timing set is screened to obtain the reference resource timing set; if the data volume of the reference resource timing set is greater than the quantity threshold, the reference resource timing set is time-sorted, and the historical resource timing set is determined according to the obtained timing sorting result and the quantity threshold.

[0090] Exemplarily, the similarity threshold may be 0.6 or 0.7, and the quantity threshold may be 14 or 20, but is not limited to the example scenario.

[0091] Specifically, the resource usage data of the reference time period corresponding to the resource similarity less than the similarity threshold is deleted from the initial resource time series set to obtain the reference resource time series set. If the amount of data in the reference resource time series set is greater than the quantity threshold, the resource usage data in the reference resource time series set is sorted in chronological order. A higher ranking indicates a longer time interval between the reference time period and the current moment. Conversely, a lower ranking indicates a closer time interval between the reference time period and the current moment. The resource usage data of the quantity threshold ranked later in the time series sorting result is selected to obtain the historical resource time series set.

[0092] For example, assuming that the reference resource time series set contains resource usage data corresponding to five historical days, namely November 15, November 16, November 17, November 18, and November 19, and the quantity threshold is 4, then the historical resource time series set contains resource usage data corresponding to four historical days, namely November 16, November 17, November 18, and November 19.

[0093] The advantage of this setting is that, under the premise of ensuring resource similarity, resource usage data of historical periods close to the current time are selected for cluster resource monitoring and prediction, further ensuring the prediction accuracy of cluster resource usage.

[0094] Based on the above embodiments, specifically, after the initial resource timing set is screened according to the similarity threshold and the resource similarity of each reference time period to obtain the reference resource timing set, the method also includes: if the amount of data of the reference resource timing set is less than or equal to the quantity threshold, the reference resource timing set is used as the historical resource timing set.

[0095] S240: Input the historical resource time series set into the pre-trained resource monitoring model.

[0096] S250: Input the historical resource time series set into the linear time series network to obtain the output of the first resource usage result of the cluster system in the future period.

[0097] S260: Input the historical resource time series set into the nonlinear time series network to obtain the output of the second resource usage result of the cluster system in the future period.

[0098] S270: Input the first resource usage result and the second resource usage result into the resource output network to obtain the output target resource usage result of the cluster system in the future period.

[0099] S240-S270 in this embodiment are similar to those in the above embodiment. Figure 1 The illustrated S110 - S140 correspond to the same or similar ones, and are not described in detail in this embodiment.

[0100] The technical solution of this embodiment is to use the reference time period closest to the current moment in the initial resource time series set of the cluster system as the reference time period, determine the resource similarity according to the resource usage data of the reference time period and the resource usage data corresponding to each reference time period, and screen the initial resource time series set according to the resource similarity of each reference time period to obtain the historical resource time series set, thereby avoiding the influence of abnormal fluctuation data on the monitoring effect of cluster resources, so that the historical resource time series set can better reflect the law of change of resource usage parameters over time, and further ensure the prediction accuracy of cluster resource usage.

[0101] The following is an embodiment of a cluster resource monitoring device provided in an embodiment of the present invention. The device and the cluster resource monitoring method of the above embodiment belong to the same inventive concept. For details not described in detail in the embodiment of the cluster resource monitoring device, reference can be made to the contents of the cluster resource monitoring method in the above embodiment.

[0102] Figure 4 A schematic diagram of the structure of a cluster resource monitoring device provided by an embodiment of the present invention. Figure 4 As shown, the device includes: a historical resource timing set input module 310, a first resource usage result output module 320, a second resource usage result output module 330 and a target resource usage result output module 340.

[0103] The historical resource time series set input module 310 is used to input the historical resource time series set of the cluster system into the pre-trained resource monitoring model; wherein the historical resource time series set includes resource usage data of the historical period, and the resource monitoring model includes a linear time series network, a nonlinear time series network and a resource output network;

[0104] The first resource usage result output module 320 is used to input the historical resource time series set into the linear time series network to obtain the output first resource usage result of the cluster system in the future period;

[0105] The second resource usage result output module 330 is used to input the historical resource time series set into the nonlinear time series network to obtain the output second resource usage result of the cluster system in the future period;

[0106] The target resource usage result output module 340 is used to input the first resource usage result and the second resource usage result into the resource output network to obtain the output target resource usage result of the cluster system in the future period.

[0107] The technical solution of this embodiment is to set a linear timing network, a nonlinear network and a resource output network in a resource monitoring model, input the historical resource timing set into the linear timing network to obtain the output of the first resource usage result, input the historical resource timing set into the nonlinear timing network to obtain the output of the second resource usage result, input the first resource usage result and the second resource usage result into the resource output network to obtain the output target resource usage result of the cluster system in the future period, so that the resource monitoring model has the ability to express multi-scale features, improves the generalization ability of the resource prediction model, solves the problem that the usage of cluster resources cannot be accurately predicted, provides data support for the timely allocation and balanced allocation of cluster resources, and thus ensures the stable operation of the cluster system.

[0108] In a specific embodiment, the linear temporal network includes a first feature module, an additive attention module, a first long short-term memory module, and a first output module;

[0109] Among them, the first feature module is used to extract features from the historical resource time series set to obtain the first time series feature, the additive attention module is used to determine the additive time series feature based on the first time series feature, the first long and short-term memory module is used to determine the first long and short-term feature based on the additive time series feature, and the first output module is used to output the first resource usage result of the cluster system in the future time period based on the first long and short-term feature.

[0110] In a specific embodiment, the nonlinear temporal network includes a second feature module, a multiplicative attention module, a second long short-term memory module, and a second output module;

[0111] Among them, the second feature module is used to extract features from the historical resource time series set to obtain the second time series feature, the multiplicative attention module is used to determine the multiplicative time series feature based on the second time series feature, the second long and short-term memory module is used to determine the second long and short-term feature based on the multiplicative time series feature, and the second output module is used to output the second resource usage result of the cluster system in the future time period based on the second long and short-term feature.

[0112] In a specific embodiment, the device further comprises:

[0113] A reference period determination module, used to use the reference period in the initial resource time series set of the cluster system that is closest to the current time as the reference period; wherein the initial resource time series set contains resource usage data corresponding to at least two reference periods respectively;

[0114] A resource similarity determination module, used to determine resource similarities according to resource usage data of the base period and resource usage data corresponding to each reference period, wherein the reference period corresponds to the resource similarity one by one;

[0115] The historical resource time series set determination module is used to filter the initial resource time series set according to the resource similarity of each reference time period to obtain the historical resource time series set.

[0116] In a specific embodiment, the historical resource time series set determination module is specifically used to:

[0117] According to the similarity threshold and the resource similarity of each reference period, the initial resource time series set is screened to obtain the reference resource time series set;

[0118] If the data volume of the reference resource time series set is greater than the quantity threshold, the reference resource time series set is time-sequenced, and the historical resource time series set is determined according to the obtained time series sorting result and the quantity threshold.

[0119] In a specific embodiment, the resource similarity determination module is specifically used to:

[0120] For each reference period, resource usage data of the benchmark period and resource usage data of the reference period are divided according to the division time interval to obtain a benchmark segmented resource data set and a reference segmented resource data set;

[0121] At least two segment similarities are determined according to the benchmark segment resource data set and the reference segment resource data set, and statistical values ​​corresponding to the at least two segment similarities are used as resource similarities of the reference time period.

[0122] The cluster resource monitoring device provided in the embodiment of the present invention can execute the cluster resource monitoring method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0123] Figure 5 A schematic diagram of the structure of an electronic device provided for one embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the ROM 12 or the computer program loaded from the storage unit 18 to the RAM 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0125] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information or data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the cluster resource monitoring method provided in the above embodiments.

[0127] In some embodiments, the cluster resource monitoring method provided in the above embodiments may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the cluster resource monitoring method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the cluster resource monitoring method in any other appropriate manner (e.g., by means of firmware).

[0128] Various embodiments of the systems and techniques described above herein may be implemented in the following systems or combinations thereof: digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The computer program for implementing the cluster resource monitoring method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed completely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or completely on a remote machine or server.

[0130] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable storage medium. Examples of machine readable storage media may include electrical connections based on at least one line, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM), flash memories, optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein may be implemented on a terminal device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the terminal device. Other types of devices may also provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), a blockchain network, and the Internet.

[0133] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0135] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cluster resource monitoring method, characterized in that: include: Inputting the historical resource time series set of the cluster system into a pre-trained resource monitoring model; wherein the historical resource time series set includes resource usage data of a historical period, and the resource monitoring model includes a linear time series network, a nonlinear time series network and a resource output network; Inputting the historical resource time series set into the linear time series network to obtain the output of the first resource usage result of the cluster system in the future period; Inputting the historical resource time series set into the nonlinear time series network to obtain output a second resource usage result of the cluster system in a future time period; The first resource usage result and the second resource usage result are input into the resource output network to obtain the output target resource usage result of the cluster system in the future period.

2. The method according to claim 1, characterized in that: The linear temporal network includes a first feature module, an additive attention module, a first long short-term memory module and a first output module; Among them, the first feature module is used to extract features from the historical resource time series set to obtain a first time series feature, the additive attention module is used to determine an additive time series feature based on the first time series feature, the first long-short-time memory module is used to determine a first long-short-time feature based on the additive time series feature, and the first output module is used to output a first resource usage result of the cluster system in a future period based on the first long-short-time feature.

3. The method according to claim 1, characterized in that The nonlinear temporal network includes a second feature module, a multiplicative attention module, a second long short-term memory module and a second output module; Among them, the second feature module is used to extract features from the historical resource time series set to obtain a second time series feature, the multiplicative attention module is used to determine the multiplicative time series feature based on the second time series feature, the second long-short time memory module is used to determine the second long-short time feature based on the multiplicative time series feature, and the second output module is used to output the second resource usage result of the cluster system in the future time period based on the second long-short time feature.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Taking the reference time period closest to the current time in the initial resource time series set of the cluster system as the reference time period; wherein the initial resource time series set contains resource usage data corresponding to at least two reference time periods respectively; Determine resource similarities according to the resource usage data of the reference period and the resource usage data corresponding to each reference period, wherein the reference period corresponds to the resource similarities one by one; According to the resource similarity of each reference time period, the initial resource time series set is screened to obtain a historical resource time series set.

5. The method according to claim 4, characterized in that The step of screening the initial resource time series set to obtain the historical resource time series set according to the resource similarity of each reference time period includes: According to the similarity threshold and the resource similarity of each reference time period, the initial resource timing set is screened to obtain a reference resource timing set; If the data volume of the reference resource timing set is greater than the quantity threshold, the reference resource timing set is time-sequenced, and the historical resource timing set is determined according to the obtained time-sequenced result and the quantity threshold.

6. The method according to claim 4, characterized in that Determining resource similarity respectively according to the resource usage data of the benchmark period and the resource usage data corresponding to each reference period includes: For each reference period, resource usage data of the reference period and resource usage data of the reference period are respectively divided according to the division time interval to obtain a reference segment resource data set and a reference segment resource data set; At least two segment similarities are determined according to the benchmark segment resource data set and the reference segment resource data set, and statistical values ​​corresponding to the at least two segment similarities are used as resource similarities of the reference time period.

7. A cluster resource monitoring device, characterized in that: include: A historical resource time series set input module, used to input the historical resource time series set of the cluster system into a pre-trained resource monitoring model; wherein the historical resource time series set contains resource usage data of a historical period, and the resource monitoring model includes a linear time series network, a nonlinear time series network and a resource output network; A first resource usage result output module, used for inputting the historical resource time series set into the linear time series network to obtain an output of a first resource usage result of the cluster system in a future period; A second resource usage result output module, used for inputting the historical resource time series set into the nonlinear time series network to obtain an output second resource usage result of the cluster system in a future time period; The target resource usage result output module is used to input the first resource usage result and the second resource usage result into the resource output network to obtain the output target resource usage result of the cluster system in the future period.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the cluster resource monitoring method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the cluster resource monitoring method according to any one of claims 1 to 6 when executed.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the cluster resource monitoring method according to any one of claims 1 to 6.

Images