Node data acquisition method and device, electronic equipment, chip and medium

Abstract

The invention provides a node data acquisition method and device, and the method comprises the steps: determining a first feature set based on the original data of a plurality of nodes in a cluster in a preset time period; preprocessing the first feature data to obtain feature vectors of a plurality of preset time dimensions; based on the first feature data, a graph structure and a second feature set which reflect the node relation in the cluster are determined through correlation calculation, the graph structure is used for training a resource prediction model, and the resource prediction model is used for predicting the resource usage amount of the nodes in the cluster. According to the method provided by the invention, the operation data of the container is considered, the state data is added as the feature data, the comprehensive cycle data of different time dimensions is utilized to obtain the training data for training the resource prediction model, and the graph structure is determined through the correlation and is used for training the model, so that the resource prediction efficiency is improved. And the model obtained by training can cope with sudden increase or sudden decrease of resources.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing, and in particular to a node data acquisition method, device, electronic device, chip and medium. Background Art

[0002] The resource utilization of cloud servers is very important to cloud service providers. The current utilization of cloud server nodes is extremely low, and there will be a surge in service requests. Therefore, loose hardware resources are often pre-allocated to computing nodes in advance, but this will also lead to waste of idle resources. At present, by predicting future resource usage, the resources of the nodes are dynamically expanded or load balanced according to the predicted values ​​to achieve the effect of improving resource utilization. Most of the prediction methods are based on numerical prediction, that is, specific prediction of the possible resource usage of the nodes in the future. These methods can only predict some resource usage with relatively regular changes. In reality, there are often some emergencies in the load microservice cluster nodes. These methods cannot predict such sudden increase or decrease of resources. In addition, the current numerical prediction technology often does not consider the operation of the container inside the node, which also leads to low prediction accuracy. Summary of the invention

[0003] In order to solve the problems existing in the related technology, the present disclosure proposes a node data acquisition method, including: based on the original data of multiple nodes in a cluster within a preset time period, determining a first feature set, the first feature set including first feature data of a first node among the multiple nodes running a first container at each timestamp, the first feature data including basic data, operation data, and status data of the first container; preprocessing the first feature data to obtain feature vectors of multiple preset time dimensions, the second feature set including a first feature matrix of the first node running the first container at the first timestamp, the first feature matrix including first feature vectors of multiple preset time dimensions; based on the first feature data, determining a graph structure reflecting the node relationship in the cluster through correlation calculation, the second feature set and the graph structure are used for training a resource prediction model, and the resource prediction model is used to predict the resource usage of the nodes in the cluster.

[0004] The node data acquisition method, device, electronic device, chip and medium provided by the present invention are to solve the problem that the operation status of the container inside the node is not considered in the related technology. Starting from the operation status of the container in the cluster node, the node portrait is portrayed, and the horizontal and vertical characteristics of the node resource usage data in the cluster are considered through a neural network model based on a graph structure. It can identify the sudden increase or decrease of load to a greater extent.

[0005] A first aspect of the present disclosure provides a method for acquiring node data, the method comprising: determining a first feature set based on original data of multiple nodes in a cluster within a preset time period, the first feature set comprising first feature data of a first node among the multiple nodes running a first container at each timestamp, the first feature data comprising basic data, running data, and status data of the first container; preprocessing the first feature data to obtain feature vectors of multiple preset time dimensions, the second feature set comprising a first feature matrix of the first node running the first container at the first timestamp, the first feature matrix comprising first feature vectors of multiple preset time dimensions; determining a graph structure reflecting the node relationship in the cluster through correlation calculation based on the first feature data, the second feature set and the graph structure are used for training a resource prediction model, and the resource prediction model is used to predict resource usage of nodes in the cluster.

[0006] In some embodiments of the present disclosure, the status data includes the life cycle and queue number of the first container, and determining the first feature set based on the original data of multiple nodes in the cluster within a preset time period includes: determining the life cycle of the first container based on the creation time and deletion time of the first container; based on the creation time, determining the queue number of multiple containers running on the first node of the first container, and the queue number reflects the running order of the first container on the first node.

[0007] In some embodiments of the present disclosure, preprocessing the first feature data includes: performing format unification processing on the first feature data to obtain second feature data; dividing the second feature data into periods of week, day, and hour, respectively, and determining a first feature vector of the first node running the first container at the first timestamp, wherein the first feature matrix includes first feature vectors with periods of week, day, and hour, respectively.

[0008] In some embodiments of the present disclosure, the method further includes: based on the difference in feature data between the current timestamp and the previous timestamp of the prediction indicator, determining second feature data whose difference exceeds a first threshold as a negative sample, and the prediction indicator is one or more categories in the first feature data.

[0009] In some embodiments of the present disclosure, based on the first feature data, determining a graph structure reflecting the relationship between nodes in the cluster through correlation calculation includes: clustering multiple containers in the cluster using unsupervised clustering to obtain multiple types of containers; for each type of container, determining the first feature data whose correlation with the prediction index meets preset conditions as the third feature data; based on the third feature data, determining the correlation coefficient between the nodes using the Pearson correlation coefficient; and constructing a graph structure based on the correlation coefficient.

[0010] In some embodiments of the present disclosure, the method further includes: resampling the feature data in the second feature set to obtain a resampled data set, wherein the proportion of negative samples in the resampled data set is greater than or equal to a second threshold, and the resampled data set includes resampled data of the multiple preset time dimensions; using the resampled data set, training the initial resource prediction model, iterating in a loop until the number of training times is reached, and obtaining the resource prediction model.

[0011] In some embodiments of the present disclosure, using a resampled data set to train an initial resource prediction model includes: inputting the resampled data set into a one-dimensional convolutional neural network to extract container features of multiple containers running on a first node among multiple nodes; based on the graph structure and container features, using a graph attention network to update the state data of the first node, the updated state data including feature data of neighbor nodes of the first node; inputting the updated state data into a recurrent neural network to obtain second feature vectors of the multiple preset time dimensions; concatenating the second feature vectors of the multiple preset time dimensions and performing linear full connection to obtain a first prediction result; adjusting the parameters of the initial resource prediction model according to the first prediction result and the first feature vector, and stopping the training when the number of iterations reaches a preset number to obtain a resource prediction model.

[0012] The second aspect of the present disclosure proposes a node data acquisition device, which includes: a first processing module, which is used to determine a first feature set based on raw data of multiple nodes in a cluster within a preset time period, the first feature set including first feature data of a first node among multiple nodes running a first container at each timestamp, the first feature data including basic data, operation data, and status data of the first container; a preprocessing module, which is used to preprocess the first feature data to obtain feature vectors of multiple preset time dimensions, the second feature set including a first feature matrix of the first node running the first container at the first timestamp, the first feature matrix including first feature vectors of multiple preset time dimensions; a second processing module, which is used to determine a graph structure reflecting the node relationship in the cluster through correlation calculation based on the first feature data, the second feature set and the graph structure are used for training a resource prediction model, and the resource prediction model is used to predict the resource usage of nodes in the cluster.

[0013] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: one or more processors; a storage device communicatively connected to the one or more processors, on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect embodiment of the present disclosure.

[0014] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0015] The fifth aspect embodiment of the present disclosure provides a computer program product, including a computer program, and the computer program implements the method described in the first aspect embodiment of the present disclosure after being executed.

[0016] In summary, according to the node data acquisition method proposed in the present disclosure, a first feature set is determined based on the original data of multiple nodes in the cluster within a preset time period, the first feature set includes the first feature data of the first node among the multiple nodes running the first container at each timestamp, and the first feature data includes the basic data, operation data, and status data of the first container; the first feature data is preprocessed to obtain feature vectors of multiple preset time dimensions, the second feature set includes the first feature matrix of the first node running the first container at the first timestamp, and the first feature matrix includes the first feature vectors of multiple preset time dimensions; based on the first feature data, through correlation calculation, a graph structure reflecting the node relationship in the cluster is determined, and the second feature set and the graph structure are used for training a resource prediction model, and the resource prediction model is used to predict the resource usage of nodes in the cluster. The method proposed in the present disclosure constructs training data based on the collected original data, taking into account the characteristics of the container and the comprehensive periodic data of multiple time dimensions, and determines the graph structure through correlation, and uses the graph attention network to train the model so that the trained model can cope with the situation of sudden increase or decrease in resources.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0019] Figure 1 A flowchart of a node data acquisition method proposed in an embodiment of the present disclosure;

[0020] Figure 2 A flowchart for determining a first feature set proposed in an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of the preprocessing process proposed in the embodiment of the present disclosure;

[0022] Figure 4A schematic diagram of a process of obtaining node data according to an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of a process for determining a graph structure proposed in an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of a process of obtaining node data according to an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of a process for training an initial resource prediction model proposed in an embodiment of the present disclosure;

[0026] Fig. 8A A system flow chart provided for an embodiment of the present disclosure;

[0027] Figure 8B A flowchart of model training provided for an embodiment of the present disclosure;

[0028] Fig. 9 A schematic diagram of the structure of a node data acquisition device provided in an embodiment of the present disclosure;

[0029] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. Embodiments are described below with reference to the accompanying drawings.

[0031] The node data acquisition method provided in this application is described in detail below with reference to the accompanying drawings.

[0032] Figure 1 Flow chart of a node data acquisition method proposed in an embodiment of the present disclosure. Figure 1 As shown, the node data acquisition method includes the following steps:

[0033] Step 101: determine a first feature set based on raw data of multiple nodes in a cluster within a preset time period.

[0034] In an embodiment of the present disclosure, the first feature set includes first feature data of a first node among multiple nodes running a first container at each timestamp, and the first feature data includes basic data, running data, and feature data of the first container.

[0035] In an embodiment of the present disclosure, there are multiple nodes in a cluster, one application can be deployed on each node, and each node can run multiple containers.

[0036] In an embodiment of the present disclosure, the first node is any node in the cluster, the first node can run multiple containers at each timestamp, and the first container is any one of the multiple containers.

[0037] In the embodiment of the present disclosure, the preset time period is a preset time period for collecting data, and the length of the preset time period may be a longer time period, such as 40 days.

[0038] In an embodiment of the present disclosure, the original data may be container detailed information, including the service category, CPU usage, memory usage, container creation time, container deletion time, etc. of each container.

[0039] In an embodiment of the present disclosure, the basic data of the first container may include the creation time and deletion time of the first container; the running data may include the service category, CPU usage, and memory usage of the first container.

[0040] In an embodiment of the present disclosure, a first feature set is determined based on original data, and first feature data of each container can be determined based on detailed container information, where the first feature data includes basic data and running data of the container, wherein the basic data is basic information of the container, including creation time and deletion time of the container, and the running data includes service category, CPU usage, memory usage, etc. of the container. The first feature data also includes status data, which is determined based on the basic data, and the status data can reflect status information of the first container running on the first node.

[0041] In an embodiment of the present disclosure, the status data may be an operation cycle of the first container determined according to basic data of the first container.

[0042] In an embodiment of the present disclosure, the state data may be to determine the order in which the first container is run on the first node according to the basic data of the first container, thereby determining the queue state of multiple containers running on the first node.

[0043] In an embodiment of the present disclosure, based on the original data, the first feature data of each container running on each node in the cluster can be obtained, so that the first feature set corresponding to the cluster can be obtained, and the first feature set includes the first feature data of each container running on each node in the cluster.

[0044] For example, to collect long-term historical data, select a time period, and each node collects detailed data for the time period. The time interval is 10 seconds. Each timestamp of each node records the detailed information of the container running on the node at the current timestamp. According to the detailed information of the container, the characteristics of the container are determined and recorded as the collected data set: in represents the data details of node n at timestamp i, x1 represents the characteristic data of the first container running, and x m Represents the characteristic data of the running m-th container.

[0045] Step 102: preprocess the first feature data to obtain feature vectors of multiple preset time dimensions.

[0046] In an embodiment of the present disclosure, the second feature set includes a first feature matrix of a first node running a first container at a first timestamp, and the first feature matrix includes a plurality of first feature vectors of preset time dimensions.

[0047] In an embodiment of the present disclosure, preprocessing may include data cleaning and time dimension division. In other words, the first feature data is first cleaned to unify the formats of multiple data in the first feature data to facilitate time dimension division.

[0048] In an embodiment of the present disclosure, preprocessing the first characteristic data may be preprocessing the basic data, operation data, and status data of each container running on each node in the cluster, wherein the division in the time dimension may be dividing the first characteristic data of each container running on each node at each timestamp in the first time period according to a preset time dimension to obtain a first characteristic vector at each preset time dimension, thereby obtaining a first characteristic matrix of the first container running on the first node at each timestamp, wherein the first characteristic matrix includes the first characteristic vector at each preset time dimension.

[0049] In the embodiment of the present disclosure, the preset time dimension may be a week period, a day period, or an hour period. In other words, by preprocessing the first feature data, a data matrix with a week period, a day period, or an hour period may be obtained.

[0050] For example, the collected data set is preprocessed, including a standard preprocessing process and dividing the data set to obtain comprehensive periodic data, where the comprehensive periodic data is data with a period of week, day, and hour.

[0051] Step 103: Based on the first feature data, determine a graph structure reflecting the node relationship within the cluster through correlation calculation.

[0052] In an embodiment of the present disclosure, the second feature set and the graph structure are used to train a resource prediction model, and the resource prediction model is used to predict resource usage of nodes in a cluster.

[0053] In an embodiment of the present disclosure, correlation calculation may be to calculate the correlation between nodes, thereby characterizing a topological relationship graph based on the correlation, obtaining the correlation between multiple nodes in the cluster, and obtaining a graph structure reflecting the node relationship in the cluster.

[0054] In an embodiment of the present disclosure, the correlation between nodes can be calculated by first determining the correlation between all features of the container running on the node and the indicator that needs to be predicted, thereby determining the feature data related to the predicted indicator, and using the determined related feature data to calculate the correlation between nodes, thereby determining a graph structure that reflects the relationship between nodes in the cluster, avoiding the resources and efficiency consumed by using all the feature data for calculation.

[0055] In an embodiment of the present disclosure, the second feature set is a set of feature matrices of each container running on each node in the cluster at each timestamp, and the data in the second feature set is used as training data and test data for training a resource prediction model, so that the obtained trained resource prediction model can predict the resource usage on each node in the cluster.

[0056] In an embodiment of the present disclosure, a graph structure is used for training a resource prediction model. The graph structure may be applied in the training process of the resource prediction model, and the model training may be performed through a graph attention network, thereby improving the efficiency of the model training.

[0057] In summary, according to the node data acquisition method proposed in the present disclosure, based on the original data of multiple nodes in the cluster within a preset time period, a first feature set is determined, the first feature set includes the first feature data of the first node among the multiple nodes running the first container at each timestamp, and the first feature data includes the basic data, operation data, and status data of the first container; the first feature data is preprocessed to obtain feature vectors of multiple preset time dimensions, the second feature set includes the first feature matrix of the first node running the first container at the first timestamp, and the first feature matrix includes the first feature vectors of multiple preset time dimensions; based on the first feature data, through correlation calculation, a graph structure reflecting the node relationship in the cluster is determined, the second feature set and the graph structure are used for training a resource prediction model, and the resource prediction model is used to predict the resource usage of nodes in the cluster. Through the method provided by the present disclosure, the operation data of the container is taken into account, and the status data is added as feature data, and the comprehensive periodic data of different time dimensions are used to obtain training data for training the resource prediction model, so that the trained model can cope with the situation of sudden increase or decrease of resources.

[0058] based on Figure 1 The embodiment shown, Figure 2 The following is a schematic diagram of a process for determining a first feature set proposed in the present disclosure. Figure 2based on Figure 1 The embodiment shown, Figure 1 Step 101 in is further described as follows: Figure 2 As shown, the operation data includes the life cycle and queue number of the first container, and the method includes the following steps:

[0059] Step 201: determine the life cycle of the first container based on the creation time and deletion time of the first container.

[0060] In an embodiment of the present disclosure, the basic data of the first container may include the creation time and deletion time of the first container. Based on the creation time and deletion time of the first container, the life cycle of the first container can be obtained, thereby obtaining the life cycle of each container running on each node in the cluster.

[0061] In an embodiment of the present disclosure, the life cycle of a container may be an absolute value of the creation time of the container minus the deletion time of the container, or an absolute value of the deletion time of the container minus the creation time of the container.

[0062] Step 202: Determine, based on the creation time, a queue number of multiple containers in which the first container is running on the first node.

[0063] In an embodiment of the present disclosure, the queue number reflects the running order of the first container on the first node.

[0064] In an embodiment of the present disclosure, multiple containers running on the first node are sorted based on the creation time to obtain a queue number corresponding to each container.

[0065] For example, when collecting data, X t n The containers are arranged in the order of creation time to ensure that the queue status characteristics of the containers can be perceived later.

[0066] In an embodiment of the present disclosure, based on the creation time of the container running on each node in the cluster, the queue number of each container on the corresponding node can be obtained.

[0067] In the embodiment of the present disclosure, the queue number of the first container is determined, so as to determine the order of multiple containers running on the first node, thereby avoiding disrupting the order of characteristic data of the containers in subsequent data processing.

[0068] In the above embodiment, by calculating the life cycle of each container and determining the queue number of each container according to the creation time, the life cycle and queue number are used as the status data of the container for subsequent training of the resource prediction model. It is possible to train the model using the horizontal and vertical characteristics of the node resource usage data in the cluster, so that the trained model can identify future sudden increases or decreases in load.

[0069] based on Figure 1 , Figure 2 The embodiment shown, Figure 3 The following is a schematic diagram of the pre-processing process proposed in the present disclosure. Figure 3 based on Figure 1 In the embodiment shown, step 102 is further defined. Figure 3 As shown, the method comprises the following steps:

[0070] Step 301, performing format unification processing on the first characteristic data to obtain second characteristic data.

[0071] In an embodiment of the present disclosure, the format unification process may include at least one of missing value filling, outlier processing, and data standardization.

[0072] In the embodiment of the present disclosure, the first characteristic data is processed in a unified format, which may be processing the basic data, operation data, and status data of each container running on multiple nodes in the cluster in a unified format.

[0073] In the embodiments of the present disclosure, missing value filling can be performed by filling the missing values ​​in the first feature data caused by data collection errors or equipment failures, etc., by filling the mean value, median, mode, specific value, or using interpolation methods.

[0074] In an embodiment of the present disclosure, outlier processing may be to delete data points in the first feature data that deviate from other values, replace them with the mean / median / mode, use a capping method, or use statistical or clustering methods to identify and process anomalies.

[0075] In an embodiment of the present disclosure, data normalization may be to scale the data in the first feature data so that it falls within a specific interval so that different features can be compared and analyzed on the same scale, such as Z-score normalization and minimum-maximum normalization.

[0076] For example, the data set is preprocessed, including missing value filling, outlier processing, and data standardization.

[0077] In the embodiments of the present disclosure, the format unification processing for the first feature data may also be other methods of data preprocessing, which is not limited by the present disclosure.

[0078] Step 302 , divide the second characteristic data into periods of week, day, and hour respectively, and determine a first characteristic vector of the first node running the first container at the first timestamp.

[0079] In an embodiment of the present disclosure, the first characteristic matrix includes first characteristic vectors with a period of week, day, and hour respectively.

[0080] In the embodiments of the present disclosure, the first characteristic vector is determined with week, day and hour as the period respectively. The second characteristic data after the format unification processing can be determined with week, day and hour as the period, and the second characteristic data of the first timestamp in each week, every day within a week and a preset number of consecutive timestamps are determined, so as to obtain the first characteristic matrix of the first node running the first container at the first timestamp, and the first characteristic matrix includes the first characteristic vector obtained by dividing the second characteristic data with week, day and hour as the period respectively.

[0081] For example, for the preprocessed data set, for node n, the specific operation is as follows: starting from timestamp i, collect the specific data of the time in the past four weeks, recorded as:

[0082] It is time series data; collect the specific data of the time in the past 7 days and record it as: Collect the last 10 data records and record them as: A data entry is By collecting data at each timestamp, a feature matrix of the nth node running the container at each timestamp is obtained, thereby obtaining a feature set of the multiple nodes in the cluster running the container at each timestamp.

[0083] In the above embodiment, by considering comprehensive periodic data with weeks, days and hours as periods, the long-term node operation status and the periodicity of changes are accurately monitored with weeks, days and hours as periods, and the second feature set obtained is used as training data to train the initial resource prediction model, so that the trained resource prediction model can predict the sudden increase or decrease of resources.

[0084] based on Figure 1 , Figure 2 , Figure 3 The embodiment shown, Figure 4 The following is a flow chart of the node data acquisition method proposed in the present disclosure. Figure 4 based on Figure 1 , Figure 2 , Figure 3 The embodiment shown, as Figure 4 As shown, the method also includes the following steps:

[0085] Step 401: Based on the difference of the feature data between the current timestamp and the previous timestamp of the prediction index, second feature data whose difference exceeds a first threshold is determined as a negative sample.

[0086] In the embodiments of the present disclosure, the prediction indicator is one or more categories of the first feature data. In other words, the prediction indicator can be one or more categories of basic data, operation data, and status data, that is, the prediction indicator can be an indicator that the trained resource prediction model needs to predict, such as CPU usage or memory usage.

[0087] In the embodiment of the present disclosure, the difference of the feature data between the current timestamp and the previous timestamp based on the prediction index may be the difference of the feature data of the same type between two adjacent timestamps.

[0088] In the embodiments of the present disclosure, the first threshold value may be a preset value used to judge the standard of negative sample, i.e., the sample with sudden increase or decrease in resources. In other words, if the value of the characteristic data of the current timestamp is significantly lower than the value of the characteristic data of the previous timestamp, the difference is a negative value, and the absolute value of the difference is compared with the first threshold value. If it is greater than the first threshold value, it indicates that the current timestamp is a sample with sudden decrease in resources; if the value of the characteristic data of the current timestamp is significantly higher than the value of the characteristic data of the previous timestamp, the difference is a positive value, and the difference is compared with the first threshold value. If it is greater than the first threshold value, it indicates that the current timestamp is a sample with sudden increase in resources.

[0089] In an embodiment of the present disclosure, determining the second feature data whose difference exceeds the first threshold as a negative sample may be determining the second feature data of the prediction indicator at the current timestamp as a negative sample.

[0090] For example, the difference between the CPU or memory usage at timestamp t and the previous timestamp is calculated, and those exceeding the threshold are recorded as negative samples.

[0091] In the above embodiment, by determining the negative samples, it can be used to sample the training data during the training process, identify the negative samples, and make the ratio of positive and negative samples in the sampled data meet the preset ratio.

[0092] Figure 5 The following is a schematic diagram of a process for determining a graph structure proposed in the present disclosure. Figure 5 based on Figure 1-Figure 4 The embodiment shown, Figure 1 Step 103 in the method is further described, the method comprises the following steps:

[0093] Step 501 : cluster multiple containers in a cluster using unsupervised clustering to obtain multiple types of containers.

[0094] In an embodiment of the present disclosure, unsupervised clustering may be performed using a K-means method.

[0095] In an embodiment of the present disclosure, unsupervised clustering is used to cluster multiple containers in a cluster. Unsupervised clustering may be performed based on feature data of all containers running in the cluster to obtain multiple types of containers.

[0096] Step 502: for each type of container, first feature data whose correlation with the prediction index satisfies a preset condition is determined as third feature data.

[0097] In an embodiment of the present disclosure, determining the first feature data whose correlation with the prediction index meets the preset conditions as the third feature data can be to screen the feature data of each type of container, and retain the feature data with high correlation with the prediction index for subsequent determination of the correlation between nodes.

[0098] In an embodiment of the present disclosure, for the characteristic data in each type of container, the correlation with the prediction index can be determined by calculating the Pearson correlation coefficient, determining the correlation between the prediction index and each type of first characteristic data, and determining the first characteristic data whose correlation meets the preset conditions as the third characteristic data, as the characteristic data of the corresponding container.

[0099] In an embodiment of the present disclosure, the preset condition may be that the correlation value is close to 1 or the correlation ranking value is in the top N digits, that is, feature data with a correlation coefficient close to 1 or in the top N digits is determined as an important feature for subsequent construction of a graph structure between nodes in the cluster.

[0100] In an embodiment of the present disclosure, the preset condition may be that the value of the correlation satisfies a preset interval, for example, the first characteristic data corresponding to the correlation value between 0.8 and 1 is determined as the third characteristic data.

[0101] In the embodiment of the present disclosure, the third feature data has a high correlation with the prediction index, which means that the third feature data can reflect the trend of the prediction index.

[0102] For example, feature screening is performed on each category of containers. According to the indicator to be predicted, taking CPU utilization as an example, the correlation between all features in the container and CPU utilization is calculated to screen out important features.

[0103] Step 503: Based on the third feature data, the correlation coefficient between the nodes is determined using the Pearson correlation coefficient.

[0104] In the embodiments of the present disclosure, the Pearson Correlation Coefficient (PCC) is a statistic used to measure the degree of linear correlation between two variables. The value range is between -1 and 1. The closer the absolute value is to 1, the stronger the linear relationship between the two variables is. The closer the absolute value is to 0, the weaker the linear relationship between the two variables is or there is no linear relationship.

[0105] In an embodiment of the present disclosure, based on the third feature data, the correlation coefficient between nodes is determined using the Pearson correlation coefficient, which can be achieved by using the feature data in each container that has a high correlation with the prediction index to calculate the correlation between nodes.

[0106] In an embodiment of the present disclosure, using the Pearson correlation coefficient to determine the correlation coefficient between nodes may be to calculate the correlation of the third feature data in different nodes to obtain the correlation coefficient between different nodes.

[0107] In an embodiment of the present disclosure, the correlation coefficient between nodes is determined based on the third feature data. A type of feature data may be used in the third feature data to calculate the correlation coefficient between the same type of feature data between different nodes, thereby obtaining the values ​​of the correlation coefficients of multiple node pairs.

[0108] In an embodiment of the present disclosure, the correlation coefficient between nodes is determined based on the third feature data. Multiple categories of feature data may be used in the third feature data. For each category of feature data, the correlation coefficient between different nodes is calculated, and the mean or median is taken as the value of the correlation coefficient of multiple node pairs.

[0109] Step 504: construct a graph structure based on the correlation coefficient.

[0110] In an embodiment of the present disclosure, a graph structure is constructed based on a correlation coefficient. Node pairs with high correlation coefficients may be determined as node pairs in the graph structure to construct a graph structure. The graph structure is used for a graph attention network in the resource prediction model training process.

[0111] In an embodiment of the present disclosure, based on the correlation coefficient, constructing a graph structure may be to construct a graph structure of multiple node pairs in a cluster according to the values ​​of the correlation coefficients of multiple node pairs. For example, when one type of data is selected for calculation, a graph structure may be constructed according to the correlation coefficients of the node pairs calculated from this type of data, or when multiple types of data are selected for calculation, the graph structure may be constructed according to the correlation coefficients of the node pairs calculated from each type of data, taking the average or median.

[0112] In an embodiment of the present disclosure, a graph structure reflects the relationship between multiple nodes in a cluster. All nodes in the cluster are connected by edges or without edges in the graph. If the correlation coefficient of a node pair is high, the node pair is connected by edges and has a higher weight. If the correlation coefficient of the node pair is low or close to 0, the nodes are not connected by edges.

[0113] For example, the Pearson correlation coefficient is used to calculate the correlation between nodes, and node pairs with high correlation are selected according to the cluster size to characterize the graph structure of the cluster, and the output topological structure data is in the form of node pairs.

[0114] In the above embodiment, the feature data in each container in the cluster is screened through pre-set prediction indicators, and the feature data with high correlation with the prediction indicators is determined as data for constructing a graph structure between multiple nodes in the cluster. This can make the obtained graph structure more realistically reflect the correlation between the nodes, and the subsequent use of the graph structure for model training can make the model prediction results obtained through training more accurate.

[0115] based on Figure 1-Figure 5 The embodiment shown, Figure 6 The following is a flow chart of the node data acquisition method proposed in the present disclosure. Figure 6 right Figure 1 Further describing step 103 in the method, the method further comprises the following steps:

[0116] Step 601, resample the feature data in the second feature set to obtain a resampled data set.

[0117] In an embodiment of the present disclosure, the proportion of negative samples in the resampled data set is greater than or equal to a second threshold, and the resampled data set includes resampled data of multiple preset time dimensions.

[0118] In an embodiment of the present disclosure, resampling the feature data in the second feature set may be resampling the second feature data in the second feature set so that the proportion of negative samples in the resampled data set is greater than or equal to a second threshold, and the second threshold is a pre-set value, such as one-half.

[0119] In an embodiment of the present disclosure, the proportion of negative samples in the resampled data set is greater than or equal to the second threshold, which may be that the proportion of the second characteristic data of a sudden increase or decrease in resources is greater than or equal to the second threshold.

[0120] In an embodiment of the present disclosure, making the proportion of negative samples in the resampled data set greater than or equal to a second threshold value can be achieved by dynamically adjusting the proportion of negative samples in the resampled data during multiple sampling processes so that the sampling results satisfy that the proportion of negative samples is higher than that of other samples, so that there are more samples of sudden increase or decrease in resources during the model training process, so that the trained model can predict the situation of sudden increase or decrease in resources.

[0121] In the embodiment of the present disclosure, by resampling the feature data in the second feature set, the resampled data set obtained is a feature data set obtained according to a period of week, day, and hour.

[0122] For example, the data in the training set is randomly resampled, and the resampling makes the ratio of the number of negative samples and positive samples exceeding the threshold be 1:1. This ratio can be changed according to the number of positive and negative examples in the collected samples to increase the proportion of negative samples.

[0123] Step 602: Use the resampled data set to train the initial resource prediction model, iterate in a loop until the number of training times is reached, and obtain the resource prediction model.

[0124] In an embodiment of the present disclosure, a resampled data set is used to train an initial resource prediction model. The initial resource prediction model may be trained by using the second feature data of multiple containers obtained by resampling as training data and test data, and iterated in a loop until a preset number of training times is reached, or by defining a loss function to make the loss value as small as possible, so that the prediction result obtained by the obtained resource prediction model in actual prediction is closer to the true value.

[0125] In an embodiment of the present disclosure, a resampled data set is used to train an initial resource prediction model. The initial resource prediction model may be trained using a set of feature data with periods of weeks, days, and hours, so that the initial resource prediction model can learn the data features with periods of weeks, days, and hours respectively, and thus be able to predict resource usage over a longer period.

[0126] In the above embodiment, by processing the collected feature data within the cluster, combining the life cycle of the container, the queue number, and comprehensively considering the periodic data of the week, day, and hour, a graph structure reflecting the correlation between the nodes in the cluster is constructed through correlation calculation, and the obtained data and graph structure are used in the training process of the resource prediction model, so that the trained model can cope with the situation of sudden increase or decrease of resources, and the prediction result is more accurate.

[0127] based on Figure 1-Figure 6 The embodiment shown, Figure 7 The following is a schematic diagram of the process of training the initial resource prediction model proposed in the present disclosure. Figure 7 based on Figure 6 The embodiment shown, Figure 6 The method further describes step 602 in the embodiment of the present invention, and comprises the following steps:

[0128] Step 701: input the resampled data set into a one-dimensional convolutional neural network to extract container features of multiple containers running on a first node among multiple nodes.

[0129] In an embodiment of the present disclosure, a one-dimensional convolutional neural network (1D-CNN) may be used to extract a feature reflecting the queue state of a container in a node, namely, the queue number of the container, and based on the queue number, extract container features of multiple containers so that the obtained container feature number is lower than the input container feature number, thereby achieving the purpose of dimensionality reduction.

[0130] In an embodiment of the present disclosure, the data input to the one-dimensional convolutional neural network is resampled data with a period of week, day, and hour, and the output is feature data after dimensionality reduction.

[0131] For example, a 1D-CNN (one-dimensional convolutional neural network) model is used to extract container features. The purpose of using a one-dimensional convolutional neural network is to better capture the order of the container queue and the dependency between the status information of the containers in the queue. The calculation process is recorded as:

[0132]

[0133] Where p represents the number of container features of the output node at timestamp t. Since a one-dimensional convolutional neural network is used, After input, the dimension of the feature number of the container in the result will be reduced, so p<n.

[0134] In the embodiments of the present disclosure, a one-dimensional convolutional neural network is used to process the resampled data, which can capture local patterns and long-term dependencies in the data and has good generalization ability and scalability.

[0135] Step 702: Based on the graph structure and container features, the state data of the first node is updated using the graph attention network.

[0136] In an embodiment of the present disclosure, the updated state data includes feature data of neighbor nodes of the first node.

[0137] In an embodiment of the present disclosure, a graph structure is used for a graph attention network, and the container features of the multiple containers obtained in step 701 are used as inputs of the graph attention network to update the state data of each node to obtain updated node state data.

[0138] In the embodiments of the present disclosure, the feature data between nodes are related, and node pairs and weights between node pairs are used in the graph structure to reflect the relationship between nodes. Based on the graph structure, a graph attention network is used to update the state data of each node, so that the feature data of adjacent nodes in the graph structure can be transmitted to each other.

[0139] In an embodiment of the present disclosure, a graph attention network based on a graph structure is used, and different weights can be assigned to different neighbor nodes of the first node through the attention coefficient, thereby improving the algorithm performance.

[0140] For example, based on the feature map of the graph attention network, the graph-based algorithm can assign different weights to different neighbor nodes of a node through the attention coefficient, thereby improving the algorithm performance. First, the normalized attention coefficient between nodes is calculated:

[0141] Where exp represents the exponential function with natural logarithm as the base, LeakyReLU is the activation function, and W are the parameters that need to be trained, T represents the transpose of the vector, Represents the set of neighbor nodes of node i. Next, update the node status:

[0142]

[0143] The updated node status contains the information transmitted by the neighboring nodes.

[0144] Step 703: input the updated state data into a recurrent neural network to obtain second feature vectors of multiple preset time dimensions.

[0145] In the embodiments of the present disclosure, the recurrent neural network can be LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit). In order to avoid the problem of gradient explosion or gradient disappearance, GRU is used to construct a recurrent neural network that can sense weeks, days, and hours as cycles.

[0146] In an embodiment of the present disclosure, the updated state data of each node includes state data with periods of week, day, and hour. By inputting it into a recurrent neural network, the state data with periods of week, day, and hour can be processed separately to obtain a second eigenvector on each time dimension, that is, a second eigenvector on the time dimension of week, day, and hour.

[0147] For example, the new node data after the state is updated is grouped into the recurrent neural network GRU according to the period of week, day, and hour:

[0148]

[0149] where r t 、z t , is an intermediate variable, W z , W r , W is the parameter that needs to be trained, h t-1 and h t is the hidden state of the previous layer of the recurrent neural network and the hidden state of the current layer. Through the above state update, the obtained h t That is the hidden state input to the next step.

[0150] Step 704: concatenate the second feature vectors of multiple preset time dimensions and perform linear full connection to obtain a first prediction result.

[0151] In an embodiment of the present disclosure, the output result of the model is obtained by concatenating and linearly fully connecting the second eigenvector of the preset time dimension output by the recurrent neural network, and the output result is the forward output result of the recurrent neural network.

[0152] For example, the weekly, daily, and hourly data will be The model is calculated to obtain three feature vectors which are concatenated and then connected to a linear fully connected layer to output the prediction results.

[0153] Step 705, adjusting the parameters of the initial resource prediction model according to the first prediction result and the first feature vector, and stopping the training when the number of iterations reaches a preset number to obtain a resource prediction model.

[0154] In an embodiment of the present disclosure, the first prediction result is compared with the actual result corresponding to the first feature vector to adjust the relevant parameters in the initial resource prediction model so that the prediction result is closer to the actual result. The training is stopped until the number of iterations reaches a preset number or the prediction result is closer to the actual result to obtain a resource prediction model.

[0155] In an embodiment of the present disclosure, a loss function is constructed based on the first prediction result and the actual result corresponding to the first feature vector, and training is performed based on the principle of minimizing the value of the loss function. When the value of the loss function is close to 0, the training ends and a resource prediction model is obtained.

[0156] In an embodiment of the present disclosure, adjusting the parameters of the initial resource prediction model according to the first prediction result and the first eigenvector may be a reverse processing process of the recurrent neural network. Through the reverse cycle, the parameters in the model may be adjusted so that the forward output result of the model may be close to the actual result.

[0157] For example, define the loss function, loop the model training, and output the prediction model model. The final output is: Compare the output with the actual result, adjust the model parameters to make the output close to the actual result, and obtain the prediction model.

[0158] In summary, the method proposed in the present disclosure determines a first feature set based on the original data of multiple nodes in a cluster within a preset time period, the first feature set includes the first feature data of the first node among multiple nodes running the first container at each timestamp, and the first feature data includes the basic data, operation data, and status data of the first container; the first feature data is preprocessed to obtain feature vectors of multiple preset time dimensions, the second feature set includes the first feature matrix of the first node running the first container at the first timestamp, and the first feature matrix includes the first feature vectors of multiple preset time dimensions; based on the first feature data, through correlation calculation, a graph structure reflecting the node relationship in the cluster is determined, and the second feature set and graph structure are used for training a resource prediction model, and the resource prediction model is used to predict the resource usage of nodes in the cluster. Starting from the operation status of the containers in the cluster nodes, this method depicts the node portraits and constructs the graph structure, which is used in the graph attention network in the model training, and comprehensively considers the horizontal and vertical characteristics of the node resource usage data in the cluster, so that the trained model can identify the sudden increase or decrease of the future load.

[0159] The following is a detailed introduction to a specific implementation of a node data acquisition method provided by an embodiment of the present disclosure.

[0160] Fig. 8A FIG. 1 is a system flow chart of this embodiment. Fig. 8A As shown, the following steps are included:

[0161] Step 1: Collect long-term historical data: This stage collects cloud cluster load data over a longer period of time, spanning more than one month.

[0162] Select a time period, and each node collects detailed data for the time period, with an interval of 10 seconds; each timestamp of each node records the detailed information of the container running on the node at the current timestamp, including but not limited to the service category of the container, the CPU usage and memory usage of the container, etc., as well as the container life cycle (container creation time minus container deletion time) and queue status, collectively referred to as container features. The collected data is recorded as: in, represents the data details of node n at timestamp i, x1 represents the characteristic data of the first container running, and x m Represents the characteristic data of the running mth container. The containers are arranged in the order of creation time to ensure that the subsequent algorithms can collect and perceive the queue status characteristics of the containers.

[0163] Step 2: Data preprocessing: preprocess the data set, including missing value filling, outlier processing, and data standardization.

[0164] This stage mainly divides the data set. For node n, if we start from timestamp i and collect the specific data of the time in the past four weeks, it can be recorded as: Collect the specific data of this moment in the past 7 days and record it as: Collect the last 10 data records, recorded as:

[0165] The data entry of node n at timestamp i is:

[0166] Calculate the difference between the CPU (or memory, etc.) at timestamp t and the previous timestamp. If it exceeds the threshold, it is recorded as a negative sample. The timestamp is occupied by time sliding. All collected data is packaged into training sets and test sets. The training set is used to train the model, and the test set is used to test the model performance during the training process for early stopping.

[0167] Step 3: Feature engineering.

[0168] First, all containers are clustered unsupervisedly (using methods such as K-means), and features are screened for containers of each category. Based on the indicator to be predicted, taking CPU utilization as an example, the correlation between all features in the container and CPU utilization is calculated to screen out the most important features.

[0169] Step 4: Characterize the node topology relationship.

[0170] The Pearson correlation coefficient is used to calculate the correlation between nodes. Node pairs with high correlation are selected according to the cluster size to characterize the graph structure of the cluster. The output topology of this step is in the form of node pairs.

[0171] Step 5: Randomly resample the data in the training set so that the ratio of negative samples to positive samples exceeding the threshold is 1:1 (the ratio can be changed according to the number of positive and negative samples in the collected samples to increase the proportion of negative samples). The three types of time series data in step 2 above are respectively input into the calculation model for training. The flowchart of model training is shown in the figure below: Figure 8B As shown, the process is as follows:

[0172] Step 5.1: The innermost layer of the model is the container feature characterization. This step uses the 1D-CNN (one-dimensional convolutional neural network) model to extract container features. The existing technology model often uses a fully connected layer in the inner layer, which cannot extract the queue status features of the container in the node. Therefore, this method considers using a one-dimensional convolutional neural network to better capture the order of the container queue and the dependency relationship between the status information of the containers in the queue. The calculation process is recorded as:

[0173]

[0174] Where p represents the number of container features of the output node at timestamp t. Since a one-dimensional convolutional neural network is used, After input, the dimension of the characteristic number of the container in the result will be reduced, so p<n, the output result Will be applied to the next step.

[0175] Step 5.2: Cluster feature characterization, feature mapping based on graph attention network. The reason for this step is that the features between nodes are related, and the topological relationship between nodes is given by step 4. In order for the model to be able to perceive this topological relationship during the calculation process, a graph-based algorithm is needed to update the node status. The graph-based algorithm used in this method can assign different weights to different neighbor nodes of the node through the attention coefficient, thereby improving the algorithm performance.

[0176] First, calculate the normalized attention coefficient between nodes:

[0177]

[0178] Where exp represents the exponential function with natural logarithm as the base, LeakyReLU is the activation function, and W are the parameters that need to be trained, and T represents the transpose of the vector Represents the set of neighbor nodes of node i. Next, update the node status:

[0179]

[0180] The updated node state contains the information passed by the neighboring nodes and can be used for the next step of calculation.

[0181] Step 5.3: Time series feature extraction. After the calculation in the previous step, the new node data after the updated state is input into this step to extract the time autocorrelation of the node. In the solution of related technologies, LSTM will have gradient explosion or gradient disappearance problems as the historical time increases. Therefore, the more commonly used recurrent neural network GRU base is used here to build a new recurrent neural network that can sense weekly, daily, and hourly cycles:

[0182]

[0183] where r t 、z t , is an intermediate variable, W z , W r , W is the parameter that needs to be trained, h t-1 and h t is the hidden state of the previous layer of the recurrent neural network and the hidden state of the current layer. Through the above state update, the obtained h tThat is the hidden state input to the next step.

[0184] Step 5.4: Output forecasts for weekly, daily, and hourly data By performing the model calculation in step 5.3, three feature vectors can be obtained, which are concatenated and then connected to the linear fully connected layer to output the prediction results.

[0185] Step 6: Define the loss function, loop the training model, and output the prediction model. The final output is

[0186]

[0187] Step 7: Apply the model to the production environment. According to the new sequence data generated, dynamically train the model according to the above steps and update the model to adapt to the dynamically changing environment.

[0188] based on Figure 1-Figure 7 The embodiment shown, as Fig. 9 As shown, the present disclosure provides a schematic structural diagram of a node data acquisition device 900. The device includes a first processing module 901, a pre-processing module 902, and a second processing module 903.

[0189] The first processing module 901 is used to determine a first feature set based on original data of multiple nodes in the cluster within a preset time period, where the first feature set includes first feature data of a first node among the multiple nodes running a first container at each timestamp, and the first feature data includes basic data, operation data, and status data of the first container.

[0190] The preprocessing module 902 is used to preprocess the first feature data to obtain feature vectors of multiple preset time dimensions. The second feature set includes a first feature matrix of the first node running the first container at the first timestamp, and the first feature matrix includes first feature vectors of multiple preset time dimensions.

[0191] The second processing module 903 is used to determine a graph structure reflecting the relationship between nodes in the cluster through correlation calculation based on the first feature data. The second feature set and the graph structure are used to train a resource prediction model, and the resource prediction model is used to predict resource usage of nodes in the cluster.

[0192] In an embodiment of the present disclosure, the status data includes the life cycle and queue number of the first container. The first processing module is also used to determine the first feature set based on the original data of multiple nodes in the cluster within a preset time period, including: determining the life cycle of the first container based on the creation time and deletion time of the first container; based on the creation time, determining the queue number of multiple containers running on the first node of the first container, the queue number reflects the running order of the first container on the first node.

[0193] In an embodiment of the present disclosure, the preprocessing module is also used to perform format unification processing on the first feature data to obtain second feature data; the second feature data is divided into periods of week, day, and hour respectively, and the first feature vector of the first node running the first container at the first timestamp is determined, and the first feature matrix includes first feature vectors with periods of week, day, and hour respectively.

[0194] In an embodiment of the present disclosure, the second processing module is also used to determine the second feature data whose difference exceeds the first threshold as a negative sample based on the difference in feature data of the prediction indicator between the current timestamp and the previous timestamp, and the prediction indicator is one or more categories in the first feature data.

[0195] In an embodiment of the present disclosure, the second processing module is also used to cluster multiple containers in the cluster using unsupervised clustering to obtain multiple types of containers; for each type of container, the first feature data whose correlation with the prediction index meets the preset conditions is determined as the third feature data; based on the third feature data, the Pearson correlation coefficient is used to determine the correlation coefficient between the nodes; based on the correlation coefficient, a graph structure is constructed.

[0196] In an embodiment of the present disclosure, the second processing module is also used to resample the feature data in the second feature set to obtain a resampled data set, the proportion of negative samples in the resampled data set is greater than or equal to the second threshold, and the resampled data set includes resampled data of the multiple preset time dimensions; use the resampled data set to train the initial resource prediction model, and iterate until the training number is reached to obtain a resource prediction model.

[0197] In an embodiment of the present disclosure, the second processing module is also used to input the resampled data set into a one-dimensional convolutional neural network to extract container features of multiple containers running on a first node among multiple nodes; based on the graph structure and container features, using a graph attention network, update the state data of the first node, and the updated state data includes feature data of neighbor nodes of the first node; input the updated state data into a recurrent neural network to obtain second feature vectors of the multiple preset time dimensions; concatenate the second feature vectors of the multiple preset time dimensions and perform linear full connection to obtain a first prediction result; adjust the parameters of the initial resource prediction model according to the first prediction result and the first feature vector, and stop training when the number of iterations reaches a preset number to obtain a resource prediction model.

[0198] In summary, the node data acquisition device proposed in the embodiment of the present disclosure determines a first feature set based on the original data of multiple nodes in the cluster within a preset time period through a first processing module, the first feature set includes the first feature data of the first node among the multiple nodes running the first container at each timestamp, and the first feature data includes the basic data, operation data, and status data of the first container; the preprocessing module preprocesses the first feature data to obtain feature vectors of multiple preset time dimensions, the second feature set includes the first feature matrix of the first node running the first container at the first timestamp, and the first feature matrix includes the first feature vectors of multiple preset time dimensions; the second processing module determines the graph structure reflecting the node relationship in the cluster through correlation calculation based on the first feature data, the second feature set and the graph structure are used for training the resource prediction model, and the resource prediction model is used to predict the resource usage of the nodes in the cluster. On the basis of the collected original data, considering the characteristics of the container and the comprehensive periodic data of multiple time dimensions, a training data set is constructed, and the graph structure is determined by correlation, and the model is trained using a one-dimensional convolutional neural network, a graph attention network, and a recurrent neural network, so that the trained model can cope with the situation of sudden increase or decrease in resources.

[0199] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0200] Fig.10 is a block diagram of an electronic device 1000 for implementing the above-mentioned node data acquisition method according to an exemplary embodiment.

[0201] Reference Fig.10 The electronic device 1000 may include a communication interface 1001, which can interact with other devices; a processor 1002, which is connected to the communication interface 1001 to interact with other devices and is used to execute the method provided by one or more of the above technical solutions when running a computer program; and a memory 1003, on which the computer program is stored. Specifically, the specific processing process of the processor 1002 can refer to the container safety detection method described in the above embodiment of the present disclosure.

[0202] Of course, in actual application, the various components in the electronic device 1000 are coupled together through the bus system 1004. It can be understood that the bus system 1004 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1004 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 1004 .

[0203] The memory 1003 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 1000. Examples of such data include: any computer program used to operate on the electronic device 1000.

[0204] The method disclosed in the above embodiment of the present application can be applied to the processor 1002, or implemented by the processor 1002. The processor 1002 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1002. The above processor 1002 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1002 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 1003, and the processor 1002 reads the information in the memory 1003, and completes the steps of the above method in combination with its hardware.

[0205] In an exemplary embodiment, the electronic device 1000 can be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), general processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0206] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the node data acquisition method described in the above embodiments of the present disclosure.

[0207] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the node data acquisition method described in the above embodiment of the present disclosure when a processor executes the computer program.

[0208] An embodiment of the present disclosure also proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals include computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the node data acquisition method described in the above embodiment of the present disclosure.

[0209] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0210] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0211] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0212] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0213] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0214] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0215] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0216] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A node data acquisition method, characterized in that: The method comprises: Determine a first feature set based on raw data of multiple nodes in the cluster within a preset time period, where the first feature set includes first feature data of a first node among the multiple nodes running a first container at each timestamp, where the first feature data includes basic data, running data, and status data of the first container; Preprocessing the first feature data to obtain feature vectors of multiple preset time dimensions, wherein the second feature set includes a first feature matrix of the first node running the first container at the first timestamp, and the first feature matrix includes first feature vectors of the multiple preset time dimensions; Based on the first feature data, a graph structure reflecting the relationship between nodes in the cluster is determined through correlation calculation. The second feature set and the graph structure are used to train a resource prediction model, and the resource prediction model is used to predict resource usage of nodes in the cluster.

2. The method according to claim 1, characterized in that The state data includes the life cycle and queue number of the first container, and the determining of the first feature set based on the original data of multiple nodes in the cluster within a preset time period includes: Determining a life cycle of the first container based on a creation time and a deletion time of the first container; Based on the creation time, a queue number among multiple containers in which the first container is run on the first node is determined, where the queue number reflects a running order of the first container on the first node.

3. The method according to claim 1 or 2, characterized in that: The preprocessing of the first characteristic data comprises: Performing format unification processing on the first characteristic data to obtain second characteristic data; The second feature data is divided into periods of week, day and hour respectively, and a first feature vector of the first node running the first container at the first timestamp is determined, wherein the first feature matrix includes first feature vectors with periods of week, day and hour respectively.

4. The method according to claim 3, characterized in that The method further comprises: Based on the difference of the feature data between the current timestamp and the previous timestamp of the prediction indicator, the second feature data whose difference exceeds the first threshold is determined as a negative sample, and the prediction indicator is one or more categories in the first feature data.

5. The method according to claim 1 or 2, characterized in that: The determining, based on the first feature data and by correlation calculation, a graph structure reflecting the node relationship within the cluster includes: Clustering multiple containers in the cluster using unsupervised clustering to obtain multiple types of containers; For each type of container, determining the first characteristic data whose correlation with the prediction index meets the preset condition as the third characteristic data; Based on the third feature data, determining the correlation coefficient between nodes using the Pearson correlation coefficient; Based on the correlation coefficient, the graph structure is constructed.

6. The method according to claim 4, characterized in that The method further comprises: Resampling the feature data in the second feature set to obtain a resampled data set, wherein the proportion of the negative example samples in the resampled data set is greater than or equal to a second threshold, and the resampled data set includes the resampled data of the multiple preset time dimensions; The initial resource prediction model is trained using the resampled data set, and the training is repeated until the number of training times is reached to obtain the resource prediction model.

7. The method according to claim 6, characterized in that The using the resampled data set to train the initial resource prediction model comprises: Inputting the resampled data set into a one-dimensional convolutional neural network to extract container features of a plurality of containers running on a first node among the plurality of nodes; Based on the graph structure and the container features, using a graph attention network, updating the state data of the first node, wherein the updated state data includes feature data of neighbor nodes of the first node; Inputting the updated state data into a recurrent neural network to obtain second feature vectors of the plurality of preset time dimensions; The second feature vectors of the plurality of preset time dimensions are concatenated and then linearly fully connected to obtain a first prediction result; According to the first prediction result and the first feature vector, the parameters of the initial resource prediction model are adjusted until the number of iterations reaches a preset number, and the training is stopped to obtain the resource prediction model.

8. A node data acquisition device, characterized in that: include: A first processing module, configured to determine a first feature set based on raw data of multiple nodes in the cluster within a preset time period, wherein the first feature set includes first feature data of a first node among the multiple nodes running a first container at each timestamp, and the first feature data includes basic data, running data, and status data of the first container; A preprocessing module, configured to preprocess the first feature data to obtain feature vectors of multiple preset time dimensions, wherein the second feature set includes a first feature matrix of the first node running the first container at a first timestamp, and the first feature matrix includes first feature vectors of the multiple preset time dimensions; The second processing module is used to determine a graph structure reflecting the relationship between nodes in the cluster through correlation calculation based on the first feature data. The second feature set and the graph structure are used to train a resource prediction model, and the resource prediction model is used to predict resource usage of nodes in the cluster.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 after being executed.

Images