Data center resource analysis system and method, storage medium and electronic equipment

Abstract

The present invention provides a data center resource analysis system and method, storage medium and electronic device, including: a data construction layer extracts configuration data and performance data from offline data and real-time data of data center resources; processes configuration data and performance data to form analysis index data; a pattern analysis layer obtains a first training data set and a second training data set; builds a clustering model, a classification model and a prediction model based on the two training data sets; applies the three models to analyze the analysis index data to obtain a first analysis result, a second analysis result and a third analysis result; a comprehensive evaluation layer is used to evaluate the three results respectively, and generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information. The system is used to build a model and use the model to conduct an in-depth analysis of data center resources, and then the analysis results output by the model are used to reasonably evaluate the construction plan, capacity adjustment information and expected equipment demand of data center resources.

Description

Technical Field

[0001] The present invention relates to the technical field of data center resource analysis, and in particular to a data center resource analysis system and method, a storage medium and an electronic device. Background Art

[0002] With the rapid development of information technology, the business scale is gradually expanding, the software functions are constantly increasing, and the demand for holidays is increasing sharply. These problems have put forward more and higher requirements for the resource construction of data centers. In the traditional construction process of data centers, judgment and planning are mostly based on manual experience. In the actual operation process, a large amount of resources will be wasted, and it is impossible to meet the unexpected sudden increase in demand. Therefore, the refined management of underlying resources has emerged, providing resource managers with more information and perspectives from a data-driven perspective.

[0003] In the operation process of the data center, in order to improve the service level for users, various operation and maintenance analysis indicator data of the server are collected. During the operation process, most of the work is to monitor the performance indicators for alarms to ensure the high-quality and stable operation of user services, but there is no in-depth mining and analysis of the overall overview of resource usage and user usage. The overall resource management lacks a refined management model, the resources of offline software are not recovered in time, and the resources are not expanded in time for the new software requirements, resulting in unreasonable resource allocation and waste of resources. Summary of the invention

[0004] In view of this, the present invention provides a data center resource analysis system and method, storage medium and electronic device, through which an in-depth analysis of data center resources can be performed, and the construction plan, capacity adjustment information and expected equipment requirements of data center resources can be reasonably evaluated.

[0005] A data center resource analysis system, comprising:

[0006] Data construction layer, model analysis layer and comprehensive evaluation layer;

[0007] The data construction layer is used to obtain offline data and real-time data of data center resources, and extract configuration data of the offline data and performance data of the real-time data; process the configuration data and the performance data to form analysis index data, and the analysis index data includes cross-sectional data and time series data;

[0008] The pattern analysis layer is used to obtain a preset first training data set and a second training data set, wherein the first training data set includes a plurality of first training data, the second training data set includes a plurality of second training data, the first training data is cross-sectional data for generating a model, and the second training data is time series data for generating a model; based on the first training data set and the second training data set, a clustering model, a classification model and a prediction model are constructed; the clustering model, the classification model and the prediction model are applied to analyze the analysis index data, and a first analysis result, a second analysis result and a third analysis result corresponding to the analysis index data outputted by the clustering model, the classification model and the prediction model respectively are obtained;

[0009] The comprehensive evaluation layer is used to evaluate the first analysis result, the second analysis result and the third analysis result respectively, and generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information corresponding to the data center resources.

[0010] The above-mentioned data center resource analysis system, optionally, the data construction layer includes:

[0011] Data parsing module and indicator enrichment module;

[0012] The data analysis module is used to obtain offline data from a preset relational database and extract configuration data from the offline data; obtain real-time data from a preset queue and extract performance data from the real-time data;

[0013] The indicator enrichment module is used to obtain preset cross-sectional data tags and time series data tags, add the cross-sectional data tags to the configuration data to obtain cross-sectional data; and add the time series data tags to the performance data to obtain time series data.

[0014] In the above data center resource analysis system, optionally, the pattern analysis layer includes:

[0015] Feature building module and algorithm analysis module;

[0016] The feature construction module is used to refine and process the cross-sectional data and the time series data to obtain a plurality of user indicator features and a plurality of data indicator features corresponding to the cross-sectional data, as well as a plurality of user indicator features and a plurality of data indicator features corresponding to the time series data; and store each of the user indicator features and each of the data indicator features in a preset data feature pool;

[0017] The algorithm analysis module is used to obtain user indicator features and data indicator features corresponding to each of the first training data, as well as user indicator features and data indicator features corresponding to each of the second training data; apply the data indicator features corresponding to each of the first training data to build a clustering model; apply the user indicator features and data indicator features corresponding to each of the first training data, as well as the user indicator features and data indicator features corresponding to each of the second training data to build a classification model; apply the data indicator features corresponding to each of the second training data to build a prediction model; extract the user indicator features and data indicator features corresponding to the cross-sectional data in the data feature pool, as well as the user indicator features and data indicator features corresponding to the time series data in the data feature pool; input the data indicator features corresponding to the cross-sectional data into the clustering model to obtain a first analysis result output by the clustering model; input the user indicator features and data indicator features corresponding to the cross-sectional data, as well as the user indicator features and data indicator features corresponding to the time series data in the data feature pool into the classification model to obtain a second analysis result output by the classification model; input the data indicator features corresponding to the time series data in the data feature pool into the prediction model to obtain a third analysis result output by the prediction model.

[0018] The above-mentioned data center resource analysis system, optionally, the algorithm analysis module is specifically used to perform dimensionality reduction processing on the data indicator features corresponding to each of the first training data to obtain cluster data; iteratively calculate the distance between each of the cluster data, and merge the cluster data with a distance less than a preset distance into the same cluster cluster, until the distance between each cluster data between any cluster clusters and each cluster data between other cluster clusters is not less than the preset distance, to obtain the clustering results corresponding to each of the cluster data; and build a clustering model based on the clustering results.

[0019] The above-mentioned data center resource analysis system, optionally, the algorithm analysis module is specifically used to construct a third training data set based on the first training data set and the second training data set, the third training data set comprising multiple third training data, each of the third training data consisting of a first training data and a second training data; set a data label corresponding to each of the third training data; and construct a classification model based on each of the third training data, the data label corresponding to each of the third training data and a preset loss function.

[0020] The above-mentioned data center resource analysis system, optionally, the algorithm analysis module is specifically used to input the data indicator characteristics corresponding to each second training data into a preset initial prediction model to obtain the training results output by the initial prediction model; determine whether the training results meet the preset training conditions, if not, adjust the model parameters of the initial prediction model, and continue to input the data indicator characteristics corresponding to each second training data into the initial prediction model until the training results currently output by the prediction model meet the training conditions, thereby obtaining the final prediction model.

[0021] The above-mentioned data center resource analysis system, optionally, the comprehensive evaluation layer includes: a resource partitioning module, a user profiling module and a demand evaluation module;

[0022] The resource partitioning module is used to evaluate the first analysis result, generate a resource partitioning construction plan corresponding to the data center resources, and output the resource partitioning construction plan to a preset server;

[0023] The user profiling module is used to evaluate the second analysis result, determine the user's usage information of the data center resources, generate resource capacity adjustment information corresponding to the data center resources, and output the resource capacity adjustment information to the server;

[0024] The demand evaluation module is used to evaluate the third analysis result, generate resource equipment demand information corresponding to the data center resources, and output the resource equipment demand information to the server.

[0025] A data center resource analysis method, the method is applied to a data center resource analysis system, the method comprising:

[0026] Obtain offline data and real-time data of data center resources, extract configuration data of the offline data, and extract performance data of the real-time data;

[0027] Processing the configuration data and the performance data to form analysis index data, wherein the analysis index data includes cross-sectional data and time series data;

[0028] Acquire a preset first training data set and a second training data set, wherein the first training data includes a plurality of first training data, the second training data includes a plurality of second training data, the first training data is cross-sectional data for generating a model, and the second training data is time series data for generating a model;

[0029] Based on the first training data set and the second training data set, construct a clustering model, a classification model and a prediction model;

[0030] Applying the clustering model, the classification model and the prediction model to analyze the analysis index data, and obtaining a first analysis result, a second analysis result and a third analysis result corresponding to the analysis index data outputted by the clustering model, the classification model and the prediction model respectively;

[0031] The first analysis result, the second analysis result and the third analysis result are evaluated respectively to generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information corresponding to the data center resources.

[0032] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned data center resource analysis method.

[0033] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to perform the above-mentioned data center resource analysis method.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] The present invention provides a data center resource analysis system, comprising: a data construction layer for acquiring offline data and real-time data of data center resources, and extracting configuration data of the offline data, and extracting performance data of the real-time data; processing the configuration data and the performance data to form analysis index data, wherein the analysis index data includes cross-sectional data and time series data; a pattern analysis layer for acquiring a preset first training data set and a second training data set, wherein the first training data includes a plurality of first training data, and the second training data includes a plurality of second training data, wherein the first training data is cross-sectional data for generating a model, and the second training data is time series data for generating a model; based on the first training data set and the second training data set, a clustering model, a classification model and a prediction model are constructed; the clustering model, the classification model and the prediction model are applied to analyze the analysis index data, and a first analysis result, a second analysis result and a third analysis result corresponding to the analysis index data outputted by the clustering model, the classification model and the prediction model are obtained; a comprehensive evaluation layer is used to evaluate the first analysis result, the second analysis result and the third analysis result respectively, and generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information corresponding to the data center resources. The system provided by the present invention constructs a model through the pattern analysis layer and applies the model to conduct an in-depth analysis of data center resources, and then reasonably evaluates the construction plan, capacity adjustment information and expected equipment requirements of data center resources through the comprehensive evaluation layer based on the analysis results of the model output. The method provided by the present invention is applied, and the "AI+" method is introduced to analyze the core characteristic indicators, and hierarchical clustering, discriminant classification, and trend prediction algorithms are combined to construct clustering, classification and prediction models, organically combining the expert experience perspective with the data-driven perspective, understanding the resource usage, drawing a portrait of the software user's resource usage, and predicting future usage needs, providing an important reference basis for the life cycle management of resource equipment, resource planning, and software deployment, and forming a data model and evaluation method based on IT operation and maintenance resource data. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 A system structure diagram of a data center resource analysis system provided by an embodiment of the present invention;

[0038] Figure 2 Another system structure diagram of a data center resource analysis system provided by an embodiment of the present invention;

[0039] Figure 3 Another system structure diagram of a data center resource analysis system provided by an embodiment of the present invention;

[0040] Figure 4 A method flow chart of a data center resource analysis method provided by an embodiment of the present invention;

[0041] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0044] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.

[0045] The embodiment of the present invention provides a data center resource analysis system, the system structure diagram of which is as follows: Figure 1 As shown, specifically including:

[0046] Data construction layer 100, pattern analysis layer 200 and comprehensive evaluation layer 300;

[0047] The data construction layer 100 is used to obtain offline data and real-time data of data center resources, and extract configuration data of the offline data and performance data of the real-time data; process the configuration data and the performance data to form analysis index data, and the analysis index data includes cross-sectional data and time series data;

[0048] The pattern analysis layer 200 is used to obtain a preset first training data set and a second training data set, wherein the first training data set includes a plurality of first training data, the second training data set includes a plurality of second training data, the first training data is cross-sectional data for generating a model, and the second training data is time series data for generating a model; based on the first training data set and the second training data set, a clustering model, a classification model and a prediction model are constructed; the clustering model, the classification model and the prediction model are applied to analyze the analysis index data, and a first analysis result, a second analysis result and a third analysis result corresponding to the analysis index data outputted by the clustering model, the classification model and the prediction model respectively are obtained;

[0049] The comprehensive evaluation layer 300 is used to evaluate the first analysis result, the second analysis result and the third analysis result respectively, and generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information corresponding to the data center resources.

[0050] In the data center resource analysis system provided by the embodiment of the present invention, the data construction layer mainly extracts the required configuration data and performance data from the offline data and the real-time data. The performance data is actually the index performance data. The cross-sectional data and time series data are obtained by analyzing the configuration data and the performance data. For details, refer to Figure 2The relationship between each data in the data construction layer is shown in Figure 1. In the data planning process, offline data is mainly used to extract configuration information, and real-time data is used for performance monitoring. Therefore, the data construction layer needs to integrate configuration and performance information, enrich the multi-dimensional information of resource data, and form standard resource cross-sectional data and resource time series data to provide subsequent data mining algorithm analysis. The pattern analysis layer mainly builds clustering models, classification models and prediction models by further deep mining cross-sectional data and time series data. Among them, the clustering model classifies and organizes different resource objects from a high-dimensional perspective, clarifies the category status of resource objects, and provides digital information assistance for the subsequent construction and deployment of basic resources; the classification model classifies the user's resource usage, understands the overall resource usage from a macro perspective, and provides a model construction analysis basis for subsequent resource recovery and resource allocation; the prediction model is used to analyze the resource needs of resource users in the future and provide guidance and suggestions for computer room construction, resource planning, and software deployment. Therefore, each model in the pattern analysis layer can output analysis results such as the corresponding categories, usage, and equipment requirements based on the data center resources. Finally, the comprehensive evaluation layer conducts further explanatory analysis based on the first analysis results, the second analysis results, and the third analysis results to form a resource-refined operation evaluation model for scenarios such as resource partitioning construction, user resource usage analysis, and future demand forecasting, and output corresponding results.

[0051] Among them, the cross-sectional data information mainly includes user software information and server-related configuration information, and the time series data information mainly includes server performance monitoring data.

[0052] By using the data center resource analysis system provided by the embodiment of the present invention, it is possible to conduct an in-depth analysis of data center resources and reasonably evaluate the construction plan, capacity adjustment information and expected equipment requirements of data center resources.

[0053] refer to Figure 3 In the data center resource analysis system provided by the embodiment of the present invention, the data construction layer includes:

[0054] Data analysis module 101 and indicator enrichment module 102;

[0055] The data analysis module 101 is used to obtain offline data from a preset relational database and extract configuration data from the offline data; obtain real-time data from a preset queue and extract performance data from the real-time data;

[0056] The indicator enrichment module 101 is used to obtain preset cross-sectional data tags and time series data tags, add the cross-sectional data tags to the configuration data to obtain cross-sectional data; and add the time series data tags to the performance data to obtain time series data.

[0057] In the embodiment of the present invention, the main function of the data analysis module is to extract configuration data and performance data. Since the configuration data of resources has a long change cycle, the configuration data can be obtained from the relational database; for performance data, the performance status fluctuates in real time, so data integration needs to obtain streaming data from the queue kafka and offline data from the big data time series database to form performance data together. The indicator enrichment module processes and integrates the configuration data and performance data, enriches the indicator information, and classifies the analysis indicator data into cross-sectional data information summarized by multiple users and time series change information of performance indicators in a single user dimension, that is, cross-sectional data and time series data, providing basic data indicators for subsequent model analysis work.

[0058] Specifically, the indicator enrichment module further processes and organizes the multi-indicator resource data output by the data analysis module to form the data feature information required in the subsequent algorithm analysis process. By summarizing the characteristics of each dimension of the data and integrating the multi-indicator dimensional feature information of the resource data, it finally forms enriched cross-sectional data and time series data, completing the data construction work required for the subsequent modules.

[0059] refer to Figure 3 In the data center resource analysis system provided by the embodiment of the present invention, the pattern analysis layer includes:

[0060] Feature construction module 201 and algorithm analysis module 202;

[0061] The feature construction module 201 is used to refine and process the cross-sectional data and the time series data to obtain a plurality of user indicator features and a plurality of data indicator features corresponding to the cross-sectional data, and a plurality of user indicator features and a plurality of data indicator features corresponding to the time series data; and store each of the user indicator features and each of the data indicator features in a preset data feature pool;

[0062] The algorithm analysis module 202 is used to obtain the user indicator features and data indicator features corresponding to each of the first training data, as well as the user indicator features and data indicator features corresponding to each of the second training data; apply the data indicator features corresponding to each of the first training data to build a clustering model; apply the user indicator features and data indicator features corresponding to each of the first training data, as well as the user indicator features and data indicator features corresponding to each of the second training data to build a classification model; apply the data indicator features corresponding to each of the second training data to build a prediction model; extract the user indicator features and data indicator features corresponding to the cross-sectional data in the data feature pool, as well as the user indicator features and data indicator features corresponding to the time series data in the data feature pool; input the data indicator features corresponding to the cross-sectional data into the clustering model to obtain a first analysis result output by the clustering model; input the user indicator features and data indicator features corresponding to the cross-sectional data, as well as the user indicator features and data indicator features corresponding to the time series data in the data feature pool into the classification model to obtain a second analysis result output by the classification model; input the data indicator features corresponding to the time series data in the data feature pool into the prediction model to obtain a third analysis result output by the prediction model.

[0063] It should be noted that the feature construction module divides the cross-sectional data and time series data into two aspects based on the cross-sectional data and time series data output by the data construction layer: user indicator features and data indicator features. User indicator features include user software information such as software name, software type, software partition, software function, change level, server category, server status, server level, network partition, etc. Data indicator features include original features and derived features. Original features include the number of resources, number of CPU cores, CPU usage, memory size, memory usage, disk capacity, disk space usage, etc.; derived features include CPU usage mean, CPU usage variance, CPU usage peak, CPU usage extreme, CPU redundancy ratio, CPU elasticity, memory usage mean, memory usage variance, memory usage peak, memory usage extreme, memory redundancy ratio, memory elasticity, disk space usage mean, disk space usage variance, disk space usage peak, disk space usage extreme, disk space redundancy ratio, disk space elasticity, performance monitoring indicator elasticity, performance monitoring indicator redundancy ratio and performance monitoring indicator usage, etc.

[0064] The performance monitoring indicator elasticity rate and the performance monitoring indicator redundancy ratio are calculated by using the performance monitoring indicator utilization rate. The calculation formula of the performance monitoring indicator elasticity rate is:

[0065]

[0066] The calculation formula for the performance monitoring indicator redundancy ratio is:

[0067]

[0068] r1 represents the elasticity rate of the performance monitoring indicator, r2 represents the redundancy ratio of the performance monitoring indicator, and x t Indicates the usage rate of performance monitoring indicators at time t.

[0069] Specifically, for the construction process of the clustering model, the algorithm analysis module is used to perform dimensionality reduction processing on the data indicator features corresponding to each of the first training data to obtain clustering data; iteratively calculate the distance between each of the clustering data, and merge the clustering data with a distance less than a preset distance into the same cluster cluster, until the distance between each clustering data between any clustering clusters and each clustering data between other clustering clusters is not less than the preset distance, and obtain the clustering results corresponding to each of the clustering data; and build a clustering model based on the clustering results.

[0070] Furthermore, during the clustering process, data indicator features corresponding to all the first training data can be extracted, or only the number of resources, number of CPU cores, memory size, disk capacity, CPU redundancy ratio, memory redundancy ratio, disk space redundancy ratio, CPU elasticity rate, memory elasticity rate, and disk space elasticity rate among the data indicator features corresponding to the first training data can be extracted.

[0071] The clustering algorithm is mainly used in the scenario of resource zoning construction. According to the clustering analysis of multidimensional cross-sectional indicators, the multidimensional data of user resource attributes and cost information is reduced in dimension. According to the clustering features extracted from the cross-sectional data, a bottom-up aggregation strategy is used for hierarchical analysis. In the initial process of the algorithm, each cluster data is used as the original cluster cluster, the distance between the two cluster data is calculated, the two clusters with the closest distance are merged, and this step is repeated until the expected number of clusters is reached. By selecting a variety of distance calculation methods such as minimum distance, maximum distance, average distance, and Euclidean distance, the clustering results are weighted and summarized, and finally the classification result with the largest weight is used as the clustering judgment result. The specific distance calculation formula is as follows:

[0072] Minimum distance:

[0073] Maximum distance:

[0074]

[0075] Average distance:

[0076]

[0077] Euclidean distance:

[0078]

[0079] Among them, x and z represent the multi-dimensional cross-sectional index information vectors of two instance objects.

[0080] In an embodiment of the present invention, the algorithm analysis module is specifically used to construct a third training data set based on the first training data set and the second training data set, wherein the third training data set includes multiple third training data, and each of the third training data is composed of a first training data and a second training data; set a data label corresponding to each of the third training data; and construct a classification model based on each of the third training data, the data label corresponding to each of the third training data, and a preset loss function.

[0081] Specifically, the classification algorithm is mainly used for user portrait scenario goals. Based on multi-dimensional cross-sectional data and time series data information, it comprehensively analyzes the user's resource usage and makes reasonable resource planning suggestions to the user based on the classification results. By constructing a gradient boosting tree for node attribute division based on each classification indicator and classification label information, the classification process uses the gradient boosting algorithm. The specific algorithm process is as follows:

[0082] First, define the training data set T = {(x1, y1), (x2, y2), …, (x N ,y N )}, loss function L = (y, f(x));

[0083] Initialization function:

[0084]

[0085] Calculate for m=1,2,…M,i=1,2,…N

[0086]

[0087] R mi Fit the regression tree and get the leaf node region R of the mth tree mj ,j=1,2,…,J

[0088] For j = 1, 2, ..., J, calculate

[0089]

[0090] renew

[0091] Finally, we get the regression tree

[0092]

[0093] Among them, x1,x2,…x Nis the third training data, which is composed of the first training data and the second training data, and contains the user indicator features and data indicator features corresponding to the first training data and the second training data; y1, y2, ...y N are the labels corresponding to each third training data respectively; Cmj is the defined parameter, the jth index of the mth tree (the number of m and j is selected according to the input data solution) The solution process of Cmj is based on the residual between the true value and the estimated value, and the parameter that minimizes the residual is selected in the iteration process as the solution result. m and j are the number of decision trees in the classification decision tree solution process and the specific data indicators involved in the classification process. In the application process, the input indicator x of the model may have N dimensions (N>m). According to y (label) and the learning process of the indicator, m decision trees will be constructed, and the classification results (A / B / C...) will be obtained for the unlabeled data according to the decision tree model.

[0094] It should be noted that a classification model can be constructed by selecting user qualitative feature information such as software type, software partition, software function, change level, server category, server status, server level, network partition, and user indicator characteristics and data indicator characteristics such as CPU usage mean, CPU usage variance, CPU usage peak, CPU usage extreme, memory usage mean, memory usage variance, memory usage peak, memory usage extreme, disk space usage mean, disk space usage variance, disk space usage peak, and disk space usage extreme.

[0095] In an embodiment of the present invention, the algorithm analysis module is specifically used to input the data indicator features corresponding to each of the second training data into a preset initial prediction model to obtain the training results output by the initial prediction model; determine whether the training results meet the preset training conditions, and if not, adjust the model parameters of the initial prediction model, and continue to input the data indicator features corresponding to each of the second training data into the initial prediction model until the training results currently output by the prediction model meet the training conditions, thereby obtaining the final prediction model.

[0096] Specifically, the prediction algorithm is mainly used for the user's future demand prediction scenario goal. According to the distribution characteristics of historical data, an autoregressive moving average model is constructed for the time series data information to predict the future trend of each time series data, providing a quantitative reference for subsequent resource capacity planning. The autoregressive moving average model can be expressed as:

[0097]

[0098] Among them, E(ε t )=0,

[0099] The estimation process of the model is as follows:

[0100] First, solve the autocorrelation coefficient of the model:

[0101]

[0102] The next step is to solve the partial autocorrelation coefficient:

[0103]

[0104] in,

[0105]

[0106] Estimate values ​​of unknown parameters in the model:

[0107]

[0108]

[0109] in, is the autoregressive coefficient, θ is the moving average coefficient, ε is the residual term of the fit, and μ is x i (1=1,2…n), ρ is the autocorrelation coefficient, which is calculated based on the sequence x, and D is the matrix composed of the autocorrelation coefficients.

[0110] It should be noted that the prediction model is constructed through data indicator characteristics such as CPU usage, CPU usage, memory usage, memory usage, disk space usage, and disk space usage.

[0111] refer to Figure 3 In the data center resource analysis system provided by the embodiment of the present invention, the comprehensive evaluation layer includes: a resource partitioning module 301, a user portrait module 302 and a demand evaluation module 303;

[0112] The resource partitioning module 301 is used to evaluate the first analysis result, generate a resource partitioning construction plan corresponding to the data center resources, and output the resource partitioning construction plan to a preset server;

[0113] The user profile module 302 is used to evaluate the second analysis result, determine the user's usage information of the data center resources, generate resource capacity adjustment information corresponding to the data center resources, and output the resource capacity adjustment information to the server;

[0114] The demand evaluation module 303 is used to evaluate the third analysis result, generate resource equipment demand information corresponding to the data center resources, and output the resource equipment demand information to the server.

[0115] It should be noted that the resource partitioning module is mainly to further evaluate and analyze the first analysis result output by the clustering model. Through the analysis results of the model, the user scale of computing-intensive, memory-intensive, and IO-intensive users is summarized to provide data support for the subsequent refined resource partitioning construction. Based on the classification model, the user portrait module uses the results of user usage to recover resources for users with long-term low usage; reasonably allocates resources for users with stable resource usage growth; and expands capacity for users with rapid or sudden resource usage growth. Through further analysis of resource analysis indicator data, the user's usage behavior can be dynamically perceived to improve the service quality for users. The main function of the evaluation module is to be applied to the future resource planning scenario of the data center. By analyzing the time series data of the long-term usage of user resources, combined with multi-dimensional time series characteristics, a prediction model of the algorithm analysis layer is constructed, and the prediction results of the final model are used to plan the demand for future underlying basic equipment, provide digital-driven guidance and suggestions for the subsequent room planning, resource balancing, and reasonable deployment of software systems in the data center, and better apply the model results to the refined operation of data center resources.

[0116] The data center resource analysis system provided by the embodiment of the present invention can be applied to hardware devices such as application servers, network switches, routers, switches, load balancers, etc., and can perform information aggregation, intelligent analysis, and monitoring display on software resources such as databases, middleware, and big data. Traditional operation methods mostly rely on historical metadata information, lacking dynamic analysis and deployment adjustment of resource usage. Through the information integration of metadata, master data, and monitoring data, the single dimension of configuration data is expanded to integrated performance monitoring data information, and cross-sectional data and time series data information are aggregated to jointly construct multi-dimensional indicator features. The "AI+" method is introduced to analyze the core feature indicators, combined with hierarchical clustering, discriminant classification, and trend prediction algorithms, to construct a multi-indicator data evaluation model, organically combining the expert experience perspective with the data-driven perspective, understanding the resource usage situation, drawing a portrait of software user resource usage, and predicting future usage needs, providing an important reference basis for the life cycle management, resource planning, and software deployment of resource equipment, and forming a data model and evaluation method based on IT operation and maintenance resource data.

[0117] The data center resource analysis system provided by the present invention has a clear structure, clear hierarchical boundaries, and a pattern analysis layer with high scalability. When other algorithm models are introduced, the process structure design of the overall device is not affected. The data model construction method can be quickly applied and deployed to resource cost analysis scenarios in other fields.

[0118] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0119] The embodiment of the present invention provides a data center resource analysis method, which is applied to a data center resource analysis system. The method flow chart of the method is as follows: Figure 4 As shown, specifically including:

[0120] S401: Obtain offline data and real-time data of data center resources, extract configuration data of the offline data, and extract performance data of the real-time data.

[0121] In the embodiment of the present invention, offline data is acquired from a preset relational database, and configuration data is extracted from the offline data; real-time data is acquired from a preset queue, and performance data is extracted from the real-time data.

[0122] S402: Process the configuration data and the performance data to form analysis indicator data, where the analysis indicator data includes cross-sectional data and time series data.

[0123] In an embodiment of the present invention, a preset cross-sectional data tag and a time series data tag are obtained, and the cross-sectional data tag is added to the configuration data to obtain the cross-sectional data; and the time series data tag is added to the performance data to obtain the time series data.

[0124] S403: Obtain a preset first training data set and a second training data set, wherein the first training data includes a plurality of first training data, and the second training data set includes a plurality of second training data, the first training data is cross-sectional data for generating a model, and the second training data is time series data for generating a model.

[0125] S404: Constructing a clustering model, a classification model and a prediction model based on the first training data set and the second training data set.

[0126] In an embodiment of the present invention, the cross-sectional data and time series data are refined and processed to obtain a plurality of user indicator features and a plurality of data indicator features corresponding to the cross-sectional data, as well as a plurality of user indicator features and a plurality of data indicator features corresponding to the time series data; each of the user indicator features and each of the data indicator features are stored in a preset data feature pool; the user indicator features and the data indicator features corresponding to each of the first training data, as well as the user indicator features and the data indicator features corresponding to each of the second training data are obtained; the data indicator features corresponding to each of the first training data are applied to construct a clustering model; the user indicator features and the data indicator features corresponding to each of the first training data, as well as the user indicator features and the data indicator features corresponding to each of the second training data are applied to construct a classification model ; Apply the data indicator features corresponding to each of the second training data to build a prediction model; extract the user indicator features and data indicator features corresponding to the cross-sectional data in the data feature pool, as well as the user indicator features and data indicator features corresponding to the time series data in the data feature pool; input the data indicator features corresponding to the cross-sectional data into the clustering model to obtain a first analysis result output by the clustering model; input the user indicator features and data indicator features corresponding to the cross-sectional data, as well as the user indicator features and data indicator features corresponding to the time series data in the data feature pool into the classification model to obtain a second analysis result output by the classification model; input the data indicator features corresponding to the time series data in the data feature pool into the prediction model to obtain a third analysis result output by the prediction model.

[0127] In an embodiment of the present invention, dimension reduction processing is performed on the data indicator features corresponding to each of the first training data to obtain cluster data; the distance between each of the cluster data is iteratively calculated, and the cluster data with a distance less than a preset distance are merged into the same cluster cluster, until the distance between each cluster data between any cluster clusters and each cluster data between other cluster clusters is not less than the preset distance, and the clustering results corresponding to each of the cluster data are obtained; and a clustering model is constructed based on the clustering results.

[0128] In an embodiment of the present invention, a third training data set is constructed based on the first training data set and the second training data set, wherein the third training data set includes a plurality of third training data, and each of the third training data is composed of a first training data and a second training data; a data label corresponding to each of the third training data is set; and a classification model is constructed based on each of the third training data, the data label corresponding to each of the third training data, and a preset loss function.

[0129] In an embodiment of the present invention, the data indicator features corresponding to each of the second training data are input into a preset initial prediction model to obtain a training result output by the initial prediction model; it is determined whether the training result meets the preset training conditions; if not, the model parameters of the initial prediction model are adjusted, and the data indicator features corresponding to each of the second training data are continuously input into the initial prediction model until the training result currently output by the prediction model meets the training conditions, thereby obtaining a final prediction model.

[0130] S405: applying the clustering model, the classification model and the prediction model to analyze the analysis index data, and obtaining a first analysis result, a second analysis result and a third analysis result corresponding to the analysis index data outputted by the clustering model, the classification model and the prediction model respectively;

[0131] S406: Evaluate the first analysis result, the second analysis result and the third analysis result respectively, and generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information corresponding to the data center resources.

[0132] In an embodiment of the present invention, the first analysis result is evaluated to generate a resource partitioning construction plan corresponding to the data center resources, and the resource partitioning construction plan is output to a preset server; the second analysis result is evaluated to determine user usage information of the data center resources, generate resource capacity adjustment information corresponding to the data center resources, and output the resource capacity adjustment information to the server; the third analysis result is evaluated to generate resource equipment demand information corresponding to the data center resources, and output the resource equipment demand information to the server.

[0133] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned data center resource analysis method.

[0134] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Figure 5 As shown, it specifically includes a memory 501 and one or more instructions 502, wherein the one or more instructions 502 are stored in the memory 501 and are configured to be executed by one or more processors 503 to perform the following operations:

[0135] Obtain offline data and real-time data of data center resources, extract configuration data of the offline data, and extract performance data of the real-time data;

[0136] Processing the configuration data and the performance data to form analysis index data, wherein the analysis index data includes cross-sectional data and time series data;

[0137] Acquire a preset first training data set and a second training data set, wherein the first training data includes a plurality of first training data, the second training data includes a plurality of second training data, the first training data is cross-sectional data for generating a model, and the second training data is time series data for generating a model;

[0138] Based on the first training data set and the second training data set, construct a clustering model, a classification model and a prediction model;

[0139] Applying the clustering model, the classification model and the prediction model to analyze the analysis index data, and obtaining a first analysis result, a second analysis result and a third analysis result corresponding to the analysis index data outputted by the clustering model, the classification model and the prediction model respectively;

[0140] The first analysis result, the second analysis result and the third analysis result are evaluated respectively to generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information corresponding to the data center resources.

[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0142] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0143] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0144] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data center resource analysis system, characterized in that: include: Data construction layer, model analysis layer and comprehensive evaluation layer; The data construction layer is used to obtain offline data and real-time data of data center resources, and extract configuration data of the offline data and performance data of the real-time data; process the configuration data and the performance data to form analysis index data, and the analysis index data includes cross-sectional data and time series data; The pattern analysis layer is used to obtain a preset first training data set and a second training data set, wherein the first training data set includes a plurality of first training data, the second training data set includes a plurality of second training data, the first training data is cross-sectional data for generating a model, and the second training data is time series data for generating a model; based on the first training data set and the second training data set, a clustering model, a classification model and a prediction model are constructed; the clustering model, the classification model and the prediction model are applied to analyze the analysis index data, and a first analysis result, a second analysis result and a third analysis result corresponding to the analysis index data outputted by the clustering model, the classification model and the prediction model respectively are obtained; The comprehensive evaluation layer is used to evaluate the first analysis result, the second analysis result and the third analysis result respectively, and generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information corresponding to the data center resources.

2. The data center resource analysis system according to claim 1, characterized in that: The data construction layer includes: Data parsing module and indicator enrichment module; The data analysis module is used to obtain offline data from a preset relational database and extract configuration data from the offline data; obtain real-time data from a preset queue and extract performance data from the real-time data; The indicator enrichment module is used to obtain preset cross-sectional data tags and time series data tags, add the cross-sectional data tags to the configuration data to obtain cross-sectional data; and add the time series data tags to the performance data to obtain time series data.

3. The data center resource analysis system according to claim 1 or 2, characterized in that: The pattern analysis layer includes: Feature building module and algorithm analysis module; The feature construction module is used to refine and process the cross-sectional data and the time series data to obtain a plurality of user indicator features and a plurality of data indicator features corresponding to the cross-sectional data, as well as a plurality of user indicator features and a plurality of data indicator features corresponding to the time series data; and store each of the user indicator features and each of the data indicator features in a preset data feature pool; The algorithm analysis module is used to obtain user indicator features and data indicator features corresponding to each of the first training data, as well as user indicator features and data indicator features corresponding to each of the second training data; apply the data indicator features corresponding to each of the first training data to build a clustering model; apply the user indicator features and data indicator features corresponding to each of the first training data, as well as the user indicator features and data indicator features corresponding to each of the second training data to build a classification model; apply the data indicator features corresponding to each of the second training data to build a prediction model; extract the user indicator features and data indicator features corresponding to the cross-sectional data in the data feature pool, as well as the user indicator features and data indicator features corresponding to the time series data in the data feature pool; input the data indicator features corresponding to the cross-sectional data into the clustering model to obtain a first analysis result output by the clustering model; input the user indicator features and data indicator features corresponding to the cross-sectional data, as well as the user indicator features and data indicator features corresponding to the time series data in the data feature pool into the classification model to obtain a second analysis result output by the classification model; input the data indicator features corresponding to the time series data in the data feature pool into the prediction model to obtain a third analysis result output by the prediction model.

4. The data center resource analysis system according to claim 3, characterized in that: The algorithm analysis module is specifically used to perform dimensionality reduction processing on the data indicator features corresponding to each of the first training data to obtain cluster data; iteratively calculate the distance between each of the cluster data, and merge the cluster data with a distance less than a preset distance into the same cluster cluster, until the distance between each cluster data between any cluster clusters and each cluster data between other cluster clusters is not less than the preset distance, to obtain the clustering results corresponding to each of the cluster data; and build a clustering model based on the clustering results.

5. The data center resource analysis system according to claim 3, characterized in that: The algorithm analysis module is specifically used to construct a third training data set based on the first training data set and the second training data set, wherein the third training data set includes a plurality of third training data, each of which is composed of a first training data and a second training data; and set a data label corresponding to each of the third training data; A classification model is constructed based on each of the third training data, a data label corresponding to each of the third training data, and a preset loss function.

6. The data center resource analysis system according to claim 3, characterized in that: The algorithm analysis module is specifically used to input the data indicator features corresponding to each of the second training data into a preset initial prediction model to obtain the training results output by the initial prediction model; determine whether the training results meet the preset training conditions, and if not, adjust the model parameters of the initial prediction model, and continue to input the data indicator features corresponding to each of the second training data into the initial prediction model until the training results currently output by the prediction model meet the training conditions, thereby obtaining the final prediction model.

7. The data center resource analysis system according to claim 1 or 3, characterized in that: The comprehensive evaluation layer includes: a resource partitioning module, a user profiling module and a demand evaluation module; The resource partitioning module is used to evaluate the first analysis result, generate a resource partitioning construction plan corresponding to the data center resources, and output the resource partitioning construction plan to a preset server; The user profiling module is used to evaluate the second analysis result, determine the user's usage information of the data center resources, generate resource capacity adjustment information corresponding to the data center resources, and output the resource capacity adjustment information to the server; The demand evaluation module is used to evaluate the third analysis result, generate resource equipment demand information corresponding to the data center resources, and output the resource equipment demand information to the server.

8. A data center resource analysis method, characterized in that: The method is applied to a data center resource analysis system, and the method comprises: Obtain offline data and real-time data of data center resources, extract configuration data of the offline data, and extract performance data of the real-time data; Processing the configuration data and the performance data to form analysis index data, wherein the analysis index data includes cross-sectional data and time series data; Acquire a preset first training data set and a second training data set, wherein the first training data includes a plurality of first training data, the second training data includes a plurality of second training data, the first training data is cross-sectional data for generating a model, and the second training data is time series data for generating a model; Based on the first training data set and the second training data set, construct a clustering model, a classification model and a prediction model; Applying the clustering model, the classification model and the prediction model to analyze the analysis index data, and obtaining a first analysis result, a second analysis result and a third analysis result corresponding to the analysis index data outputted by the clustering model, the classification model and the prediction model respectively; The first analysis result, the second analysis result and the third analysis result are evaluated respectively to generate a resource partition construction plan, resource capacity adjustment information and resource equipment demand information corresponding to the data center resources.

9. A storage medium comprising stored instructions, wherein: When the instruction is executed, the device where the storage medium is located is controlled to execute the data center resource analysis method as described in claim 8.

10. An electronic device comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to perform the data center resource analysis method as claimed in claim 8.

Images