System and method for quality of service violation awareness and response for cloud-based black-box applications

Abstract

A service quality violation perception and response system and method for a black box application on a cloud, comprising: an offline data preparation module, and an application classification marking module, a monitoring and collecting module, a QoS prediction module and an interference perception module running in real time on each server node, the application ensures the QoS level of the black box service in the public cloud environment, and responds when the key service occurs QoS violation, that is, the interference degree of the shared resource is sorted and the interference source is located at the same time, which provides support for service scheduling and improves the resource utilization rate of the server on the cloud.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, and particularly relates to a service quality violation perception and response system and method for black-box applications on a cloud. BACKGROUND

[0002] Resource utilization of a cloud service data center server is one of the focuses of cloud vendors. In order to improve the resource utilization of the data center, the cloud service usually adopts a hybrid deployment scheme. Although the hybrid deployment can effectively improve the resource utilization on the cloud, it introduces resource competition between applications, which may cause QoS reduction or even violation of some businesses. However, the existing cloud scheduling for the hybrid deployment is mostly based on load balancing, and such a static strategy cannot perceive the changes in resource requirements, resource sensitivity and competition interference of the applications with the change in business load, which may cause continuous occurrence of business QoS violation and bring immeasurable loss. SUMMARY

[0003] The present application is directed to the limitation that the existing technology needs high-privilege information in the online business QoS guarantee stage. In the face of more and more black-box businesses on the cloud and more and more complex hybrid deployment situations, the existing technology is insufficient in maintaining the QoS of such businesses due to the high information requirement, and the present application proposes a service quality violation perception and response system and method for black-box applications on a cloud, which guarantees the QoS level of the black-box businesses in a public cloud environment and responds when a key business occurs QoS violation, sorts the interference degree of the shared resources, determines the interference resources and locates the interference sources, provides support for resource allocation and business scheduling, and improves the resource utilization of the servers on the cloud.

[0004] The present application is implemented by the following technical solutions:

[0005] The application relates to a service quality violation perception and response system for a black box application on a cloud, comprising an offline data preparation module and application classification and labeling modules, a monitoring and collecting module, a QoS prediction module and an interference perception module which are set on each server node and run in real time, wherein: the offline data preparation module prepares offline data for online business applications; the application classification and labeling module classifies and labels newly online black box applications to obtain application categories and resource allocation labels; the monitoring and collecting module collects runtime indexes of all applications on the node and outputs the collected data of the black box application to the QoS prediction module for analysis; the QoS prediction module analyzes whether QoS violation occurs in the online application according to the runtime indexes and outputs application information of the QoS violation to the interference perception module; the interference perception module requests the detection and collecting module to acquire all application resource allocation label information and the collected index data in the monitoring and collecting period, analyzes the resource which has the greatest influence on the QoS violation application, that is, the interference resource, and performs interference source positioning and outputs to the upper cluster scheduler.

[0006] The black box application refers to an online business application which cannot directly obtain application types, internal functions and specific information and has QoS requirements.

[0007] The offline data includes an application resource usage vector, a function call list library, a QoS violation prediction model, a workload prediction model and a resource pressure-QoS curve cluster, that is, a pressure curve cluster or a resource pressure-QoS probability distribution model, the above data are bound with corresponding applications and saved in a matched database to provide information support for the real-time running modules on each node.

[0008] The application relates to an interference source detection method based on the above system, which comprises an offline preparation stage and an online monitoring stage, wherein: in the offline preparation stage, a corresponding QoS violation prediction model, a workload prediction model and a resource pressure-QoS curve cluster or a resource pressure-QoS probability distribution model based on a logarithmic normal distribution are established for online business applications on the cloud, and resource usage vectors and function call stack information of the business running under different workload states are collected, and the function call stack information is used to build a function call list library of the online business applications on the cloud; in the online monitoring stage, when a black box application is deployed, corresponding resource allocation tags are added to the black box application according to its resource allocation, and then the black box application is classified in two levels by collecting the resource usage vectors and function call stack information of the black box application running time, the application with the highest similarity in the database is matched, and the type of the black box application is determined; after the classification and labeling level is completed, the resource usage rate of various shared resources, operating system statistical information and eBPF script statistical information during the running of the online business are continuously collected in the online monitoring stage, whether QoS violation occurs is judged through the QoS violation prediction model, when it is found that QoS violation occurs, the workload prediction model of the QoS violation application is called to predict the current workload of the QoS violation application, the corresponding resource pressure-QoS curve or resource pressure-QoS probability distribution model is selected, the influence degree of each shared resource and the competing application on the QoS is analyzed, the interference resource is determined, and the interference source is located.

[0009] Technical effects

[0010] The application realizes the monitoring of the QoS of the black box application and the response of the QoS violation; the resource pressure-QoS curve or the resource pressure-QoS probability distribution model is realized for different sizes of data sets, which is used to reflect the influence of the application QoS under the application of different pressures of a certain resource; the resource allocation tags are marked for the application, the isolation and competition relationship between the applications on different resources is analyzed according to the resource allocation tags, and then the interference source of the application with QoS violation is accurately located. Compared with the prior art, the application can guarantee the quality of service of the application without grabbing the specific information of the user deploying the black box application; for different sizes of data sets, the corresponding QoS sensitivity model modeling method can be selected to analyze the sensitivity of the application to different resource pressures under different states; the resource allocation tags of the application allow the system to accurately analyze the mutual influence between different applications on different resource dimensions under the condition of complex mixed deployment in a finer granularity. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 It is a schematic diagram of the system of the application;

[0012] Figure 2 It is an offline data preparation flowchart;

[0013] Figure 3The sample outlier schematic diagram;

[0014] Figure 4 The curve drawing process schematic diagram;

[0015] Figure 5 The pressure curve drawing stage schematic diagram;

[0016] In the figure: (a) sample scatter plot after noise reduction; (b) reference curve; (c) maximum pressure curve; (d) minimum pressure curve; (e) pressure curve; (f) pressure curve group;

[0017] Figure 6 Online monitoring flow chart;

[0018] Figure 7 The black box application secondary classification flow chart;

[0019] Figure 8 The offline preparation flow chart in the embodiment 2 of the application;

[0020] Figure 9 The resource pressure-QoS probability distribution model construction flow chart schematic diagram;

[0021] Figure 10 The scene 1 experimental result schematic diagram;

[0022] In the figure: (a) scene 1 online-app1 real-time P99 tail delay and QoS model prediction result; (b) scene

[0023] 1 online-app2 real-time P99 tail delay and QoS model prediction result;

[0024] Figure 11 The scene 2 experimental result schematic diagram;

[0025] In the figure: (a) scene 2 online-app1 real-time P99 tail delay and QoS model prediction result; (b) scene

[0026] 2 online-app2 real-time P99 tail delay and QoS model prediction result;

[0027] Figure 12 The scene 3 experimental result schematic diagram;

[0028] In the figure: (a) scene 3 online-app1 real-time P99 tail delay and QoS model prediction result; (b) scene

[0029] 3 online-app2 real-time P99 tail delay and QoS model prediction result;

[0030] Figure 13 The scene 4 experimental result schematic diagram;

[0031] Fig. (a) Scenario 4 online-app1 real-time P99 tail latency and QoS model prediction results; (b) Scenario

[0032] 4 online-app2 real-time P99 tail latency and QoS model prediction results. DETAILED DESCRIPTION

[0033] Embodiment 1

[0034] As Figure 1 shown, the embodiment relates to a service quality violation perception and response system for cloud-based black-box applications, which includes an offline data preparation module and an application classification and labeling module, a monitoring and collection module, a QoS prediction module, and an interference perception module running in real time on each server node, wherein: the offline data preparation module prepares offline data for online business applications on the cloud; the application classification and labeling module classifies and labels newly online black-box applications to obtain application categories and resource allocation labels; the monitoring and collection module continuously collects runtime indicators for all applications on the node and outputs the collected data of online applications to the QoS prediction module for analysis; the QoS prediction module analyzes whether QoS violation occurs for online applications based on the runtime indicators and outputs application information of QoS violation to the interference perception module; the interference perception module requests the monitoring and collection module to obtain all application resource allocation label information and the collected indicator data within the monitoring and collection period, analyzes the resource that has the greatest impact on the QoS violation application, i.e., the interference resource, and performs interference source positioning and outputs to the upper cluster scheduler.

[0035] The offline data preparation module comprises an information collection unit, a resource usage vector analysis unit, a function call list analysis unit, a QoS violation prediction model establishment unit, a workload prediction model establishment unit, and a resource pressure sensitivity model establishment unit. The information collection unit uses a stress generation tool to generate different stresses under different resource dimensions, runs the application under different stresses and different workloads, and collects the function call stack, resource utilization, operating system statistics, and eBPF script statistics during application runtime, and collects the QoS indicators of the application under different conditions. The resource usage vector analysis unit obtains the utilization information of various resources of the application, analyzes the usage of the application on different resources during the running process, generates the resource usage vector of the application according to the usage of the application on different dimension resources, and generates the function call list of the application according to the function call list of the application under different workloads. The function call list analysis unit analyzes the function call stack of the application when running under different workloads, arranges the function call list in descending order according to the calling time of each function, and generates the function call list library of the application according to the function call list of the application under different workloads. The QoS violation prediction model establishment unit establishes and trains the QoS violation prediction model of the application according to the QoS indicators of the application under different conditions and the resource utilization, operating system statistics, and eBPF script statistics during runtime. The workload prediction model establishment unit establishes and trains the workload prediction model of the application according to the indicators collected when the application runs under different workloads under different conditions. The resource pressure sensitivity model establishment unit selects the resource pressure-QoS curve cluster or the resource pressure-QoS probability distribution model of the application according to the scale of the information collected by the data collection unit.

[0036] The data generated by the resource usage vector analysis unit, the function call list analysis unit, the QoS violation prediction model establishment unit, the workload prediction model establishment unit, and the resource pressure sensitivity model establishment unit is bound to the application and saved in the database matched with the offline data preparation module.

[0037] The application classification and labeling module comprises an initial information collection unit, an application classification unit, and a resource allocation labeling unit. The initial information collection unit collects resource allocation information during the initial running of the application after it goes online, and collects function call stack information of the black box application at the same time if the current application is a black box application. The application classification unit analyzes the resource usage vector according to the resource usage information of the black box application, and generates the function call list according to the function call stack information of the black box application, adopts a two-level classification algorithm to identify the type of the black box application, and analyzes the resource allocation of the application and labels it with the corresponding resource allocation label. At the same time, the application classification and labeling module maintains the information of all applications running on the current server node.

[0038] The monitoring and collecting refers to obtaining application information of each application running on the current server node from the application classification marking module, monitoring each index of all applications running on the node in real time, observing the change of the application deployment state on the node and activating the application classification marking module when a new application goes online.

[0039] The application information includes application name of each application on the node and application type and resource allocation label of the application analyzed by the application classification marking module.

[0040] The indexes include resource utilization, operating system statistical information and eBPF script statistical information, wherein the resource utilization includes CPU utilization, memory occupancy, memory bandwidth, LLC occupancy, I / O read bandwidth, I / O write bandwidth, network transmission bandwidth and network receiving bandwidth; the operating system statistical information and the eBPF script statistical information include TCP connection number, page error frequency, CPU migration frequency, average read I / O delay, average write I / O delay, page cache hit frequency, page cache miss frequency, dirty page cache frequency, read hit rate, write hit rate, average CPU allocation time, and average CPU waiting time.

[0041] The QoS prediction refers to obtaining a QoS violation prediction model of the corresponding type of the online application from the database of the offline resource module, and judging in real time whether the online application has QoS violation according to each index of the application running collected by the monitoring and collecting module. When the online application has violation, the interference awareness module is activated, and the QoS violation application information is transmitted to the interference awareness module.

[0042] The interference awareness refers to requesting an application workload prediction model from the database of the offline data preparation module according to each index of the QoS violation application running, predicting the current workload of the application, obtaining resource allocation label and resource utilization of each dimension of the application running on the node from the monitoring and collecting module, and requesting a resource pressure-QoS curve or resource pressure-QoS probability distribution of the application under the current workload from the database of the offline data preparation module. The interference awareness module sorts the influence degree of various dimension resource competition on the QoS violation application based on the resource pressure-QoS curve or resource pressure-QoS probability distribution, sorts the influence degree of each competing application on the application, and finally analyzes the interference resource of the QoS violation application, that is, the resource with the largest interference and the interference source, that is, the application with the largest influence on the QoS violation application.

[0043] As Figures 2-7As shown, the QoS violation awareness and response method based on the above system in this embodiment includes an offline preparation stage and an online monitoring stage, wherein: in the offline preparation stage, the offline data preparation module establishes corresponding QoS violation prediction models, workload prediction models and resource pressure-QoS curve clusters for online business applications on the cloud, collects resource usage vectors of the business, and collects function call lists under different workload states to build a function call list library for online business applications on the cloud; in the online monitoring stage, the deployed black box application is labeled with a corresponding resource allocation label according to its resource allocation, then the black box application is classified by collecting its runtime resource usage vector and function call list, the application with the highest similarity in the database is matched to determine the type of the black box application, then the resource usage rate of various shared resources, operating system statistical information and eBPF script grabbing information during online business operation are continuously collected, the QoS violation prediction model is used to determine whether QoS violation occurs, and the workload prediction model is called to predict the current workload of the application when QoS violation occurs, the corresponding resource pressure-QoS curve, i.e. the pressure curve, is selected and the influence of each shared resource and competing container on QoS is calculated, the interference resource is analyzed and the QoS interference source is located.

[0044] The resource usage rate includes CPU utilization, LLC occupancy, memory bandwidth occupancy, read I / O read bandwidth, write I / O bandwidth and network I / O bandwidth.

[0045] The offline preparation stage specifically includes:

[0046] Step 1) According to the performance of the cloud server, specify the workload range in which the online application can normally run.

[0047] Step 2) Use the stress generation tool to run the application with different workloads in different resource dimensions, and collect various indicators, function call stack information and QoS indicators of the application under different conditions by the information collection unit in the offline data preparation module.

[0048] The stress generation tool includes stress-ng to generate CPU occupancy competition stress, memory bandwidth competition stress and write I / O bandwidth competition stress; rdtset to generate LLC occupancy competition stress; fio to generate read I / O bandwidth competition stress;

[0049] wondershaper to generate network receive bandwidth and network transmission bandwidth competition stress.

[0050] The resource dimensions are CPU occupancy, LLC occupancy, memory bandwidth, I / O read bandwidth, I / O write bandwidth and network I / O bandwidth, and different degrees of resource pressure are generated on these resource dimensions through the stress generation tool.

[0051] The QoS of the online application is characterized by the request response 99% tail delay (P99 tail delay), and since the P99 tail delay has random jitter, each index also has fluctuations within the error range, so multiple data (not less than 3 times) need to be collected under the same conditions.

[0052] Step 3) The resource usage vector analysis unit, the function call list analysis unit, the QoS violation prediction model establishment unit, the workload prediction model establishment unit and the resource pressure sensitive model establishment unit respectively construct the resource usage vector of the online service application on the cloud, build the function call list library, construct the QoS violation prediction model, the workload prediction model and the resource pressure-QoS curve cluster according to the data collected by the information collection unit, and store the above results in the corresponding application after binding to the supporting database for access and use by the modules on each node in the online monitoring process, specifically including:

[0053] 3.1) Construct the resource usage vector: the resource usage vector U is a multi-dimensional vector, and the dimension is the dimension of all shared resources of the application, wherein each element corresponds to the usage of a certain resource during application runtime: if a certain resource is used during application runtime, the corresponding element will be marked 1; otherwise, marked 0.

[0054] For example, when the resource usage vector format is U=(u cpu ,u memory-benwitdth ,u io-write ,u io-read ,…), wherein: u src ∈{0,1}, u src =1 indicates that the application uses the resource src during runtime, and when the second dimension element is 1, it indicates that the application has memory access behavior; u src =0 indicates that the resource is not used during application runtime.

[0055] 3.2) Build the function call list library: the function call stack information is collected by the performance analysis tool perf, and the functions in the function call stack are arranged in descending order according to the calling time to obtain the function call list; the set of function call lists corresponding to the application running under different workloads constitutes the function call list library of the application.

[0056] 3.3) Building and training the QoS violation prediction model: set the QoS violation threshold according to the application response delay requirement, that is, when the QoS of the application exceeds the threshold, the application has QoS violation. Considering that the application P99 tail delay has obvious random fluctuations, the application adopts a classification model, that is, a classification prediction model is established according to whether the sample corresponding QoS has violation, which specifically includes:

[0057] i) Feature selection: in order to improve the model training efficiency, the correlation between various indicators collected and application QoS violation behavior needs to be selected, and the sequential feature selector (Sequential Feature Selector) is used for automatic feature selection, and the feature set with the highest prediction accuracy is selected according to the heuristic method.

[0058] ii) Model building and training: according to the cloud service demand, the QoS standard is specified for the application, and the QoS violation threshold is divided (that is, when the delay exceeds the threshold, it means that the application has QoS violation). Based on whether the QoS corresponding to the data sample has violation, the data set is divided into two categories. Because the number of samples with QoS violation is obviously less than the number of samples without QoS violation when collecting data, the self-paced ensemble classifier (SPE) modeling algorithm based on uneven data classification is used to establish and train the QoS violation prediction model.

[0059] 3.4) Building and training the workload prediction model: in the actual running scene, the resource pressure-QoS curve corresponding to different workloads of online application has obvious difference, when the interference resource is analyzed and the interference source is located based on the pressure curve, the corresponding pressure curve needs to be selected according to the current workload of the application. At the same time, considering that the server side cannot directly obtain the workload of online business in public cloud environment, it is necessary to build a workload prediction model to infer the current workload according to various indicators during business running, which specifically includes:

[0060] i) Feature selection: the same feature selection algorithm as the QoS violation prediction model is used.

[0061] ii) Model building and training: based on KNN, the workload prediction model is built and the workload prediction model is trained.

[0062] 3.5) Building resource pressure-QoS curve cluster: according to the QoS and the available amount of different resources when the online application runs under different load and different resource pressure, the pressure curve cluster of the application is built, which specifically includes:

[0063] i) Data denoising: due to the random jitter of P99 tail delay, the P99 tail delay data collected under the same condition is not the same, and even outliers may appear, so the Mid-Score algorithm is used to eliminate outliers.

[0064] As shown in Figure 3 , for an online application under the same workload, a scatter plot of all samples collected under the increasing dimension of resource pressure, the horizontal axis is the resource pressure value, and the vertical axis is the P99 tail delay value, wherein: the sample points circled are outliers. The existence of outliers has a great impact on the credibility of the resource pressure-QoS curve drawn, so data denoising is needed to eliminate abnormal outliers.

[0065] The Mid-Score algorithm refers to: the median of the sample under the same condition is denoted as m, and the Mid-Score is calculated as follows: Wherein: x i is each sample value, n is the number of samples, and when the sample M≥1.7, the sample is an outlier.

[0066] ii) Curve drawing: after denoising the data samples, the sample QoS under each level of resource pressure in the scatter plot of the denoised samples is averaged to obtain the reference curve as shown in Figure 5 (a) The reference curve is then used to construct the maximum pressure curve and the minimum pressure curve. Figure 5 (b) The reference curve is then used to construct the maximum pressure curve and the minimum pressure curve.

[0067] The maximum pressure curve construction method: the maximum pressure curve follows the principle that under the same workload, as the resource pressure increases from left to right, the curve is monotonically non-decreasing, and when the P99 tail delay corresponding to a certain level of pressure is smaller than the P99 tail delay corresponding to the adjacent smaller pressure level, the maximum value of the current P99 tail delay is used to cover it.

[0068] The minimum pressure curve construction method: the minimum pressure curve follows the principle that under the same workload, as the resource pressure decreases from right to left, the curve is monotonically non-increasing, and when the P99 tail delay corresponding to a certain level of pressure is greater than the P99 tail delay corresponding to the adjacent larger pressure level, the minimum value of the current P99 tail delay is used to cover it.

[0069] Considering that the P99 tail delay has random jitter, the P99 tail delay corresponding to some pressure levels may be abnormal (outlier peak or outlier valley) when constructing the maximum pressure curve and the minimum pressure curve. The outlier peak determination algorithm is used to determine the outlier peak in the maximum pressure curve, or the outlier valley determination algorithm is used to determine the outlier valley in the minimum pressure curve.

[0070] The outlier peak determination algorithm: follow the increasing direction of resource pressure, there are still k larger pressure levels corresponding data to be processed. When k>3 and starting from the adjacent lower level of pressure, there are k / 2 consecutive samples whose P99 tail delay is lower than the P99 tail delay of the current processing sample, the current sample is determined as an outlier peak.

[0071] The outlier valley determination algorithm: follow the decreasing direction of resource pressure, there are still k smaller pressure levels corresponding data to be processed. When k>3 and starting from the adjacent lower level of pressure, there are k / 2 consecutive samples whose P99 tail delay is higher than the P99 tail delay of the current processing sample, the current sample is determined as an outlier valley.

[0072] The processing method for outlier peaks and outlier valleys is the same, both of which use linear interpolation of the left and right adjacent sample values to replace the original outlier values. The maximum pressure curve is shown as Figure 5 (c) blue curve, and the minimum pressure curve is shown as Figure 5 (d) green curve. The average of the maximum pressure curve and the minimum pressure curve is taken to obtain the pressure curve, which is shown as Figure 5 (e) red curve.

[0073] iii) In the same resource dimension, the pressure curve set corresponding to different workloads constitutes a pressure curve group, as shown in Figure 5 (f). When the pressure curves corresponding to two adjacent workloads w1 and w2 in the pressure curve set are known, the pressure curve of workload w3 (w1<w3<w2) can be obtained by interpolation of the pressure curves of w1 and w2. The pressure curve groups corresponding to different resources constitute a pressure curve cluster.

[0074] The data generated by steps 1-3 are all bound to the application and stored in the database of the offline data preparation module. The offline data preparation module will repeat the above three steps for various online business applications on the cloud to prepare offline data and provide information support for each module in the online stage.

[0075] As shown in Figure 6 , the online monitoring stage includes:

[0076] Step 4) Application classification and labeling stage: when a new application is online, the resource allocation labeling unit in the application classification and labeling module labels the resource allocation tag of the application according to the resource allocation of the application, and checks whether the application is a black-box application. When the application is a black-box application, the initial information collection unit specifies the initial running time of the application, and collects the resource usage information and function call stack information of the black-box application during the specified time, and transmits them to the application classification unit. The application classification unit generates a resource usage vector and a function call list according to the above-mentioned resource usage information and function call stack information of the black-box application, and classifies the black-box application by using a two-level classification method according to the above-mentioned data. The application classification and labeling module transmits the resource allocation tags of the applications on the node and the application type information of the black-box application to the monitoring and collecting module, and then enters the monitoring and collecting stage, which specifically includes:

[0077] 4.1) Labeling resource allocation tag: the resource allocation labeling unit labels the application according to the specific resource allocated to the application. For example, the application is allocated to a specific CPU core of a NUMA node, or the application is allocated to a specific network card. In the same resource dimension, only two applications with the same resource allocation tag have a competitive relationship.

[0078] Through the resource allocation tag of the application, the application can more accurately analyze the isolation and competition relationship between various applications in a complex resource mixed isolation environment, thereby providing more accurate support for the subsequent interference awareness module.

[0079] 4.2) Black-box application classification: the application classification unit classifies the black-box application by using a two-level classification method as shown in Figure 7 : first, the black-box application is coarsely classified according to the resource usage vector, that is, the first level of classification; then, the function call list library of the application with the same resource usage vector is read from the database prepared by the offline data preparation module, and the similarity between the black-box function call list and each function call list in the read application function call list library is calculated, and finally the black-box application is classified into the benchmark application type with the highest similarity, that is, the second level of classification.

[0080] The similarity, that is, the similarity between the black-box application Black and the benchmark application Wherein: A is any function call list in the benchmark function App function call list, and B is the function call list of the black-box application Black.

[0081] Step 5) Monitoring collection phase: When it is not the first cycle of monitoring collection, the monitoring collection module will perform node application deployment update check, update the monitoring application list and check whether there are new applications online in the last monitoring cycle. When there are new applications online, the application classification marking module is activated, and the application classification marking phase is entered. After the new online application is processed, the monitoring collection cycle is entered. The monitoring collection module periodically and concurrently collects various indicators of the runtime of all running applications on the node. Specifically, offline applications only collect resource usage, and online applications collect additional indicators in addition to resource usage according to the QoS violation prediction model and the required input features of workload prediction. The collected indicators of online applications are output to the QoS prediction module.

[0082] Step 6) QoS prediction phase: The QoS prediction module calls the corresponding QoS violation prediction model of the online application, and judges whether there is QoS violation of the application on the node according to the collected indicators of the online application runtime. When there is no QoS violation of the application, the monitoring collection module is directly notified to perform node application deployment update check; when there is QoS violation of the application, the application information is output to the interference awareness module, and the interference awareness phase is entered.

[0083] Step 7) Interference awareness phase: After the interference awareness module receives the information of the QoS violation application V from the QoS prediction module, it requests the monitoring collection module to obtain all application resource allocation tag information and the collected application runtime indicators in the monitoring collection cycle. The workload prediction model of application V is called to predict its current workload, and after obtaining the resource pressure curve corresponding to the current workload of the application, the interference awareness module performs interference resource analysis and interference source positioning on application V, which specifically includes:

[0084] 7.1) Interference resource analysis: analyze the resource pressure p i of application V in each dimension resource i : When application C is an application deployed on the same node as application V, the resource allocation tag of application C on resource i is C i , the resource allocation tag of application V on resource i is V C,i , and the occupancy of application C on resource i is R i . Then the resource pressure p C≠V of application V on resource i is p i =∑ i Col(C C,i ,V i )R i , where Col(C i ,V i ) determines whether the resource allocation tags of applications C and V on shared resource i are the same. If the resource allocation tags are the same, it indicates that there is competition between the two applications on the shared resource i: Analyzing the resource pressure of each dimension resource of application V, judging the influence of the resource pressure of each dimension resource on the QoS of application V according to the pressure curve, i.e. qos i (p i ), wherein: qos i is the pressure curve of application V on resource i, and is sorted, wherein: the shared resource with the greatest influence on the QoS of application V, i.e. the interference source:

[0085] 7.2) Interference source positioning: the degree of interference Q of other applications C deployed on the node on the QoS of V c =∑ i qos i (Col(C i ,V i )R C,i ), and is sorted according to the degree of interference of other applications, wherein: the application with the greatest degree of interference on V is the interference source:

[0086] After analyzing the interference resource and positioning the interference source, the interference awareness module outputs the interference resource and interference source information to the cluster scheduler, which may perform resource allocation or interference source scheduling according to different scheduling strategies. After the upper-layer scheduler completes the work, the monitoring and collecting module is woken up again for node application deployment update checking.

[0087] Embodiment 2

[0088] In different scenarios, for example, in the offline preparation stage, a large enough base of data is collected (under the same conditions, i.e. under the same resource pressure environment, the application runs under the same working load, and the sampling number is not less than 100 times), and a resource pressure-QoS probability distribution model based on a logarithmic normal distribution is used to replace the resource pressure-QoS curve cluster.

[0089] The difference between this embodiment and embodiment 1 is the construction method of the application resource pressure sensitivity model in the offline preparation stage and the interference awareness stage in the online monitoring stage, which is specifically:

[0090] As shown in Figure 8 , this embodiment uses a resource pressure-QoS probability distribution model based on a logarithmic normal distribution, and the specific construction process is as follows:

[0091] Step i) When the sampling base under the same conditions is large enough, there is no need to additionally perform data noise reduction, instead, based on a logarithmic normal distribution, the distribution of the QoS of the application running under a specified resource pressure environment and a specified working load is fitted using samples. When the number of collected samples is n, and the sample value of the QoS is q i, i = 1, 2, …, n). Using maximum likelihood estimation, the parameters of the lognormal distribution density function are obtained: The QoS probability distribution f(x; μ, σ) of the application under the current condition is obtained.

[0092] The QoS probability distribution of the application under different pressures of the same dimension resource under the same workload constitutes the resource pressure-QoS probability distribution. When the resource pressure-QoS probability distribution of a certain resource is known, and the QoS probability distributions corresponding to the resource pressures a and b are known, then the QoS probability distribution corresponding to the resource pressure c (a < c < b) is obtained by interpolation of the QoS probability distributions corresponding to a and b.

[0093] The resource pressure-QoS probability distribution of the application under different workloads under the same resource dimension constitutes the resource pressure-QoS probability distribution set of the application under the resource dimension. When the resource pressure-QoS probability distributions of workloads w1 and w2 are known in the resource pressure-QoS probability distribution set of the application, then the resource pressure-QoS probability distribution of workload w3 (w1 < w3 < w2) is obtained by interpolation of the resource pressure-QoS probability distributions of w1 and w2.

[0094] Step ii) integrates the resource pressure-QoS probability distribution sets corresponding to each dimension resource of the online application to constitute the resource pressure-QoS probability distribution model of the application.

[0095] In the interference awareness stage in this embodiment: when the resource pressure-QoS probability distribution model is constructed based on the lognormal distribution, the interference resource analysis and interference source positioning of the interference awareness module are performed by the following methods:

[0096] The interference resource analysis is performed by the following formula in step 7.1): i = ∑ C≠V Col(C i ,V i )R C,i The resource pressure p i experienced by the application V on the resource i is obtained by the formula. Based on the resource pressure-QoS probability distribution model of the application V, the QoS probability distribution φ i (x; p i ) of the application V under the resource pressure p i is obtained, where p i ≥ 0, and φ i (x; p i ) represents the probability density of the application QoS being 1 / x under the resource pressure p i , and x represents the pressure p i experienced by the resource i on the application.The influence coefficient of the application QoS is, since the application QoS is represented by the P99 tail delay in the application, the greater x is, the smaller the influence of the resource i on the application QoS (always ensure x>0). In particular, φ i (x;0)≡0. Based on the mutual independence between the QoS probability distributions under different dimensional resource pressures, the resource pressure p i The interference degree Inter i of the application V to the QoS is calculated. Further, the interference resource, that is, the resource with the greatest interference degree to the QoS of the application V, is wherein: indicates that when the resource pressure p j of the resource j is greater than the resource pressure p i of the resource i, the influence of the resource j on the application QoS is greater than the influence of the resource i on the application QoS. indicates that the influence of all other resource pressures on the application QoS is less than the influence of the resource pressure p i of the resource i on the application QoS.

[0097] The interference source positioning, when the application C is an application deployed on the same node as the application V, the resource allocation label of the application C on the resource i is C i , the resource allocation label of the application V on the resource i is V i , and the occupation amount of the application C on the resource i is R C,i . According to Embodiment 1, the application C will interfere with the QoS of the application V only when the application C has the same resource allocation label as the application V on the resource i, that is, Col(C i ,V i )=1. According to the resource pressure-QoS probability distribution model of the application V, when Col(C i ,V i )=1, the probability distribution φ i (x;Col(C i ,V i )R C,i ) of the influence of the application C on the QoS of the application V on the resource i is obtained, wherein: φ i (x;Col(C i ,V i )R C,i ) indicates that when the occupation amount of the application C on the resource i is R C,i , the interference corresponding to the QoS influence coefficient x has a corresponding probability density. When there are k different resources on the node, the influence of the application C on the QoS of the application V is subject to the distribution Similarly, the interference degree Inter of application C to application V is calculated by using C to application V C The probability of the greatest QoS impact of application C to application V Further, the interference resource is obtained Wherein: The probability that the QoS impact of application D to application V is weaker than the QoS impact of application C to application V (i.e., y>x); The probability distribution that the QoS impact of all other competing applications to application V is weaker than the QoS impact of application C to application V.

[0098] Through specific experiments, three online business containers and one offline business simulation container are deployed on a machine with the following hardware configuration: CPU: 2xIntel Xeon E5-2630 v4@2.20GHz, CPU core number: 40, L3 cache: 25MB, DRAM: 256GB, hard disk: 1.1T SSD, network bandwidth: 1000Mbps; the operating system of the machine is OpenEuler22.09 of Linux kernel version 5.10.0. The images corresponding to the three online containers are: mysql:5.7, nginx:1.22.1 and fortio / fortio, respectively named as black-app1, black-app2 and online-app, and the client applications of the three online containers are: sysbench, open loop wrk2 and fortio client; the two containers created based on the mysql:5.7 and nginx:1.22.1 images in the online container are black box application containers. The image of the offline business simulation container is colinianking / stress-ng, and the container name is offline-app, which will generate pressure on the CPU during running.

[0099] In the experimental scenario, the containers have different isolation and competition relationships on different resources, and the mixed deployment state is complex: online-app1 and online-app2, offline-app have a competition relationship on CPU and memory bandwidth, and are isolated from online-app3; the four applications have a competition relationship on other resources.

[0100] In the offline data preparation stage, the offline data preparation module prepares offline data for online business applications on the cloud, including Redis, Memcached, Nginx, MySQL, etc. To verify the feasibility of the system, the experimental verification is divided into three main parts: black box application classification verification, system prediction QoS violation verification and interference awareness verification.

[0101] Black box application classification verification: After the two black box online containers are online, the application classification marking module of the system determines the application type of the black box application. The results show that the accuracy of online-app1 being judged as a MySQL application is 95.8 %;

[0102] The accuracy of online-app2 being judged as an Nginx application is 99.2 %. In addition, the application classification method in the system is additionally used to classify the containers whose images are Redis and Memcahed, and the result accuracies are 94.8 % and 94.9 % respectively. That is, the average accuracy of the black box application classification of the system is 96.2 %.

[0103] System prediction QoS violation verification: In the above environment, the load of the two black box online containers and whether the offline container is running are controlled, the accuracy of the QoS prediction module of the system in predicting whether the black box application occurs QoS violation is verified under different scenarios (scenarios ①-④), and in the test process, the QoS prediction module judges whether the QoS of the black box application occurs violation every 15 s (that is, the monitoring period of the monitoring and collecting module). The results of judging online-app1 and online-app2 are as shown in Figures 10-13

[0104] Scenario ①: online-app1 runs in a high load state, online-app2 runs in a low load state, and offline-app is not online. In this scenario, neither of the two black box applications occurs QoS violation, and the QoS prediction module makes correct prediction in the experiment.

[0105] Scenario ②: online-app1 runs in a high load state, online-app2 runs in a low load state, and offline-app is online to exert resource pressure. In this scenario, online-app1 occurs QoS violation, and the prediction accuracy of the QoS prediction module is 95.8 %; online-app2 does not occur QoS violation, and the QoS prediction module makes correct prediction all the time.

[0106] Scenario ③: online-app1 runs in a low load state, online-app2 runs in a high load state, and offline-app is not online. In this scenario, neither of the two black box applications occurs QoS violation, and the prediction accuracies of the QoS prediction module for the two are 100 % and 98.6 % respectively. ​

[0107] Scenario ④: online-app1 runs in low load state, online-app2 runs in high load state, and offline-app goes online to impose resource pressure. In this scenario, no QoS violation occurs for online-app1, and the QoS prediction module makes correct predictions throughout; for online-app2, QoS violation occurs at 98.6% of the QoS prediction accuracy.

[0108] In all scenarios, the prediction accuracy of the system for QoS violation of the black box application is not less than 95%.

[0109] Interference awareness verification: In scenarios ② and ④ above, QoS violation occurs for the black box application. The above two scenarios are repeated 50 times respectively to verify the effectiveness of the system in interference awareness. The results show that in all scenarios, the system correctly labels the resource allocation for each container and identifies the isolation or competition relationship of each container on different resources in the load mixing scenario. In scenario ②, online-app1 violates in 50 experiments, and the system interference awareness module successfully analyzes the interference resource as CPU occupancy and successfully locates the interference source as offline-app; in scenario ④, online-app2 violates in 50 experiments, and the system interference awareness module successfully analyzes the interference resource as CPU occupancy and successfully locates the interference source as offline-app.

[0110] Compared with the prior art, the present application realizes black box application type identification, QoS monitoring and QoS violation response based on the online business application data database obtained by the offline data acquisition module. In the public cloud scenario, the system can guarantee the service quality of the application without capturing the specific information of the user-deployed black box application, further improving the privacy protection of the user. The present application realizes two sets of application QoS sensitivity model modeling methods for different sizes of online application data sets. These two methods can adapt to different sizes of data sets to provide analysis of the sensitivity of online applications to different resource pressures in different states. The present application labels the resource allocation of the application deployed on the cloud with a resource allocation label, analyzes the isolation and competition relationship between applications on different resources according to the resource allocation label, and then accurately analyzes the interference resource of the application that violates QoS and locates the interference source. This method provides more granular analysis for the system in a complex mixed deployment state, and can more accurately identify the mutual influence of different applications on different resources, improving the accuracy of interference resource analysis and interference source positioning.

[0111] The above specific embodiments can be partially adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application, the protection scope of the present application is subject to the claims and is not limited by the above specific embodiments, each implementation within the scope is subject to the present application.

Claims

1. A system for quality of service violation awareness and response for cloud based black box applications, the system comprising: The application comprises: An offline data preparation module and application classification and labeling modules, monitoring and collecting modules, QoS prediction modules and interference awareness modules running in real time on each server node, wherein: the offline data preparation module prepares offline data for online business applications; the application classification and labeling modules classify newly online black box applications and label all applications to obtain black box application categories and resource allocation labels for all applications; the monitoring and collecting modules collect runtime indicators for all applications on the node and output the collected data for the black box applications to the QoS prediction modules for analysis; the QoS prediction modules analyze whether there is a QoS violation of online applications according to the runtime indicators and output application information of the QoS violation to the interference awareness modules; the interference awareness modules request the monitoring and collecting modules to obtain resource allocation label information for all applications and the collected indicator data within the monitoring and collecting period and analyze the resource that has the greatest impact on the QoS violation application, referred to as the interference resource, and perform interference source positioning and output to the upper cluster scheduler; The offline data comprises: application resource usage vectors, function call list libraries, QoS violation prediction models and workload prediction models, and resource pressure-QoS curve clusters or resource pressure-QoS probability distribution models, which are bound to the corresponding applications and saved in the supporting database to provide information support for the real-time running modules on each node; The monitoring and collecting refers to obtaining application information of each application running on the current server node from the application classification and labeling modules, monitoring each indicator of the running applications on the node in real time, observing changes in the application deployment state on the node, and activating the application classification and labeling modules when a new application is online; The QoS prediction refers to obtaining the QoS violation prediction model of the corresponding type of online application from the database of the offline resource module, and judging in real time whether there is a QoS violation of online applications according to the collected indicators of the application runtime of the monitoring and collecting module, activating the interference awareness module and passing the QoS violation application information to it when there is a violation of online applications; The interference awareness refers to requesting the application workload prediction model from the database of the offline data preparation module according to the indicators of the QoS violation application runtime, predicting the current workload of the application, obtaining the resource allocation label and runtime resource utilization of each application on the node, and requesting the resource pressure-QoS curve or resource pressure-QoS probability distribution of the application under the current workload from the database of the offline data preparation module, sorting the impact degree of various dimensional resource competition on the QoS violation application based on the resource pressure-QoS curve or resource pressure-QoS probability distribution, sorting the impact degree of each competing application on the application, and finally analyzing the interference resource and interference source of the QoS violation application. The interference source is the application that has the greatest impact on the QoS violation application.

2. The cloud-facing black-box application QoS violation awareness and response system of claim 1, wherein, The offline data preparation module comprises an information collection unit, a resource usage vector analysis unit, a function call list analysis unit, a QoS violation prediction model establishment unit, a workload prediction model establishment unit, and a resource pressure sensitivity model establishment unit. The information collection unit uses a stress generation tool to generate different stresses under different resource dimensions, runs the application under different stresses and different workloads, and collects the function call stack, resource utilization, operating system statistics, and eBPF script statistics during application runtime, and collects the QoS indicators of the application under different conditions. The resource usage vector analysis unit obtains the utilization information of various resources of the application, analyzes the usage of the application on different resources during application running, generates the resource usage vector of the application according to the usage of the application on different dimension resources, the function call list analysis unit analyzes the function call stack of the application when running under different workloads, and arranges the function call list in descending order according to the calling time of each function to obtain the function call list, and generates the function call list library of the application according to the function call list of the application under different workloads. The QoS violation prediction model establishment unit establishes and trains the QoS violation prediction model of the application according to the QoS indicators of the application under different conditions and the resource utilization, operating system statistics, and eBPF script statistics during running. The workload prediction model establishment unit establishes and trains the workload prediction model of the application according to the indicators collected when the application runs under different workloads under different conditions. The resource pressure sensitivity model establishment unit selects the resource pressure-QoS curve cluster or the resource pressure-QoS probability distribution model of the application according to the scale of the information collected by the data collection unit.

3. The cloud-facing black-box application QoS violation awareness and response system of claim 1, wherein, The application classification and labeling module comprises an initial information collection unit, an application classification unit, and a resource allocation labeling unit. The initial information collection unit collects resource allocation information during the initial running of the application after it goes online, and collects function call stack information of the black box application at the same time. The application classification unit analyzes the resource usage vector according to the resource usage information of the black box application, and generates the function call list according to the function call stack information of the black box application. The application classification and labeling module maintains all application information running on the current server node while identifying the type of the black box application by adopting a two-level classification algorithm. The resource allocation labeling unit analyzes the resource allocation of the application and labels it with the corresponding resource allocation label.

4. A method of detecting an interference source based on the system of any one of claims 1-3, characterized in that, Comprise: An offline preparation stage and an online monitoring stage, wherein: the offline preparation stage establishes corresponding QoS violation prediction models, workload prediction models, and resource pressure-QoS curve clusters or resource pressure-QoS probability distribution models based on lognormal distribution for online business applications on the cloud, and collects resource usage vectors and function call stack information of the business running under different workload states, and the function call stack information is used to build a function call list library of the online business applications on the cloud; in the online monitoring stage, when a black box application is deployed, a corresponding resource allocation label is added according to the resource allocation of the black box application, and then the black box application is classified in two levels by collecting the resource usage vector and the function call stack information of the black box application running at runtime, the application with the highest similarity in the database is matched, and the type of the black box application is determined; After the classification and labeling level is completed, the resource usage rate of various shared resources, operating system statistical information and eBPF script statistical information during the online business running are continuously collected in the online monitoring stage, and when QoS violation of the business is found, the workload prediction model is called to predict the current workload of the QoS violation application, the corresponding resource pressure-QoS curve or resource pressure-QoS probability distribution model is selected, the influence degree of each shared resource and the competing application on QoS is analyzed, the interference resource is determined, and the interference source is located.

5. The method of interference source detection according to claim 4, characterized in that, The offline preparation stage specifically includes: Step 1) according to the performance of the cloud server, specify the workload range in which the online application can normally run; Step 2) use the stress generation tool to run the application under different resource dimensions and different workloads, and collect various indicators, function call stack information and QoS indicators of the application under different conditions by the information collection unit in the offline data preparation module; Step 3) the resource usage vector analysis unit, the function call list analysis unit, the QoS violation prediction model establishment unit, the workload prediction model establishment unit and the resource pressure sensitive model establishment unit construct the resource usage vector of the online business application on the cloud, build the function call list library, construct the QoS violation prediction model, the workload prediction model and the resource pressure-QoS curve cluster according to the data collected by the information collection unit, and store the above results in the corresponding application after binding, so as to access and use by the modules on each node in the online monitoring process.

6. The method of claim 4, wherein the step of determining the presence of an interfering source comprises the step of: The online monitoring stage includes: Step 4) Application classification and labeling stage: when a new application is online, the resource allocation labeling unit in the application classification and labeling module labels the resource allocation tag of the application according to the resource allocation of the application, and checks whether the application is a black-box application; when the application is a black-box application, the initial information collection unit specifies the initial running time of the application, collects the resource usage information and function call stack information of the black-box application during the specified time, and transmits the information to the application classification unit; the application classification unit generates a resource usage vector and a function call list according to the resource usage information and function call stack information of the black-box application, respectively, and classifies the black-box application by using a two-level classification method according to the data; the application classification and labeling module transmits the resource allocation tags of the applications on the node and the application type information of the black-box application to the monitoring and collection module, and then enters the monitoring and collection stage; Step 5) Monitoring and collection stage: when it is not the first cycle of monitoring and collection, the monitoring and collection module performs node application deployment update checking, updates the monitoring application list, and checks whether a new application is online in the last monitoring cycle; when a new application is online, the application classification and labeling module is activated, and the application classification and labeling stage is entered; after the new online application is processed, the monitoring and collection cycle is entered; the monitoring and collection module periodically and simultaneously collects various indicators of the runtime of all running applications on the node; Step 6) The QoS prediction module calls the QoS violation prediction model corresponding to the online application, judges whether there is QoS violation of the application on the node according to the collected indicators of the runtime of the online application; when there is no QoS violation of the application, the monitoring and collection module is directly notified to perform node application deployment update checking; when there is QoS violation of the application, the application information is output to the interference perception module, and the interference perception stage is entered; Step 7) Interference perception stage: after the interference perception module receives the information of the QoS violation application V from the QoS prediction module, the interference perception module requests the monitoring and collection module to obtain all application resource allocation tag information and various indicators of the runtime of the application collected in the monitoring and collection cycle, calls the workload prediction model of the application V to predict the current workload of the application V, obtains the resource pressure curves corresponding to the current workload of the application V, and then the interference perception module performs interference resource analysis and interference source positioning on the application V.

7. The method of claim 4, wherein the step of determining the presence of an interfering source comprises the step of: The resource pressure-QoS probability distribution model based on the lognormal distribution is constructed by the following steps: ​ Step 1: When the number of collected samples is , the sample value of QoS is , the parameters of the lognormal distribution density function are obtained by maximum likelihood estimation: , , and the QoS probability distribution of the application under the current condition is obtained ; Step 2) Integrate the resource pressure-QoS probability distribution sets corresponding to each dimension resource of the online application to form the resource pressure-QoS probability distribution model of the application.

8. The interference source detection method according to claim 4, characterized in that, The interference resource analysis, when based on the stress curve cluster, specifically comprises: analyzing resource stress suffered by the application V in each dimension resource : when the application C is an application deployed on the same node as the application V, a resource allocation label of the application C on the resource i is , a resource allocation label of the application V on the resource i is , an occupancy of the application C on the resource i is , and resource stress suffered by the application V on the resource i is , wherein: determining whether the resource allocation labels of the applications C and V on the shared resource i are the same, and the same resource allocation labels indicate that the applications C and V compete for the shared resource i: analyzing resource stress suffered by the application V in each dimension resource, judging the influence of the dimension resource stress on the QoS of the application V according to the stress curve, wherein: is a stress curve of the application V on the resource i, and the shared resource with the greatest influence on the QoS of the application V is the interference source: ; The interference resource analysis, when constructing the resource pressure-QoS probability distribution model based on the lognormal distribution, specifically includes: obtaining the resource pressure suffered by the application V on the resource i through Formula , obtaining the QoS probability distribution of the application V under the resource pressure of each dimension based on the resource pressure-QoS probability distribution model of the application V , wherein , denotes the probability density of the application QoS under the resource pressure , , is the pressure generated by the resource i to the application, is the influence coefficient of the application QoS when the resource pressure of the resource i is , based on the mutual independence between the QoS probability distributions under the resource pressures of different dimensions, the resource pressure interferes with the QoS of the application V is the degree of interference of the resource pressure to the QoS of the application V , the resource with the greatest interference to the QoS of the application V is , wherein denotes that the influence of the resource pressure of the resource j on the application QoS is less than the influence of the resource pressure of the resource i on the application QoS ; denotes the probability distribution of the influence of all other resource pressures on the application QoS, which is less than the influence of the resource pressure of the resource i on the application QoS.​ 9. The method of claim 4, wherein the step of detecting the interference source comprises the step of: The interference source positioning, when based on the pressure curve cluster, specifically includes: the interference degree of other application C on V deployed on the node , which is sorted according to the interference degree of other applications, wherein: the application with the largest interference degree on V is the interference source: ; ​ The interference source positioning, when constructing the resource stress-QoS probability distribution model based on the lognormal distribution, specifically comprises the following steps. , the resource allocation label of the application V on the resource i is , the occupation amount of the application C on the resource i is , according to the resource stress-QoS probability distribution model of the application V, when , the probability distribution of the QoS influence of the application C on the application V on the resource i is , wherein: the interference corresponding QoS influence coefficient of the application C on the application i is when the occupation amount of the application C on the resource i is the corresponding probability density, when there are k different resources on the node, the QoS influence of the application C on the application V is subjected to the distribution ; according to the interference resource analysis, the interference degree of the application C on the application V is , the probability of the QoS influence of the application C on the application V being the largest is further obtaining the interference resource , wherein: the probability of the QoS influence of the application D on the application V being weaker than the QoS influence of the application C on the application V is ; and the probability distribution of the QoS influence of all other competing applications on the application V being weaker than the QoS influence of the application C on the application V.

Images