Detection method and system for Simpson paradox phenomenon in cloud application change scene and application

Abstract

The invention discloses a detection method for Simpson paradox phenomenon in a cloud application change scene, and the detection method comprises the following steps: 1, constructing a performance index unified model according to the performance index of a cloud application instance; 2, judging whether the change trend of the performance indexes meets necessary conditions or not, excluding the performance indexes which are impossible to generate Simpson paradox phenomena, and reducing the detection space; 3, counting the value domain size of each grouping feature, and selecting a detection strategy based on the number of elements in the value domain; and 4, implementing the selected detection strategy and outputting a detection result. The invention further discloses a detection system for implementing the detection method, and the detection system has wide application value.

Description

Technical Field

[0001] The present invention belongs to the fields of cloud computing, grayscale testing and performance engineering, and relates to a detection method, system and application of the Simpson's paradox phenomenon in a cloud application change scenario. Background Art

[0002] As the scale of cloud services in data centers continues to expand, improving cloud application performance has become one of the core tasks to ensure the quality of cloud services and meet user needs. Among them, cloud application changes are an effective way to optimize application performance. In order to ensure that each cloud application change plan can achieve the expected results (that is, the performance is improved after the application change), grayscale testing is required before the cloud application is officially launched. Grayscale testing aims to verify whether the corresponding performance indicator data change trend is in line with expectations when the application performs the same task before and after the change, so as to evaluate the effectiveness and reliability of the change. However, in this verification process, when it comes to aggregating performance indicator data from multiple instances, [1] Simpson's paradox may occur. Simpson's paradox refers to the fact that the same data set shows opposite trends in overall observation and group observation. In the cloud application change scenario, Simpson's paradox is specifically manifested in that the trend of changes in the aggregated indicators used to measure the overall performance of the cloud application may not accurately reflect the real effect of the change. More seriously, in large-scale data centers, cloud applications change frequently, and each change may be accompanied by the occurrence of Simpson's paradox. If there is a lack of effective detection of this phenomenon, changes that can improve the overall performance of cloud applications in grayscale testing may actually reduce performance after actual implementation. Conversely, it may also lead to the negation of change plans that can effectively improve application performance and thus not be formally implemented. Therefore, in order to ensure that cloud application changes meet the expected goals and prevent being misled by erroneous conclusions drawn from the data before the changes are formally implemented, it is particularly necessary to detect Simpson's paradox on grayscale test data.

[0003] In early research [2][3] In the data set, the data volume is small and the variables are few, so the Simpson's paradox phenomenon in the data can be found by manual calculation. Today, the early Simpson's paradox phenomenon detection based on manual calculation is no longer able to meet the needs of today's large and complex data sets. For this reason, some automated Simpson's paradox phenomenon detection methods have been derived. [4] et al. divided the common Simpson's paradox phenomenon into two categories according to the type of reversal trend: the first trend is that the relative ratio of the two groups of results is reversed, that is, the sign of the difference in the ratio of the contingency table is reversed; the second trend is that the sign of the correlation between the two variables is reversed. In response to the second type of Simpson's paradox phenomenon, Xu et al. proposed a detection method that directly divides the entire data set into multiple subgroups using categorical variables. Wang [5]Aiming at the problem of detecting the second type of Simpson's paradox phenomenon, et al. proposed an intelligent detection tool Simnet based on machine learning, which transforms the detection task into an optimization task and can search for the optimal grouping method from potential grouping methods within a limited time and complete the detection. [6][7] All of them adopt the detection method of traversing all potential grouping methods. Compared with the early manual calculation detection method, the automated detection method improves the detection efficiency and accuracy of the Simpson's paradox phenomenon. However, when applying the existing Simpson's paradox phenomenon automated detection method to cloud application change scenarios, there are still some shortcomings and limitations:

[0004] 1. Lack of unified detection methods. In cloud application change scenarios, due to the wide variety of cloud applications and the complexity of change types, the performance indicators that need to be verified in each grayscale test are different. Currently, there is a lack of relevant research to propose a unified detection method for the Simpson's paradox phenomenon in different situations in cloud application change scenarios. If there is no comprehensive and effective detection method applied to cloud application change scenarios, misleading information in the data set may be found, which may lead to errors in cloud application change decisions, and then cause the change plan that reduces cloud application performance to be formally implemented or the change plan that improves cloud application performance to be rejected.

[0005] 2. Large detection space and low detection efficiency. The Simpson's paradox phenomenon that occurs in cloud application change scenarios is mainly of the first type. Existing work mainly uses detection methods that traverse all grouping methods. When the number of features used for grouping is large, the detection space of the existing work for the first type of Simpson's paradox phenomenon detection method is large and time-consuming; when the value range of a certain grouping feature exceeds a certain value, the number of potential grouping methods of the grouping feature is very large. [5] , it is impossible to complete the traversal within a limited time. Summary of the invention

[0006] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a detection method and system for the Simpson's paradox phenomenon in cloud application change scenarios, specifically including a detection model for the Simpson's paradox phenomenon in cloud application change scenarios (including version iteration, configuration update, etc.) and integrating an automated detection method and system, adopting different detection strategies to adapt to different detection needs, and being able to detect the Simpson's paradox phenomenon in the performance indicator change trend before and after the change. Specifically, according to different data sets in different cloud application change scenarios, an applicable detection method is automatically selected to detect the Simpson's paradox phenomenon.

[0007] The present invention mainly solves two technical problems. First, for the wide variety of applications and performance indicators in cloud application change scenarios, as well as the diversity of change types, how to construct a general detection method that can adapt to different situations is an important issue. The present invention strives to cover various types of Simpson's paradox phenomena in different cloud application change scenarios by constructing a detection model for the Simpson's paradox phenomenon in which the sign of the difference between the ratios of multiple performance indicators is reversed, thereby improving the breadth and applicability of detection. Second, how to reduce the detection space and improve the efficiency of detection tools. In cloud application change scenarios, due to the diversity of grouping features and performance indicators, the scale of the detection space is too large, and traditional strategies are difficult to handle effectively. In addition, when the intelligent grouping detection strategy is adopted, the number of potential grouping methods conforms to the Bell number law, and when the value range of the grouping feature exceeds a certain value, it cannot be traversed within a limited time. The present invention solves the detection bottleneck caused by the large detection space and insufficient traversal efficiency by optimizing the grouping strategy and detection algorithm, significantly improving the efficiency and capability of the detection tool, so that it can efficiently complete the detection of the Simpson's paradox phenomenon in the cloud application change scenario.

[0008] The present invention provides a method for detecting the Simpson's paradox phenomenon in a cloud application change scenario, the detection method comprising the following steps:

[0009] Step 1: Build a unified performance indicator model based on the performance indicators of cloud application instances;

[0010] Step 2: Determine whether the trend of performance indicators meets the necessary conditions, exclude performance indicators that are unlikely to cause Simpson's paradox, and narrow the detection space;

[0011] Step 3: Count the range of each grouping feature, and select a detection strategy based on the number of elements in the range;

[0012] Step 4: Implement the selected detection strategy and output the detection results.

[0013] In step 1, the performance indicators include resource utilization, average response time, average instruction cycle number, etc.; the unified model of the performance indicators is expressed as follows: performance indicator = performance indicator numerator item / performance indicator denominator item;

[0014] In a specific implementation, the resource utilization may further include CPU utilization, memory utilization, disk utilization, etc.;

[0015] The numerator of the performance index of the CPU utilization is the CPU usage, and the denominator of the performance index is the CPU specification;

[0016] The numerator of the performance indicator of the memory utilization is the memory usage, and the denominator of the performance indicator is the memory specification;

[0017] The numerator of the performance index of the disk utilization is the disk usage, and the denominator of the performance index is the disk specification;

[0018] The numerator of the performance indicator of the average response time is the total request time, and the denominator of the performance indicator is the number of requests;

[0019] The numerator of the performance indicator of the average number of instruction cycles is the number of instruction cycles, and the denominator of the performance indicator is the number of instructions.

[0020] The numerator item data and the denominator item data of the performance indicator are obtained through a monitoring system or by using a performance analysis tool including perf.

[0021] In step 2, the performance indicators in step 1 are used to determine whether the Simpson's paradox occurs according to the following formula:

[0022]

[0023] Among them, a j represents the numerator before the change is applied, b j represents the denominator before the change is applied, c j Indicates the numerator term after application of the change, d j It represents the denominator after the application is changed. It is judged by whether the change trend of the cloud application performance indicators in each subgroup is opposite to the overall change trend. If the change trends of the performance indicators of all subgroups are opposite, it means that the Simpson's paradox phenomenon has occurred. If there is at least one subgroup with the same change trend as the overall trend, it means that the Simpson's paradox phenomenon has not occurred.

[0024] The necessary conditions for the above-mentioned Simpson's paradox phenomenon to occur are as follows:

[0025]

[0026] This formula is used to pre-screen the grouping features and performance indicators that may lead to the Simpson's paradox phenomenon, and narrow the detection space of the Simpson's paradox phenomenon.

[0027] In step 2, for each performance indicator, the ratio difference before and after the change is applied in each group is calculated; when the ratio difference of the performance indicator does not meet the necessary conditions for the sign change in all groups, it is eliminated from the detection space; and a set of performance indicators that meet the conditions is generated for use in subsequent detection steps.

[0028] In step three, the value range of the grouping feature is counted. When the number of elements in the value range is less than or equal to 1, Simpson's paradox phenomenon detection is not performed; when the number of elements in the value range is equal to 2, the potential grouping method is unique, and a direct grouping detection strategy is used; when the number of elements in the value range is greater than 2, an intelligent grouping detection strategy based on machine learning is used to search for a grouping method that satisfies Simpson's paradox among multiple potential grouping methods.

[0029] The direct group detection strategy includes the following steps:

[0030] Step i, group the data according to the value range of the grouping feature, and each group of data corresponds to a subset;

[0031] Step ii: for each subset of data, calculating the change trend of the performance indicator based on the unified model;

[0032] Step iii, determine whether the change trend of the performance indicator after grouping is opposite to the change trend of the overall data;

[0033] Step iv: If there is a group with reversed sign, it is determined to be Simpson's paradox phenomenon, otherwise, Simpson's paradox phenomenon does not exist;

[0034] In step 4, the detection results of relevant grouping characteristics and performance indicators are output;

[0035] and / or,

[0036] The intelligent grouping detection strategy based on machine learning includes the following steps:

[0037] Step a, define the value range mapping of the grouping feature;

[0038] Step b, using a multi-layer perceptron model to train the input data to minimize a loss function, wherein the loss function is defined according to the matching degree of the subgroup ratio difference with the overall ratio difference having opposite signs;

[0039] Step c, learning an optimized group mapping method;

[0040] Step d, determining whether the change trend of the performance index after grouping satisfies the Simpson's paradox condition;

[0041] In step 4, the detection results are output, including the optimized grouping method and the detected performance indicators and grouping characteristics.

[0042] The present invention also provides a detection system for implementing the above detection method, the detection system comprising: a data layer, a strategy layer, and a computing layer;

[0043] The data layer is used to receive performance data of cloud applications and store input data tables including numerator items, denominator items and grouping characteristics;

[0044] The strategy layer dynamically selects direct grouping or intelligent grouping detection strategy according to the grouping feature value domain of the input data table;

[0045] The computing layer detects the performance indicators based on the selected strategy and outputs the results and related data of whether there is Simpson's paradox phenomenon.

[0046] The present invention also provides the above detection method, or the application of the above detection system in unifying the Simpson's paradox phenomenon detection in cloud application change scenarios, reducing the detection space, improving the detection efficiency, etc.

[0047] The beneficial effects of the present invention include: the present invention adopts a data-driven method to establish a unified model of multi-instance cloud application diversified performance indicators for Simpson's paradox phenomenon detection, covering various types of Simpson's paradox phenomena in different cloud application change scenarios, and improving the breadth and applicability of the detection method. On the basis of this model, a method for reducing the detection space is further proposed to effectively reduce the complexity of subsequent detection tasks, improve the efficiency of detection tools, and reduce detection time. When facing different input data to be detected, it can automatically select an applicable detection strategy to detect the Simpson's paradox phenomenon according to the value range of the grouping feature in the data. Compared with the direct grouping detection strategy, the proposed intelligent grouping strategy based on machine learning applied to the cloud application change scenario takes into account the grouping method that all values ​​of the grouping feature can form a new subgroup after combination, and can detect more potential Simpson's paradox phenomena. In addition, the strategy converts the detection task into an optimization task. In the case of a large order of magnitude of potential grouping methods, the detection results can be obtained within a limited time, which effectively improves the detection capability.

[0048] Based on the detection results, the present invention helps to correct the misleading performance indicator change trend of cloud applications during grayscale testing, provides more reliable data support for cloud application change decisions, ensures that the implementation effect of cloud application changes meets expectations, and also provides a certain reference value for how to reduce the possibility of Simpson's paradox through the design of cloud application change grayscale testing experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 It is a flow chart of the detection method of the present invention.

[0051] Figure 2It is a diagram of the detection system architecture of the present invention. DETAILED DESCRIPTION

[0052] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0053] The present invention provides a detection method for the Simpson's paradox phenomenon in cloud application change scenarios, and the detection method includes the following steps: Step 1, constructing a unified performance indicator model based on the performance indicators of cloud application instances; Step 2, judging whether the change trend of the performance indicators meets the necessary conditions, excluding performance indicators that are unlikely to cause the Simpson's paradox phenomenon, and narrowing the detection space; Step 3, counting the value range size of each grouping feature, and selecting a detection strategy based on the number of elements in the value range; Step 4, implementing the selected detection strategy and outputting the detection result. The present invention also provides a detection system for implementing the above detection method, which has a wide range of application value.

[0054] Specifically, the present invention proposes a method for detecting the Simpson's paradox phenomenon in a cloud application change scenario. The detection method specifically involves a unified modeling method for diversified performance indicators of cloud applications for detecting the Simpson's paradox phenomenon and a method for narrowing the detection space based on the model. A detection strategy selection mechanism using a direct grouping detection strategy or an intelligent grouping strategy based on machine learning is used to achieve detection of the Simpson's paradox phenomenon.

[0055] 1. Unified modeling method for diverse performance indicators of cloud applications for Simpson’s paradox detection

[0056] As shown in Table 1, the main performance indicators of cloud applications include: resource (including CPU, memory, disk, etc.) utilization, average response time, and average instruction cycle number. The calculation formula for the utilization efficiency of the above-mentioned performance indicators can be summarized as: performance indicator = performance indicator numerator / performance indicator denominator. For each instance of cloud applications, the numerator and denominator data of these performance indicators can be obtained by the monitoring system or engineers using performance analysis tools (such as perf). Using the numerator and denominator data of a single instance, when calculating the performance indicators of multiple instances of cloud applications, it involves the aggregation of multiple instance indicators. For example, to calculate the CPU utilization of an application in a certain period of time, it is necessary to sum the CPU resource usage and CPU resource specifications of all instances of the application in the period of time to obtain the total CPU resource usage and the total CPU resource specifications, and then divide them to obtain the final result. The calculation method of other performance indicators in Table 1 is similar.

[0057] Table 1 Examples of calculation methods for multiple cloud application performance indicators

[0058] Performance Indicators Performance index numerator Performance index denominator CPU Utilization CPU usage CPU Specifications Memory Utilization Memory usage Memory specifications Disk Utilization Disk usage Disk specifications Average response time Total request time Number of requests Average instruction cycles Instruction cycle number Number of instructions

[0059] Whenever an application changes, the above performance indicators change in trend before and after the change due to the characteristics of their aggregation method. [1] , may be confused by some characteristics of the application instance (such as the deployed data center, computer room, machine model, instance type, etc.), resulting in Simpson's paradox phenomenon. As shown in Table 2, grouping applications according to a certain characteristic will result in Simpson's paradox phenomenon. Therefore, in the cloud application change scenario, the cloud application performance indicators in Table 1 can be modeled using the contingency table in Table 2, and condition (1) is used to determine whether Simpson's paradox phenomenon occurs, that is, to determine whether the change trend of cloud application performance indicators in each subgroup is opposite to the overall change trend. If the change trend of performance indicators of all subgroups is opposite, it means that Simpson's paradox phenomenon has occurred; if there is at least one subgroup with the same change trend as the overall trend, it means that Simpson's paradox phenomenon has not occurred.

[0060]

[0061] Among them, sgn is a sign function that maps positive numbers, zeros, and negative numbers to 1, 0, and -1, respectively. It is used here to characterize the change trend of cloud application performance indicators before and after the change. "1", "0", and "-1" represent a decrease, no change, and an increase in trend, respectively. Condition (1) The equality of the first two equations represents that the change trend of any two subgroup application performance indicators is the same, and the third equation represents that the overall change trend of cloud application performance indicators is opposite to the change trend of any subgroup application performance. And "1" and "-1" represent opposite trends. The above three equations cannot be equal to 0.

[0062] The contingency table intuitively compares the grouped data and the overall data before and after the application change, and describes the difference in the change of the cloud application performance indicators of the group or the whole in the form of ratio difference. On this basis, the sign of the difference in the change of the above cloud application performance indicators is determined according to formula (1) to complete the Simpson's paradox phenomenon detection.

[0063] Table 2 Contingency table for Simpson's paradox test

[0064]

[0065] 2. Detection method and system of Simpson's paradox phenomenon in cloud application change scenario

[0066] The detection method and detection system in the present invention are constructed on the basis of a unified model of cloud application diversified performance indicators for Simpson's paradox phenomenon detection. In order to verify the effect of a certain change in a cloud application, the grouping feature is used to find the grouping method in the data that conforms to Simpson's paradox. The specific method is to judge whether each grouping feature will confuse the change trend of a certain performance indicator. The flowchart of the detection method is as follows: Figure 1 As shown in the figure, the architecture diagram of the detection system is as follows Figure 2 As shown, the system architecture is divided into a data layer, a strategy layer and a computing layer.

[0067] The detection method in the present invention specifically includes the following 4 steps:

[0068] Step 1. Input data

[0069] Table 3 is the header of the input data table used for Simpson's paradox phenomenon detection in this method. The data table includes the numerator and denominator data of the performance indicators of multiple instances of the current application before and after a certain change, as well as the feature data used for grouping. Each row of the complete data table represents the grayscale test data of a cloud instance, where the change status includes the two states before and after the cloud application changes, the numerator and denominator of x current key performance indicators of the change, and y features that can be used for grouping (including the data center, cluster, computer room where the instance is located, the deployed model, etc.).

[0070] Table 3 Input data header

[0071] Change Status Molecular term 1 Denominator 1 Molecular term 2 Denominator 2 … Numerator x Denominator x Feature 1 Feature 2 … Feature

[0072] Step 2. Detect space reduction

[0073] Given a feature j (j = 1, 2, ..., y) used for grouping, traverse each performance indicator p (p = 1, 2, ..., x) in turn to detect whether feature j confuses the performance indicator p before and after the cloud application change. Before the formal detection, the necessary conditions of condition (1) are determined to screen out performance indicators that are unlikely to cause the Simpson's paradox phenomenon, thereby reducing the detection space. The necessary conditions of condition (1) are as follows:

[0074]

[0075] Condition (2) indicates that the difference between the maximum and minimum values ​​of the performance indicators in all groups is greater than the absolute value of the difference between the overall performance indicators before and after the application change. Condition (2) is a necessary condition for the Simpson's paradox in the detection model of the present invention to hold true. It can be understood that if feature j wants to confuse the trend of performance indicator p before and after the cloud application change, the effect size of feature j on the performance indicator needs to exceed the effect size of the cloud application change itself on the performance indicator. [8] .

[0076] If condition (2) is not met, the current detection feature j will not confuse the change trend of the performance indicator p before and after the cloud application change, so the Simpson's paradox phenomenon will not occur and it will not be included in the subsequent detection process; otherwise, the Simpson's paradox phenomenon may occur, and feature j and performance indicator p need to be included in the subsequent detection process.

[0077] Step 3. Detection strategy selection

[0078] For each feature j (j = 1, 2, ..., y) used for grouping, the range of the feature is counted and used as the basis for the selection of the detection strategy. When the number of elements in the range is less than or equal to 1, the Simpson's paradox phenomenon is not detected (the feature cannot form a group); when the number of elements in the range is equal to 2, the potential grouping method is unique, and the direct grouping detection strategy is used; when the number is greater than 2, the intelligent grouping detection strategy based on machine learning is used to search for a grouping method that satisfies the Simpson's paradox among many potential grouping methods. After the detection strategy for each grouping feature j is selected, the selected detection strategy is used to detect each performance indicator p (p = 1, 2, ..., x) screened in step 2 in turn in step 4.

[0079] Step 4. Implement the detection strategy and output the results

[0080] Direct group detection strategy:

[0081] First, according to the given feature j and performance index p, the input data is directly grouped according to the different values ​​of feature j (the range of feature j is 2, divided into two groups) according to the unified modeling method of diversified performance indicators of cloud applications, and aggregated to obtain a contingency table in the form of Table 2. The contingency table contains performance index-related data of multiple groups and overall data before and after the cloud application change.

[0082] Second, according to condition (1), detect whether there is Simpson's paradox in the contingency table, that is, it can be concluded: directly according to the grouping of feature j, whether the trend of indicator p has reversed. If Simpson's paradox is established, output the indicator p and feature j that form Simpson's paradox, as well as the data results in the form of Table 2.

[0083] Intelligent grouping detection strategy based on machine learning:

[0084] In cloud application change scenarios, the features used for grouping may include multiple values, such as a cloud instance can be deployed on machines in multiple clusters. The strategy of grouping directly based on different feature values ​​ignores the method of combining different feature values ​​to form a group and then performing Simpson's paradox detection after forming fewer groups. Therefore, this strategy is implemented through the following steps.

[0085] a. For a given feature j and performance index p, this strategy converts the detection task into an optimization task. By setting the loss function, the ratio difference of each subgroup is made to have the opposite sign to the overall data ratio difference as much as possible, so as to search for potential Simpson's paradox phenomena. Define C as the mapping from different values ​​of feature j to specific groups (those mapped to the same group are grouped together), T is the instance change state, A is the numerator of the index, and B is the denominator of the index. It is the difference of the overall performance index of cloud applications.

[0086] b. Set the loss function to The multi-layer perceptron model is used to learn the grouping method in the data that may form the Simpson's paradox phenomenon. MRD represents the ratio difference of all groups calculated by mapping C and data items T, A and B (the ratio difference of the kth group is ), avg is the averaging function, averaging the ratio differences of all groups, minimizing the loss function By combining the overall performance index change trend with the average change difference of each group performance index, the samples with opposite change trends to the overall data and consistent change trends within the groups are clustered in the same group.

[0087] c. Use the above strategy to learn the input data to obtain the grouping mapping C, that is, the grouping method.

[0088] d. Use condition (1) to determine whether Simpson’s paradox holds when grouping method C is used.

[0089] e. If Simpson's paradox holds true, output the index p and feature j that form Simpson's paradox, the grouping mapping C, and the data results as shown in Table 3.

[0090] Example 1: Direct group detection strategy

[0091] In order to improve the resource utilization (CPU utilization and memory utilization) of the application and reduce resource costs, the instance specifications of a cloud database application that meets certain conditions are downgraded. In order to verify the effect of this change, the resource usage data of all instances of the application before and after the change were selected for analysis, including the numerator CPU usage and memory usage and the denominator CPU specifications and memory specifications related to resource utilization before and after the change of all instances, as well as the implementation of downgrades, which can be used for grouping, and whether these features cause the change trend of resource utilization before and after the change to form the Simpson's paradox phenomenon to be detected.

[0092] When applying the detection method of the present invention, the detection space is first narrowed down according to condition (2). For example, whether the feature downgrade situation causes the change trend of cloud application memory utilization to form the Simpson's paradox phenomenon needs further detection and cannot be directly screened out, because according to the data in Table 4, Condition (2) is satisfied.

[0093] Since the implementation of downgrade is a binary variable, this feature divides all instances into a group with unchanged specifications and a group with implemented downgrade. Therefore, a direct grouping strategy is used to detect whether there is a Simpson's paradox phenomenon in the change trend of resource utilization. The Simpson's paradox phenomenon of the memory utilization change trend detected in this embodiment is shown in Table 4. The overall memory utilization of all instances decreased from 70.09% to 61.50% before and after the change. If only this indicator is considered, it can be found that this change has greatly reduced the memory utilization of the application, and the conclusion that this change is unreasonable is drawn. However, the memory utilization of the instance of the unchanged specification group increased from 53.63% to 53.68%, and the memory utilization of the instance of the downgrade group increased from 82.46% to 85.00%, all of which have been improved to varying degrees. This phenomenon shows that this change has played a role in improving the memory utilization of the application. At the same time, it also shows that the change trend of a single overall memory utilization indicator cannot effectively reflect the effect of this change. Through the detection of the Simpson's paradox phenomenon, some misleading information can be found to avoid this information from having an adverse impact on the final decision.

[0094] Table 4 Output results of Example 1

[0095]

[0096] Example 2: Intelligent grouping detection strategy based on machine learning

[0097] In order to improve the service quality of a cloud application, engineers optimized the response time of the application. Before the optimization is officially launched, grayscale testing is required to verify the optimization effect. The input data includes the total response time of each instance of the application before and after the change, the number of requests, the deployed clusters, and other data. The following is an explanation of the Simpson's paradox phenomenon detected by this tool that the average response time of the application is changing due to the deployed clusters:

[0098] Table 5 is the data of the cloud application classified and summarized according to the change status and deployment cluster. Because a total of 5 clusters have deployed the application, the intelligent group detection strategy based on machine learning is used for detection, and the output mapping is shown in Table 6. According to the mapping relationship in Table 6, the data in Table 5 is aggregated to obtain Table 7, and the Simpson's paradox phenomenon is determined to be established by condition (1). It can be found from the data that the average response time of the instance deployed in cluster a of the cloud application is significantly higher than the overall average response time before and after the application change, and the number of instance processing requests in cluster a before the change in this grayscale test accounts for only about 2% of the total, while the number of instance processing requests in cluster a after the change accounts for about 27% of the total, which significantly improves the overall average response time after the application change. This becomes the main reason why the average response time of the overall data application after the change is higher than before the change. The intelligent group detection strategy adopted by the present invention separates samples with the opposite trend of the overall performance index change by optimizing the loss function, and separates cluster a into a group alone, and the remaining cluster categories into a second group. Therefore, this grayscale test cannot reject this optimization simply based on the information that the overall average response time of all instances has increased, because this optimization reduces the average response time for instances in both groups, and the cluster becomes a feature that confuses the results of this grayscale test. The present invention not only discovers misleading information, but also provides reference value for the design of subsequent grayscale test schemes by detecting features that are easy to confuse test results. Subsequent grayscale tests can fully consider the impact of features that are easy to confuse test results during the scheme design stage. For example, when designing a grayscale test experiment in the current case, the cluster categories of instance deployment should be balanced to ensure that the proportion of requests processed by each cluster instance in the grayscale test is consistent with the actual production environment as much as possible.

[0099] Table 5 is a cloud application response time data table, which includes the total response time, request volume, and average response time of different cluster data before and after the change;

[0100] Table 6 is a cluster and group mapping result table, which is obtained by learning the intelligent detection grouping strategy, and contains multiple clusters and their grouping groups;

[0101] Table 7 is the output result table of Example 2, which is obtained by summarizing Table 5 according to the grouping method of Table 6, and includes the total response time, request volume and average response time of the two groups of data and the overall data before and after the cloud application change.

[0102] Table 5 Response time data of a cloud application

[0103]

[0104]

[0105] Table 6 Cluster and group mapping results

[0106] Cluster Grouping a Group 1 b Group 2 c Group 2 d Group 2 e Group 2

[0107] Table 7 Output results of Example 2

[0108]

[0109] Example 3: Direct group detection strategy

[0110] In order to verify whether a cloud application can save more resources after a certain optimization, that is, consume less resources under the premise of completing the same workload. The resource usage data before the change and the grayscale test resource usage data of all instances of the same size and specifications of the application under the same workload were selected for analysis, including the numerator items CPU usage, memory usage, disk usage and the denominator items CPU specifications, memory specifications, disk specifications, and machine models, clusters and other features that can be used for grouping. Since the instance specifications in this experiment are consistent and the resource utilization can reflect the size of resource usage, it is tested whether these features used for grouping cause the change trend of resource utilization before and after the change to form the Simpson's paradox phenomenon.

[0111] When applying the detection method of the present invention, the detection space is first narrowed down according to condition (2). For example, whether the model of the instance deployment causes the change trend of the cloud application CPU utilization to form the Simpson's paradox phenomenon needs further detection and cannot be directly screened out, because according to the data in Table 4, Satisfies condition (2). For each grouping feature, it is necessary to detect whether the performance indicators that are not eliminated from the test space have Simpson's paradox phenomenon. The following is a further explanation of the final test results:

[0112] Since there are two types of models for the cloud application deployment, the model feature is a binary variable. This feature divides all instances into model group 1 and model group 2. Therefore, the direct grouping strategy is used to detect whether there is a Simpson's paradox phenomenon in the change trend of resource utilization. The Simpson's paradox phenomenon of the CPU utilization change trend detected in this embodiment is shown in Table 8. The overall memory utilization of all instances decreased from 8.55% to 7.30% before and after the change. If only this indicator is considered, it can be found that this change reduces the CPU utilization of the application and reduces the resources required to complete the same workload. It is concluded that this change is reasonable and effective. However, the CPU utilization of the model 1 group instances increased from 5.03% to 5.70% and the CPU utilization of the model 2 group instances increased from 8.68% to 8.90%, all of which have been improved to varying degrees. This phenomenon shows that this change has played a role in improving the CPU utilization of the application and increased the CPU resource consumption. It also shows that the change trend of a single application's overall CPU utilization indicator cannot effectively reflect the effect of this change. Through the Simpson's paradox phenomenon detection, misleading information can be discovered to avoid adverse effects of this information on the final decision.

[0113] Table 8 Output results of Example 3

[0114]

[0115] References

[0116] [1]RUDAS T.Lectures on Categorical Data Analysis[M]. New York, NY: Springer US, 2018

[0117] [2]Peter JB,EA Hammel,and JW O'connell.1975.Sex Bias in GraduateAdmissions:Data from Berkeley[J].Science 1975(187):398–404.

[0118] [3]BLYTH C R.On Simpson's Paradox and the Sure-Thing Principle[J].Journal of the American Statistical Association,1972,67(338):364-366.

[0119] [4]CHENGUANG X,Sarah M B,and CHRISTAN G.Detecting Simpson’s Paradox[J].In Proceedings of the Thirty-First International Florida ArtificialIntelligence Research Society Conference,2018,221–224.

[0120] [5]WANG J,HE J,XU W,et al.Learning to Discover Various Simpson’sParadoxes[C].Proceedings of the 29th ACM SIGKDD Conference on KnowledgeDiscovery and Data Mining.Long Beach CA,2023:5092-5103.

[0121] [6]TIAN S,ZHU X.Data analytics in transport:Does Simpson’s paradoxexist in rule of ship selection for port state control?[J].ElectronicResearch Archive,2023,31(1):251-272.

[0122] [7]Jay Xu,Jian Pei,and Zicun Cong.Finding Multidimensional Simpson'sParadox[C].SIGKDD Explor.Newsl.2022,24(2):48–60.

[0123] [8]Schield M.Simpson’s paradox and Cornfield’s conditions[J].ASAProc.Sect.Stat.Educ,1999:106-111.

[0124] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0129] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A detection method for Simpson's paradox phenomenon in cloud application change scenarios, characterized in that: The detection method comprises: Step 1: Build a unified performance indicator model based on the performance indicators of cloud application instances; Step 2: Determine whether the trend of performance indicators meets the necessary conditions, exclude performance indicators that are unlikely to cause Simpson's paradox, and narrow the detection space; Step 3: Count the range of each grouping feature, and select a detection strategy based on the number of elements in the range; Step 4: Implement the selected detection strategy and output the detection results.

2. The detection method according to claim 1, characterized in that In step 1, the performance indicators include resource utilization, average response time, and average instruction cycle number; the resource utilization includes CPU utilization, memory utilization, and disk utilization; The unified model of performance index is expressed as follows: performance index = performance index numerator item / performance index denominator item; The numerator of the performance index of the CPU utilization is the CPU usage, and the denominator of the performance index is the CPU specification; The numerator of the performance indicator of the memory utilization is the memory usage, and the denominator of the performance indicator is the memory specification; The numerator of the performance index of the disk utilization is the disk usage, and the denominator of the performance index is the disk specification; The numerator of the performance indicator of the average response time is the total request time, and the denominator of the performance indicator is the number of requests; The numerator of the performance indicator of the average number of instruction cycles is the number of instruction cycles, and the denominator of the performance indicator is the number of instructions.

3. The detection method according to claim 1, characterized in that In step 2, the performance indicators in step 1 are used to determine whether the Simpson's paradox occurs according to the following formula: Among them, a j represents the numerator before the change is applied, b j represents the denominator before the change is applied, c j Indicates the numerator term after application of the change, d j Indicates the denominator after the application is changed; it is determined by whether the change trend of the cloud application performance indicators in each subgroup is opposite to the overall change trend. If the change trends of the performance indicators of all subgroups are opposite, it means that the Simpson's paradox phenomenon has occurred; if there is at least one subgroup with the same change trend as the overall trend, it means that the Simpson's paradox phenomenon has not occurred; The necessary conditions for the above-mentioned Simpson's paradox phenomenon to occur are as follows: It is used to pre-screen the grouping features and performance indicators that may cause the Simpson's paradox phenomenon, and narrow the detection space of the Simpson's paradox phenomenon.

4. The detection method according to claim 3, characterized in that In step 2, for each performance indicator, the ratio difference before and after the change is applied in each group is calculated; when the ratio difference of the performance indicator does not meet the necessary conditions for the sign change in all groups, it is eliminated from the detection space; and a set of performance indicators that meet the conditions is generated for use in subsequent detection steps.

5. The detection method according to claim 1, characterized in that In step three, the range of the grouping feature is counted. When the number of elements in the range is less than or equal to 1, Simpson's paradox phenomenon detection is not performed; when the number of elements in the range is equal to 2, the potential grouping method is unique, and a direct grouping detection strategy is used; when the number of elements in the range is greater than 2, an intelligent grouping detection strategy based on machine learning is used to search for a grouping method that satisfies Simpson's paradox among multiple potential grouping methods.

6. The detection method according to claim 5, characterized in that The direct group detection strategy includes the following steps: Step i, group the data according to the value range of the grouping feature, and each group of data corresponds to a subset; Step ii: for each subset of data, calculating the change trend of the performance indicator based on the unified model; Step iii, determine whether the change trend of the performance indicator after grouping is opposite to the change trend of the overall data; Step iv: If there is a group with reversed sign, it is determined to be Simpson's paradox phenomenon, otherwise there is no Simpson's paradox phenomenon.

7. The detection method according to claim 5, characterized in that: The intelligent grouping detection strategy based on machine learning includes the following steps: Step a, define the value range mapping of the grouping feature; Step b, using a multi-layer perceptron model to train the input data to minimize a loss function, wherein the loss function is defined according to the matching degree of the subgroup ratio difference with the overall ratio difference having opposite signs; Step c, learning an optimized group mapping method; Step d: determine whether the trend of the performance index after grouping satisfies the Simpson's paradox condition; if there is a grouping with sign reversal, it is determined to be the Simpson's paradox phenomenon, otherwise, the Simpson's paradox phenomenon does not exist.

8. The detection method according to claim 1, characterized in that In step 4, when the direct grouping detection strategy is adopted, when the Simpson's paradox phenomenon exists, the detection results of the relevant grouping features and performance indicators are output; When an intelligent grouping detection strategy based on machine learning is adopted, when the Simpson's paradox phenomenon exists, the output includes the optimized grouping method, the detected performance indicators and the detection results of the grouping characteristics.

9. A detection system for implementing the detection method according to any one of claims 1 to 8, characterized in that: The detection system includes: a data layer, a strategy layer, and a computing layer; The data layer is used to receive performance data of cloud applications and store input data tables including numerator items, denominator items and grouping characteristics; The strategy layer dynamically selects direct grouping or intelligent grouping detection strategy according to the grouping feature value domain of the input data table; The computing layer detects the performance indicators based on the selected strategy and outputs the results and related data of whether there is Simpson's paradox phenomenon.

10. Application of the detection method according to any one of claims 1 to 8, or the detection system according to claim 9, in unifying the detection of Simpson's paradox phenomenon in cloud application change scenarios, reducing the detection space, and improving the detection efficiency.

Images