Data processing method and computer program product

Abstract

Embodiments of the present description provide a data processing method and a computer program product. The method comprises: obtaining a working performance sequence in a target performance pattern, wherein the working performance sequence comprises working performance pairs consisting of working loads of performance acquisition time points and resource usage rates corresponding to the working loads; on the basis of a preset time window and a preset time interval, performing sequential division on the working performance sequence to determine a plurality of sub-working performance sequences; determining, from among the plurality of sub-working performance sequences, a target sequence and a reference sequence adjacent to and preceding the target sequence, and determining a target difference distance between the target sequence and the reference sequence on the basis of the target sequence and the reference sequence; and detecting the target performance pattern on the basis of the target difference distance to obtain a performance detection result. Automatic identification and detection of performance pattern drift is implemented on the basis of the target difference distance, and the timeliness and accuracy of detection are improved.

Description

Data processing methods and computer program products

[0001] This application claims priority to Chinese Patent Application No. 202410501967.1, filed on April 24, 2024, entitled "Data Processing Method and Computer Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing method. Background Technology

[0003] In recent years, a large number of microservices have been widely deployed in large-scale data centers. Typically, in a microservice architecture, an application is composed of a large number of lightweight and stateless microservices that coordinate to handle user request workloads by calling each other. Each microservice contains multiple instances as a container to manage large workloads in parallel; this approach helps to automatically scale the resource capacity of microservices by starting or stopping containers.

[0004] Given the highly dynamic nature and potential anomalies of microservice workloads, the industry currently focuses on achieving proactive capacity planning. The core of this work is accurately constructing the relationship between microservice performance and workload, i.e., performance patterns. Based on this, elastic scaling frameworks predict and plan future resource demands and capacity by forecasting future workload scale and the established performance patterns, ultimately achieving elastic scaling of microservice resource capacity (i.e., adding or removing containers). Specifically, for frameworks that ensure stable CPU (Central Processing Unit) utilization, the core is constructing CPU usage patterns; however, in actual production, especially in large-scale data center environments, reasonably constructing CPU usage patterns and ensuring stable CPU utilization on microservice containers faces significant challenges.

[0005] Specifically, the factors influencing microservice performance patterns can be broadly categorized into two aspects: internal factors and external factors. The primary internal factor is business behavior. When microservices change their behavior through factors such as version upgrades and hardware upgrades, their performance patterns may also change. However, effectively adapting to time-varying performance patterns for container autoscaling decisions remains an open challenge. Furthermore, automatically quantifying the impact of pattern changes in a timely and precise manner in actual production remains an open and unresolved issue. This limitation can lead to a decline in QoS (Quality of Service) and even affect service availability. Summary of the Invention

[0006] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0007] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0008] Obtain the work performance sequence under the target performance mode, wherein the work performance sequence includes work performance pairs consisting of the workload at each performance acquisition time point and the resource utilization rate corresponding to the workload;

[0009] The work performance sequence is sequentially divided according to a preset time window and a preset time interval to determine multiple sub-work performance sequences;

[0010] From the plurality of sub-work performance sequences, a target sequence and the previous reference sequence adjacent to the target sequence are determined, and the target difference distance between the target sequence and the reference sequence is determined based on the target sequence and the reference sequence.

[0011] Based on the target difference distance, the target performance pattern is detected to obtain the performance detection result.

[0012] According to a second aspect of the embodiments of this specification, a data processing system is provided, including a detection unit and a scheduling unit, wherein...

[0013] The detection unit is used to apply the above-mentioned data processing method to obtain performance detection results, and to determine the current performance mode when the performance detection result is the target detection result.

[0014] The scheduling unit is configured to determine a container adjustment strategy for the container based on the predicted workload and the current performance mode, and to expand or shrink the container according to the container adjustment strategy.

[0015] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0016] Memory and processor;

[0017] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0019] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0020] An embodiment of this specification provides a data processing method comprising: acquiring a work performance sequence under a target performance mode, wherein the work performance sequence includes work performance pairs consisting of workload at each performance acquisition time point and resource utilization corresponding to the workload; sequentially dividing the work performance sequence according to a preset time window and a preset time interval to determine multiple sub-work performance sequences; determining a target sequence and the previous reference sequence adjacent to the target sequence from the multiple sub-work performance sequences; determining a target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence; and detecting the target performance mode based on the target difference distance to obtain a performance detection result.

[0021] Based on this, the data processing method acquires the working performance sequence under the target performance mode, and divides the working performance sequence into multiple sub-working performance sequences according to a preset time window and a preset time interval. Using the preset time window as the statistical granularity, the target sequence and the previous reference sequence adjacent to the target sequence are determined from the multiple sub-working performance sequences. By calculating the target difference distance between the two, the difference in performance mode between the target sequence and the reference sequence is obtained, thereby realizing the detection of the target performance mode and obtaining the detection result of whether the target performance mode has changed. Based on the quantified target difference distance, the automatic identification and detection of performance mode drift is realized, and the timeliness and accuracy of detection can be improved in the case of automatic detection. Attached Figure Description

[0022] Figure 1 is a schematic diagram of a data processing method provided in one embodiment of this specification;

[0023] Figure 2 is a flowchart of a data processing method provided in one embodiment of this specification;

[0024] Figure 3 is a schematic diagram of window sliding for drift detection in a sliding window mode according to an embodiment of this specification;

[0025] Figure 4 is a schematic diagram of a pattern drift detection method provided in one embodiment of this specification;

[0026] Figure 5 is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification;

[0027] Figure 6 is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0028] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0029] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0030] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0031] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0033] Microservice applications: Cloud applications typically use a microservice framework to break down a large cloud application into multiple modular microservices. Different microservices communicate with each other through network calls, and multiple microservices form a call chain to communicate collaboratively to fulfill user requests to the cloud application. Each specific microservice consists of multiple container replicas, and microservices are generally stateless.

[0034] Resource capacity: For a microservice framework, resource capacity refers to the total amount of resources allocated to a microservice by the cluster management system, or the resource quota required by the microservice to handle service requests while ensuring performance. Since microservices often consist of multiple stateless container replicas, it can be quantified as the number of container replicas required. For a specific resource, resource capacity can also be defined as the allocated quota, such as CPU capacity, measured in cores.

[0035] Resource capacity planning: For microservice frameworks, the capacity planning problem is to predict the resource capacity required by microservices in the future in order to ensure key performance indicators such as resource efficiency or service quality.

[0036] Elastic scaling: For a microservice framework, elastic scaling refers to adjusting the resource capacity quota reserved for microservices based on capacity planning results. Specifically, it includes two methods: 1) horizontal scaling, which changes the number of containers held by the microservice; 2) vertical scaling, which modifies the quota for each container. Because microservice containers are typically stateless, horizontal scaling is used to implement elastic scaling in the embodiments of this specification.

[0037] Workload: In the embodiments of this specification, workload is defined as the amount of project-level work that a microservice needs to process, quantified as RPS (requests per second), that is, the number of project-level requests per second of the microservice is used to measure the workload.

[0038] Resource usage and resource utilization: Because microservices are typically stateless, CPU resources become a critical bottleneck. CPU utilization is used to measure resource utilization. Accordingly, resource usage is defined as the amount of resources consumed by a microservice or a container per unit of time. CPU utilization is defined as CPU usage, i.e., CPU utilization = (CPU usage) / (CPU capacity) × 100%; where CPU usage and CPU capacity are measured in cores.

[0039] Resource efficiency: Resource efficiency can be understood as the average amount of resources used to complete the same workload; under the same workload, the smaller the resource usage, the higher the resource efficiency.

[0040] Service Quality Performance: Service Quality (QoS) refers to the performance of a microservice in completing its workload, as perceived by the user or defined by the user, such as tail latency and end-to-end latency.

[0041] Performance patterns: In a microservices scenario, for any elastic scaling framework, it focuses on keeping one or more performance metrics stable. Performance patterns describe or quantify the relationship between the performance metrics that the framework focuses on and the workload.

[0042] CPU usage patterns: A special case of performance patterns. In scenarios where stable CPU utilization is a concern, it is necessary to construct a relationship pattern between CPU usage and workload. By estimating the amount of CPU usage required to handle the target workload, the elastic scaling framework can be guided to perform reasonable capacity planning.

[0043] Microservice upgrade: At the project level, the core of a microservice is its business logic, that is, the behavior of the microservice when it is called to implement service request processing. With the support of continuous integration and continuous delivery (CI / CD) tools and container technology, changing or optimizing the behavior of microservices to adapt to application requirements has become a routine and frequent practice in microservices. This change behavior is called microservice version upgrade.

[0044] Pattern drift: Essentially, a performance pattern can be understood as the joint distribution of performance metrics and workload. However, over time, this joint distribution maintained by the performance pattern may change, leading to significant errors in usage prediction based on outdated performance patterns. In other words, previously constructed performance patterns may no longer be suitable for the current actual performance patterns, thus failing to solve capacity planning problems; this phenomenon is called pattern drift. In particular, microservice updates often cause unpredictable performance pattern drift.

[0045] Considering the dynamic nature of workloads, implementing an auto-scaling framework that can effectively alleviate resource oversupply and undersupply is crucial for microservices to maintain resource efficiency and guarantee Quality of Service (QoS). Since most microservices are lightweight and stateless, CPU (Central Processing Unit) resources often become critical resources for microservices in real-world scenarios. Numerous previous studies have shown that maintaining CPU utilization within a reasonable range can effectively guarantee the resource efficiency and QoS of microservices. Therefore, the goal of an auto-scaling framework is to ensure the stability of CPU utilization.

[0046] Practical experience has revealed a significant impact of version upgrades: the effect of version upgrades on capacity and performance patterns is highly uncertain. Some upgrades can cause a substantial increase or decrease in CPU usage for microservices, sometimes exceeding 100%, resulting in a significant change in CPU usage patterns.

[0047] To mitigate the impact of performance pattern drift, an empirical approach can be used to relearn performance patterns at predefined time intervals; alternatively, developers can manually trigger performance pattern updates via microservice upgrade notifications. However, simply relearning performance patterns periodically is insufficient to capture changes in a timely manner, and triggering adaptive pattern learning via developer upgrade notifications places a significant maintenance burden on auto-scaling frameworks in large-scale data centers. Therefore, accurately and automatically detecting microservice pattern drift in production environments is crucial for achieving precise capacity tuning.

[0048] This specification provides a data processing method, and also relates to a data processing system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0049] Referring to Figure 1, Figure 1 shows a schematic diagram of a data processing method provided according to an embodiment of this specification, which is applied to server 102.

[0050] Specifically, server 102 includes a detection unit 104 and a scheduling unit 106. The detection unit 104 acquires a work performance sequence under the target performance mode, wherein the work performance sequence includes work performance pairs consisting of workload and resource utilization corresponding to the workload. According to a preset time window and a preset time interval, the work performance sequence is sequentially divided to determine multiple sub-work performance sequences. From the multiple sub-work performance sequences, a target sequence and the previous reference sequence adjacent to the target sequence are determined. Based on the target sequence and the reference sequence, the target difference distance between the target sequence and the reference sequence is obtained. Based on the target difference distance, the target performance mode is detected to obtain a performance detection result.

[0051] If the detection unit 104 obtains the performance detection result, it can further determine that the target performance mode has shifted if the performance detection result indicates that the performance mode has changed. Then, it can determine the current performance mode that meets the requirements of the current automatic resilience framework. If the current performance mode is sent to the scheduling unit 106, the automatic resilience framework in the scheduling unit 106 determines the container adjustment strategy for the container based on the current performance mode and the predicted workload. The scheduler in the scheduling unit 106 schedules the container 108 according to the container adjustment strategy, that is, expands or shrinks the container 108.

[0052] Server 102 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that a server can be implemented as a distributed server cluster composed of multiple servers, or as a single server. A server can also be a server in a distributed system, or a server integrated with blockchain. Furthermore, a server can be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0053] The data processing method provided in this specification involves acquiring a working performance sequence under a target performance mode, dividing the working performance sequence into multiple sub-working performance sequences according to a preset time window and a preset time interval. Using the preset time window as the statistical granularity, a target sequence and the previous reference sequence adjacent to the target sequence are determined from the multiple sub-working performance sequences. By calculating the target difference distance between these two, the difference in performance modes between the target sequence and the reference sequence is obtained, thereby realizing the detection of the target performance mode and obtaining a detection result of whether the target performance mode has changed. Based on the quantified target difference distance, automatic identification and detection of performance mode drift are achieved, and in the case of automatic detection, the timeliness and accuracy of detection can be improved.

[0054] See Figure 2, which shows a flowchart of a data processing method provided in one embodiment of this specification, specifically including the following steps.

[0055] Step 202: Obtain the working performance sequence under the target performance mode, wherein the working performance sequence includes working performance pairs consisting of the workload at each performance acquisition time point and the resource utilization rate corresponding to the workload.

[0056] The target performance mode can be understood as the performance mode expected without mode drift, or the previously determined current performance mode. The current performance mode can be understood as the performance mode that accurately represents the current automatic elasticity framework. The automatic elasticity framework can be understood as an elastic scaling framework, a software architecture design pattern that can automatically adjust resource allocation according to changes in workload or system conditions to ensure stable system performance, rapid response, and cost-effectiveness. In the embodiments of this specification, the prediction and planning of future resource requirements and resource capacity can be realized based on the predicted workload and performance mode, and the elastic scaling of microservice resource capacity can be achieved. The work performance sequence can be understood as a data set arranged in chronological order according to the performance collection time point, recording the performance of various aspects of the system when handling different workloads under the target performance mode. The performance collection time point can be understood as the time point at which work performance data is collected under the target performance state. Work performance data includes, but is not limited to, the aforementioned workload, resource utilization, memory utilization, etc.

[0057] Resource utilization can be understood as the proportion of various resources used and consumed to complete the above workload, such as CPU utilization, memory utilization, network bandwidth utilization, etc. It is an important performance indicator for measuring system efficiency and performance.

[0058] Specifically, under the target performance mode, a series of work performance data for a certain microservice are collected. The work performance data exists in the form of "work performance pairs". Each work performance pair contains the workload situation under a specific workload and the resource utilization information at the corresponding time, reflecting the specific situation of resource usage under a certain workload. By obtaining work performance pairs over a period of time, a work performance sequence composed of work performance pairs is obtained. The work performance pairs in this work performance sequence are arranged according to the time of performance collection.

[0059] In one or more embodiments of this specification, workload and resource utilization data at different time points are collected, work performance pairs are constructed and sorted to obtain a complete work performance sequence. The specific implementation method is as follows:

[0060] The process of obtaining the working performance sequence under the target performance mode includes:

[0061] Determine the time points for collecting performance data under the target performance mode within the preset time period;

[0062] Determine the container corresponding to each performance acquisition time point, as well as the initial workload and initial resource utilization of the container;

[0063] Based on the initial workload of the container and the initial resource utilization rate, determine the workload and the corresponding resource utilization rate at each performance acquisition time point;

[0064] Work performance pairs are formed based on the workload and the resource utilization rate corresponding to the workload, and the work performance sequence is determined based on the work performance pairs.

[0065] The preset time period can be understood as a pre-defined range of time during which the performance of the system or automatic elastic framework is continuously monitored, thereby enabling precise adjustments to the container. A container is a standard software unit that encapsulates an application and its dependencies in containerization technology (such as Docker).

[0066] Initial workload can be understood as the amount of tasks or project pressure undertaken by the container at each performance sampling point, which can be computing tasks, data processing tasks, etc.; initial resource utilization can be understood as the proportion of computing resources (such as CPU, memory, etc.) occupied or consumed by the container to the total resources.

[0067] Specifically, since each microservice consists of multiple container replicas, the performance sequence of the microservice is determined by statistically analyzing the performance data of the corresponding containers.

[0068] For example, determine the performance data collection time point T1, and determine that the container corresponding to time T1 includes container A and container B, as well as the initial workload and initial resource utilization of container A and container B at time T1. Obtain the workload at time T1 from the initial workload of container A and container B, and obtain the resource utilization at time T1 from the initial resource utilization of container A and container B. Then, form a performance pair corresponding to time T1 based on the workload and resource utilization at time T1. In the case of multiple performance data collection time points, obtain the performance pairs corresponding to each performance data collection time point using the above method. Then, sort the performance pairs according to the chronological order of the performance data collection time points to obtain a performance sequence composed of the performance pairs.

[0069] In practical applications, when collecting performance data, it can be collected in chronological order. Then, the performance pairs can be directly arranged according to the time order of the performance collection time points to obtain a performance sequence composed of each performance pair.

[0070] The data processing method provided in the embodiments of this specification can change the workload of a container as the tasks it performs change, which in turn causes changes in resource utilization. Therefore, the workload and corresponding resource utilization at each point in time can be recorded and analyzed to obtain the performance trend of the system at different points in time, thereby enabling performance optimization, capacity planning, or troubleshooting.

[0071] In one or more embodiments of this specification, performance data corresponding to each performance collection time point is obtained by determining the average workload, average resource utilization, and the number of containers at each performance collection time point. The specific implementation method is as follows:

[0072] The step of determining the workload and corresponding resource utilization at each performance acquisition time point based on the initial workload of the container and the initial resource utilization includes:

[0073] Based on the number of containers, the initial workload of the containers, and the initial resource utilization rate, determine the average workload and the average resource utilization rate corresponding to each performance acquisition time point;

[0074] Based on the number of containers, the average workload, and the average resource utilization, the workload and the corresponding resource utilization at each performance acquisition time point are determined.

[0075] Here, the number of containers can be understood as the number of container instances running at the performance collection time point; the average workload can be understood as the average of the total workload of the corresponding container executing tasks at the performance collection time point; and the average resource utilization can be understood as the average resource utilization of the corresponding container at the performance collection time point. In the embodiments of this specification, it is the average CPU usage, expressed using the average RPS.

[0076] Specifically, based on the number of containers N at the performance collection time point, statistical methods are used to obtain the average workload and average resource utilization of the N containers at that performance collection time point. The overall workload at the performance collection time point is determined based on the average workload and the number of containers N, and the overall resource utilization at the performance collection time point is determined based on the average resource utilization and the number of containers N. Combining these two data points, a workload-performance pair corresponding to the performance collection time point is generated. This workload-performance pair represents the overall workload and resource utilization of the microservice at the performance collection time point.

[0077] The data processing method provided in the embodiments of this specification calculates the average workload and average resource utilization of all containers at the performance collection time point, which helps to analyze and understand the overall load status of microservices at that performance collection time point, thereby providing data support for making more accurate resource scheduling decisions in the future.

[0078] Step 204: Divide the work performance sequence sequentially according to the preset time window and preset time interval to determine multiple sub-work performance sequences.

[0079] The preset time window can be understood as a pre-defined time range. For example, if the preset time window is one hour, the performance of the microservices within that hour will be analyzed as a whole. The preset time interval can be understood as dividing the entire preset time period according to a certain time interval. For example, when the preset time window is slid, the sliding time span is 10 minutes. By sliding the preset time window, the entire preset time period is divided, and by dividing the preset time period, the performance sequence is also divided.

[0080] Specifically, based on the pre-set time window size and time interval, the original continuous work performance sequence is divided into multiple sub-work performance sequences.

[0081] For example, if the preset time window is 1 hour and the time interval is 10 minutes, then the work performance data subsets within 1 hour will be extracted from the work performance sequence in units of 10 minutes, and each subset is a sub-work performance sequence.

[0082] This method allows for convenient analysis of microservice performance trends over different time periods, identification of abnormal fluctuations, or more refined performance evaluation and optimization for different time windows.

[0083] Step 206: Determine the target sequence and the previous reference sequence adjacent to the target sequence from the plurality of sub-working performance sequences, and determine the target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence.

[0084] The target sequence can be understood as the sub-work performance sequence currently being analyzed among multiple sub-work performance sequences; the reference sequence can be understood as the sub-work performance sequence compared with the target sequence, which is the previous sub-work performance sequence adjacent to the target sequence, without any limitation here; the target difference distance can be understood as the distance used to quantify the difference between the target sequence and the reference sequence, including but not limited to LSDD (Local Statistical Discrepancy Degree) distance, Euclidean distance, etc., which reflects the degree of change in the correlation between workload and resource utilization of the target sequence and the reference sequence under the target performance model.

[0085] Specifically, by determining the target sequence and the previous reference sequence adjacent to the target sequence, and by analyzing the working performance pairs in the sequence, the target difference distance between the target sequence and the reference sequence is determined, and the difference between the target sequence and the reference sequence is obtained based on the target difference distance.

[0086] In one or more embodiments of this specification, the correlation distribution result is determined by the working performance pairs in the sequence, and the first target difference distance between the two sequences is determined based on the target correlation distribution result corresponding to the target sequence and the reference correlation distribution result of the reference sequence. The specific implementation is as follows:

[0087] Determining the target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence includes:

[0088] Based on the performance pairs in the target sequence, determine the target association distribution result, and based on the performance pairs in the reference sequence, determine the reference association distribution result;

[0089] Based on the target association distribution results and the reference association distribution results, a first target difference distance is determined between the target sequence and the reference sequence.

[0090] Specifically, determining the target association distribution result based on the job performance pairs in the target sequence, and determining the reference association distribution result based on the job performance pairs in the reference sequence, includes:

[0091] Based on the work performance pairs in the target sequence, determine the target correlation distribution results between workload and resource utilization in the target sequence; and

[0092] Based on the work performance pairs in the reference sequence, determine the reference correlation distribution results between workload and resource utilization in the reference sequence.

[0093] The correlation distribution result can be understood as the correlation or distribution pattern between workload and resource utilization rate analyzed based on the work performance pairs in the work performance sequence, which is used to characterize the performance pattern. In the embodiments of this specification, the correlation distribution result is used to characterize the CPU usage pattern based on the resource utilization rate as the CPU utilization rate.

[0094] The target correlation distribution result can be understood as the correlation distribution result determined based on the workload performance in the target sequence; the reference correlation distribution result can be understood as the correlation distribution result determined based on the workload performance in the reference sequence. The first target difference distance can be understood as the numerical difference between the target correlation distribution result and the reference correlation distribution result calculated through specific mathematical methods or algorithms, reflecting the degree of change in the correlation between workload and resource utilization in the target sequence and the reference sequence under the target performance model.

[0095] Specifically, based on the target difference distance, it can be determined whether the target performance pattern has changed significantly. When it is necessary to determine the target difference distance, the target difference distance can be calculated and determined by the target association distribution result and the reference association distribution result.

[0096] In practical applications, the probability density function of the joint distribution of workload and resource utilization can be used to represent the correlation distribution result; that is, based on the work performance pairs in the target sequence, the probability density function corresponding to the target sequence is determined, and the probability density function corresponding to the target sequence is used to characterize the target correlation distribution result; based on the work performance pairs in the reference sequence, the probability density function corresponding to the reference sequence is determined, and the probability density function corresponding to the reference sequence is used to characterize the reference correlation distribution result.

[0097] In statistics and probability theory, for continuous random variables, the probability density function describes the probability distribution of the variable's values ​​within a certain interval.

[0098] Specifically, taking the calculation of the target correlation distribution result as an example, by statistically modeling the work performance pairs in the target sequence, their probability density functions are determined. This probability density function represents the joint distribution law between workload and resource utilization in the target sequence, that is, the target correlation distribution result.

[0099] By calculating and comparing the probability density functions (i.e., the correlation distribution results) corresponding to the target sequence and the reference sequence, the first target difference distance is obtained through mathematical calculation. Then, based on the first target difference distance, it is determined whether there is a significant change in the target performance pattern.

[0100] Taking the LSDD distance method to quantify the target difference distance as an example, when the performance pattern is statistically analyzed with a preset time window as the granularity, since a sub-work performance sequence corresponds to a preset time window, determining the LSDD distance of the correlation distribution results between sub-work performance sequences can be understood as determining the difference in performance patterns between preset time windows.

[0101] Based on this, by detecting whether the target performance mode has shifted between preset time windows, the target time window can be identified. The probability density function of the sub-work performance sequence corresponding to the target time window has a significant deviation from the probability density function of the sub-work performance sequence corresponding to the previous preset time window. That is, the target time window can be understood as the preset time window for determining the performance mode of the target.

[0102] In the case of detecting whether the target performance mode has shifted between preset time windows, a sliding window method can be used to detect the mode shift; as shown in Figure 3, Figure 3 shows a sliding window schematic diagram of a sliding window mode shift detection provided in an embodiment of this specification.

[0103] In the embodiments of this specification, the granularity of the statistical performance model is the time window. That is, by sliding a preset time window, the difference in CPU usage patterns between different time windows is detected and quantified, that is, the change in the joint distribution of CPU usage and workload RPS. The sliding window method is used to effectively measure the probability density difference between adjacent preset time windows.

[0104] In practical applications, for any given microservice τ, the performance sequence within a preset time period is obtained. The preset time window size is represented by W, the preset time interval is represented by S, the workload is represented by X = {X1, X2, ...}, and the CPU utilization is represented by Y = {Y1, Y2, ...}.

[0105] Specifically, Where t is any time, and n t Let τ be the number of containers in the microservice at time t. and The n at time t t Average RPS and average CPU usage per container.

[0106] As shown in Figure 3, from (X1,Y1)-(X W ,Y W Within a preset time period, the sub-work performance sequence of the first preset time window 0, when sliding the preset time window according to a preset time interval, slides from preset time window 0 to preset time window 1. The starting sequence in the sub-work performance sequence slides from (X1,Y1) to (X... S,Y S ), that is, the next sub-work performance sequence is (X S Y S )-(X S+W Y S+W ), and so on, until the sub-work performance sequence corresponding to the i-th preset time window i is (X i·S Y i·S )-(X i·S+W Y i·S+W ).

[0107] Z t =(X t ,Y t Z is defined as the data pair of two metrics, workload and CPU utilization, at time t, i.e., the performance pair in the above embodiment; let Z be... i Z represents the sequence of metric records within each preset time window i (i.e., the sub-work performance sequence in the above embodiment), and describes the CPU usage pattern of microservice τ as Z. i The PDF (probability density function) on the surface is denoted as P. i (Z); By obtaining the probability density function (P0, P1…P) corresponding to each sub-work performance sequence. i ), calculate the difference between each probability density function, where the difference between probability density functions is expressed using D(P i-1 P i ), where D represents the LSDD distance between probability density functions.

[0108] For example, for any two preset time windows u and v, the LSDD distance is defined as follows: D 2 (P u P v )=∫(P u (Z)-P v (Z)) 2 dZ is used to measure the LSDD distance between preset time windows u and v.

[0109] Given the significant uncertainty surrounding the impact of version upgrades on performance patterns, to accurately detect pattern drift, the performance pattern representation of microservices should remain robust during version upgrades under varying workloads, unaffected by the upgrade process. Therefore, the preset time window size W should encompass both workload peaks and troughs, ensuring that each preset time window includes the same version of the microservice, and the preset time interval S for window sliding should span the entire version upgrade process. Since microservice workloads often exhibit daily periodicity—for example, approximately 95% of a microservice version upgrade might take around 8 hours—W can be set to 48 hours and S to 8 hours. Of course, in practice, W and S can be set according to specific circumstances and are not limited here.

[0110] Of course, in practical applications, the sub-performance sequence corresponding to the preset time window 0 can be used as a reference sequence, because the performance mode corresponding to the preset time window 0 is the target performance mode. Given the probability density function P0 corresponding to the sub-performance sequence within the preset time window 0, the probability density function P0 of each sub-performance sequence corresponding to other preset time windows can be calculated. i Thus, by calculating the probability density function P i The difference D(P0, P) between the probability density function P0 and the probability density function P0 i Determine the probability density function P. i The difference between the performance mode corresponding to the preset time window i and the target performance mode corresponding to the preset time window 0 is used to detect whether mode drift has occurred in the preset time window i relative to the target performance mode.

[0111] However, since the target performance mode may experience more than one mode drift within a preset time period, and the degree of change between these mode drifts is also different (e.g., the first mode drift changes the target performance mode to performance mode 1, and the second mode drift changes the performance mode 1 to performance mode 2), using the sub-working performance sequence corresponding to the preset time window 0 as a reference sequence and comparing the target sequence with the sub-working performance sequence corresponding to the preset time window 0 to detect whether mode drift has occurred may result in inaccurate detection. Therefore, in the embodiments of this specification, the previous sub-working performance sequence adjacent to the target sequence is used as the reference sequence; however, the selection of the reference sequence can be determined according to the actual situation and is not limited here.

[0112] The data processing method provided in the embodiments of this specification can accurately quantify the distribution differences of microservice performance in different time periods by calculating the probability density functions corresponding to the target sequence and the reference sequence, thereby accurately detecting whether there is a pattern drift phenomenon in the target performance pattern.

[0113] In one or more embodiments of this specification, the target difference distance can be obtained directly using the target metric model. The specific implementation is as follows:

[0114] Based on the target sequence and the reference sequence, a second target difference distance between the target sequence and the reference sequence is obtained using a target metric model.

[0115] The target metric model can be understood as a mathematical model used to measure and compare the differences between two sub-work performance sequences; the second target difference distance can be understood as a numerical index calculated by the target metric model to quantify the degree of difference in performance patterns between the target sequence and the reference sequence.

[0116] Specifically, the target sequence and reference sequence are determined, and a pre-set or selected target metric model is used to conduct an in-depth analysis and comparison of the two sub-work performance sequences; the difference distance between the target sequence and the reference sequence is calculated through the target metric model.

[0117] For example, when the target metric is a Gaussian Radial Basis Function (RBF) model, the sub-work performance sequence Z corresponding to the preset time window u is... u and the sub-work performance sequence Z corresponding to the preset time window V v Z is directly estimated using the Gaussian radial basis function model. u and Z v The LSDD distance between them is

[0118] The data processing method provided in the embodiments of this specification can accurately and efficiently calculate the second target difference distance between the target sequence and the reference sequence through the target metric model, thereby enabling timely determination of whether there is a significant performance change or performance drift of the target sequence compared to the reference sequence based on the second target difference distance.

[0119] Step 208: Detect the target performance mode based on the target difference distance to obtain the performance detection result.

[0120] The performance test results include two types: those indicating a change in performance mode and those indicating no change in performance mode.

[0121] Specifically, after obtaining the target difference distance, the target performance mode is detected based on the target difference distance. If the target difference distance is large and exceeds a certain threshold, it indicates that the target performance mode has shifted, and the performance detection result is that the performance mode has changed. If the target difference distance is small and less than a certain threshold, it indicates that the target performance mode is still relatively stable, and the performance detection result is that the performance mode has not changed.

[0122] In one or more embodiments of this specification, in order to accurately obtain the difference distance threshold compared to the target difference distance, and thus determine the accurate performance detection result, a preset confidence level and a preset number of tests are determined. The specific implementation method is as follows:

[0123] Before detecting the target performance pattern based on the target difference distance and obtaining the performance detection result, the method further includes:

[0124] Determine the preset confidence level and the preset number of tests;

[0125] The difference distance threshold is determined based on the target sequence, the reference sequence, the preset confidence level, and the preset number of tests.

[0126] In statistics, confidence level usually refers to the probability that the conclusion is correct in hypothesis testing. For example, a 95% confidence level means that in repeated experiments, there is at least a 95% probability that the conclusion is correct. In the field of performance analysis, confidence level can be used to determine the reliability of the difference distance threshold.

[0127] The number of tests can be understood as the number of times the target sequence and the reference sequence are rearranged or sampled for comparison. A stable difference distance distribution is calculated using statistical methods, and the difference threshold at a given confidence level is determined.

[0128] The difference distance threshold can be understood as a critical value calculated based on the comparison results between the target sequence and the reference sequence, combined with the confidence level and the number of permutation tests. If the difference distance between the target sequence and the reference sequence exceeds this threshold, it can be considered that the target performance pattern has changed significantly and pattern drift has occurred.

[0129] Specifically, the confidence level and the number of tests can be preset according to the actual situation. For example, the confidence level can be preset to 95% and the number of tests can be preset to 100. The target sequence and the reference sequence can be compared and analyzed to establish a difference distance threshold. When the actual calculated target difference distance exceeds the difference distance threshold, it can be considered that the target performance mode has changed or drifted.

[0130] Following the previous example, this specification proposes an online pattern drift detection method based on the LSDD metric. For each microservice, let i d 'Indicates the preset time window in which the most recently detected performance mode drift occurred. Drift detection will be performed once after each preset time window i ends.

[0131] This detection process can be viewed as a hypothesis testing process, with a set confidence level of 1-μ, where μ represents the false positive rate (i.e., the probability of false alarms). The basic form of hypothesis testing is as follows: H0: H0 represents the null hypothesis, which states that the microservice will perform well within a preset time window i. d The performance mode and the preset time window i remain unchanged in this case.

[0132] If the null hypothesis H0 is rejected, it indicates that a performance mode drift occurred within the preset time window i; if H0 is accepted, it indicates that no mode drift occurred within the preset time window i. Specifically, H0 is accepted or rejected according to the following criteria: That is, if (Preset time window i) d The LSDD distance between the sub-work performance sequence corresponding to the preset time window i and the sub-work performance sequence is greater than the difference distance threshold T. μ If H0 is not found, then H0 is rejected, indicating that mode drift has occurred; otherwise, H0 is accepted in other cases, indicating that mode drift has not occurred.

[0133] At each detection, the difference distance threshold T is determined by performing m permutation tests. μ Algorithm 1 details how to merge reference data using random permutations. and test data Z i ,calculate The distribution of the variance was analyzed, and the difference distance threshold T was determined based on the false positive rate μ and the number of permutation tests m (i.e., the preset number of tests mentioned above). μ For example, let's set μ to 0.05 and m to 100. The main steps of Algorithm 1 are as follows:

[0134] Input: Reference data and test data and Z i False positive rate μ, number of tests m;

[0135] Output: Hypothesis testing Difference distance threshold T μ .

[0136] Will and Z i Merge into a new dataset Zall ;

[0137] Initialize the threshold candidate value set T cad An empty list;

[0138] For m iterations, Z is randomly selected each time. all Divide the dataset into two equal datasets Z0' and Z1', and calculate the difference between them. The value is added to the threshold candidate value set T. cad middle;

[0139] After the loop ends, find T. cad The (1-μ)×100 percentile is used as the difference distance threshold T for hypothesis testing. μ .

[0140] The data processing method provided in the embodiments of this specification can control the probability of false alarms by setting the false positive rate μ, thereby reducing the impact of meaningless alarms (i.e., false positives) on operation and maintenance decisions and making performance test results more reliable; the difference distance threshold T is dynamically determined by using a permutation test method. μ This means that the difference distance threshold is not fixed, but dynamically adjusted according to the actual data distribution and the number of tests, thus enhancing the adaptability and effectiveness of the data processing method.

[0141] In one or more embodiments of this specification, the performance detection result is determined to be either a change in performance mode or no change in performance mode by comparing the target difference distance and a difference distance threshold. The specific implementation is as follows:

[0142] The step of detecting the target performance pattern based on the target difference distance to obtain performance detection results includes:

[0143] Based on the target difference distance and the difference distance threshold, the target performance pattern is detected to obtain the performance detection result.

[0144] Specifically, the step of detecting the target performance pattern based on the target difference distance and the difference distance threshold to obtain performance detection results includes:

[0145] If the target difference distance is determined to be greater than the difference distance threshold, the performance detection result of the target performance mode is determined to be a change in performance mode;

[0146] If the target difference distance is less than or equal to the difference distance threshold, the performance detection result of the target performance mode is determined to be that the performance mode has not changed.

[0147] Specifically, when the calculated target difference distance is greater than the difference distance threshold, it can be determined that the performance mode between the target sequence and the reference sequence has changed, and the performance detection result is determined to be "performance mode changed". Then, subsequent steps are carried out based on the performance detection result, that is, the current performance mode is determined.

[0148] If the target difference distance is less than or equal to the difference distance threshold, then the target performance mode can be considered to be maintained within the normal range. In this case, the detection result is determined as "performance mode has not changed", indicating that the performance of the target performance mode is in line with expectations and no additional attention or intervention is required.

[0149] The data processing method provided in the embodiments of this specification detects the target performance pattern by comparing the target difference distance with the difference distance threshold, effectively monitors changes in microservice performance, and thus promptly discovers and handles potential performance problems, ensuring the stability and efficiency of microservice operation.

[0150] In one or more embodiments of this specification, after determining that the performance test result indicates a change in performance mode, it is necessary to determine the actual current performance mode in order to achieve accurate container planning based on the current performance mode. The specific implementation method is as follows:

[0151] After determining that the performance detection result of the target performance mode is a change in the performance mode, the method further includes:

[0152] Obtain the predicted sub-operating performance sequence following the target sequence in the operating performance sequence;

[0153] The current performance mode is determined based on the performance pairs in the predicted sub-performance sequence.

[0154] The predicted sub-work performance sequence can be understood as the sub-work performance sequence corresponding to the predicted time window after the detected mode drift.

[0155] Specifically, if the performance detection result of the target performance mode is determined to be a change in performance mode, it indicates that the microservice has experienced performance mode drift within the preset time window corresponding to the target sequence. Therefore, the sub-work performance sequence after the target sequence in the work performance sequence is determined as the predicted sub-work performance sequence. The current performance mode after the performance mode drift is determined by using the work performance pairs in the predicted sub-work performance sequence.

[0156] In practical applications, based on the LSDD metric detection algorithm, if the LSDD distance between the CPU usage pattern and the target CPU usage pattern (i.e., the target performance pattern in the above embodiment) within a certain time window is greater than the difference distance threshold, the pattern drift time window can be located. Based on the working performance sequence arranged in chronological order, data (i.e., predicted sub-working performance sequence) within the time window range that conforms to the current CPU usage pattern (i.e., the current performance pattern) can be determined. Based on the located predicted sub-working performance sequence, the current CPU usage pattern can be accurately quantified.

[0157] The data processing method provided in the embodiments of this specification obtains the performance sequence of the predicted sub-work by locating the time window in which the mode drift occurs, thereby accurately and efficiently determining the current performance mode based on the performance sequence of the predicted sub-work within the time window range that conforms to the current performance mode.

[0158] In one or more embodiments of this specification, to reduce errors, the target performance mode is considered to have shifted only after a preset number of consecutive performance tests show a change in performance mode, thus determining the current performance mode. The specific implementation is as follows:

[0159] After determining the target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence, the method further includes:

[0160] The target sequence is labeled, and it is determined whether there are any unlabeled sub-operating performance sequences among the plurality of sub-operating performance sequences.

[0161] If so, then continue with the steps of determining the target sequence and the previous reference sequence adjacent to the target sequence from the plurality of sub-working performance sequences, and determining the target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence.

[0162] Specifically, after determining the target difference distance between the target sequence and the reference sequence, the target sequence is marked, and it is determined whether there are any unmarked sub-working performance sequences among the multiple sub-working performance sequences, that is, whether there are any sub-working performance sequences among the multiple sub-working performance sequences that have not been used as the target sequence.

[0163] If they exist, the target sequence and the previous reference sequence adjacent to the target sequence are determined from the unlabeled sub-operating performance sequences. The target difference distance between the target sequence and the reference sequence is then determined based on the target sequence and the reference sequence. This process continues until all sub-operating performance sequences in the operating performance sequence have been labeled. Based on this, the target difference distance between each pair of adjacent sub-operating performance sequences in the operating performance sequence is obtained.

[0164] The data processing method provided in the embodiments of this specification, by labeling target sequences and performing cyclic operations, can determine the target difference distance between adjacent sub-operating performance sequences in the operating performance sequence, thereby accurately detecting mode drift in the operating performance sequence.

[0165] In one or more embodiments of this specification, to reduce false positives, the performance test result for the target performance mode is determined to have changed only when a preset number of consecutive performance test results indicate a change in performance mode. The specific implementation is as follows:

[0166] The step of detecting the target performance pattern based on the target difference distance to obtain performance detection results includes:

[0167] If it is determined that there are no unlabeled sub-work performance sequences among the plurality of sub-work performance sequences, the target performance pattern is detected according to the target difference distance to obtain each performance detection result, wherein each performance detection result includes performance pattern change or performance pattern no change;

[0168] If, among the performance test results indicating a change in the performance mode, there exists a preset number of consecutive performance test results, then the performance test results for the target performance mode are determined to indicate a change in the performance mode.

[0169] The preset quantity can be understood as a pre-set quantity. Based on this preset quantity, it can be determined how many consecutive times the performance mode changes before the performance detection result of the target performance mode is determined to be a performance mode change.

[0170] Specifically, when all sub-work performance sequences in the work performance sequence have been labeled, the target difference distance between each sub-work performance sequence is obtained, and then the performance detection results are obtained by comparing each target difference distance with the difference distance threshold.

[0171] Taking a preset quantity of 5 as an example, the distribution of performance test results indicating a change in performance mode is statistically analyzed. Only when there are 5 or more consecutive performance test results indicating a change in performance mode is the performance test result of the target performance mode determined to be a change in performance mode. That is, if performance mode drift occurs in 5 consecutive sub-work performance sequences, it can be confirmed that the performance mode of the current microservice or system has indeed changed significantly, and the current performance mode status can be determined accordingly.

[0172] The data processing method provided in the embodiments of this specification determines that the performance test result of the target performance mode has changed only when there are a preset number of consecutive performance test results indicating a change in performance mode. This reduces the possibility of misjudgment and improves the accuracy of mode drift detection.

[0173] In one or more embodiments of this specification, after determining that the performance detection result indicates a change in performance mode, the current performance mode is determined based on the predicted sub-work performance sequence after the performance mode change. The specific implementation is as follows:

[0174] After determining that the performance detection result of the target performance mode is a change in the performance mode, the method further includes:

[0175] A target performance test result is determined from the preset number of consecutive performance test results, wherein the target performance test result is any one of the preset number of consecutive performance test results;

[0176] Determine the target sequence corresponding to the target performance detection result, and determine the predicted sub-working performance sequence following the target sequence in the working performance sequence;

[0177] The current performance mode is determined based on the performance pairs in the predicted sub-performance sequence.

[0178] Specifically, taking a preset quantity of M as an example, if the performance mode changes after M consecutive performance tests, the target sequence corresponding to any one of the M performance tests that shows a change in performance mode is obtained, and the sub-working performance sequence following the target sequence is used as the predicted sub-working performance sequence. Based on the working performance pairs in the predicted sub-working performance sequence, the current performance mode is determined.

[0179] In practical applications, Algorithm 2 describes a continuous drift detection procedure. Its main purpose is to detect potential changes in microservice performance patterns in real time, and when a drift of a certain number of consecutive prediction time windows (denoted as θ) is detected, to determine the exact prediction time window i in which the drift occurred. d The following are the main steps of Algorithm 2:

[0180] Initialize the temporary drift counter to 0, and create a set I to store candidate drift window indices. cad ;

[0181] In an infinite loop, each detection cycle (i.e., time window i) is continuously tested;

[0182] Get the reference dataset of the current window i and test dataset Z i And calculate the threshold T using the algorithm corresponding to Algorithm 1. μ .

[0183] If the calculation yields If the window is considered to be drifting, the value of the temporary drift counter is increased, and the current window i is added to the candidate set of drift window indices I. cad middle;

[0184] if If it is not greater than the threshold, then clear I. cad And reset the temporary drift counter to 0;

[0185] When θ consecutive window drifts are detected (i.e., counter ≥ θ), then from I... cad Select the smaller window index i from the set d This serves as the exact window for the first detection of drift, and the result is returned; the loop continues, performing detection continuously.

[0186] Wherein, time window i d The corresponding sub-work performance sequence can be understood as the target sequence corresponding to the target detection result when the first performance detection result in M ​​times is the target detection result.

[0187] The data processing method provided in the embodiments of this specification can collect the latest performance data after determining the time window in which the drift occurs, accurately capture the current performance pattern, and provide a basis for subsequent microservice capacity adjustment.

[0188] In one or more embodiments of this specification,

[0189] The step of determining the current performance mode based on the performance pairs in the predicted sub-performance sequence includes:

[0190] Based on the job performance pairs in the predicted sub-job performance sequence, a statistical learning model is used to obtain the split points of the job performance pairs and the target parameters corresponding to each segment, wherein each segment is determined by dividing the job performance pairs in the predicted sub-job performance sequence through the split points.

[0191] The current performance mode is determined based on the target parameters corresponding to each segment.

[0192] Specifically, by applying statistical learning models (such as decision trees, K-means clustering, and isolated forests), key split points are identified in the performance pair dataset of the predicted sub-work performance sequence. These split points divide the predicted sequence into several segments with different performance characteristics. For each segment, key parameters or feature values ​​reflecting the performance pattern of that segment are calculated based on the performance pair data within it, such as the slope and intercept of the performance curve. Based on the target parameters corresponding to each segment, the performance pattern of the current microservice can be comprehensively analyzed and described, which helps to identify the behavioral characteristics of the microservice in different workload or resource utilization ranges.

[0193] In practical applications, there is a significant correlation between CPU usage and workload intensity in microservices. The two are positively correlated, meaning that as workload intensity increases, CPU usage usually increases accordingly. In order to balance versatility, accuracy, and efficiency, the embodiments in this specification use a continuous piecewise regression function to fit this nonlinear relationship. Furthermore, by referring to Taylor's theorem, this complex relationship can be approximated by a set of polynomials.

[0194] The workload-CPU usage relationship of each microservice is represented as a set of local linear models customized for different workload ranges. Using local models can maintain the stability and accuracy of the model.

[0195] For a specific microservice τ, at time t, assume the total workload is Y. t CPU usage is X t Then, a set F consisting of L local linear models is defined to simulate CPU usage patterns. This set is represented by L+1 split points (x0, x1, ..., x...). L Divide the CPU usage range into L segments, each corresponding to a different linear function:

[0196] Where, sgn(·) = 1 if true, otherwise 0, and furthermore (α l ,β l Then ) represents the parameters of the local linear function in the Lth segment; specifically, sgn() is a symbolic function that returns 1 if the condition within the parentheses is true, otherwise it returns 0; (α l ,β l ) represents the slope and intercept parameters of the local linear function in segment L.

[0197] For example, a generalized random forest (GRF) can be used as a statistical learning model. It is an ensemble method consisting of multiple regression trees. GRF can estimate the aforementioned set of piecewise linear functions F through ensemble methods, and automatically determine the optimal segmentation strategy, including the split points (x0, x1, ..., x), during the regression tree construction process.L ) and its corresponding linear model parameters {(α) l ,β l The optimal parameters (1≤l≤L) were obtained during the analysis of the leaf nodes inside the regression tree.

[0198] The data processing method provided in the embodiments of this specification customizes a set of local linear models for different workload ranges of each microservice, rather than using a single comprehensive model, thereby improving the model's relevance and accuracy. By using continuous piecewise regression functions and local linear models, it can flexibly handle the complex nonlinear relationship between workload and CPU usage, and adapt to performance pattern changes brought about by microservice version upgrades, thereby improving the model's stability and generalization ability under frequent update conditions.

[0199] In one or more embodiments of this specification, a container adjustment plan is determined based on the current performance mode, thereby reasonably expanding or shrinking the container. The specific implementation is as follows:

[0200] After determining the current performance mode based on the performance pairs in the predicted sub-performance sequence, the process further includes:

[0201] Based on the predicted workload and the current performance pattern, determine the container tuning strategy for the container;

[0202] The container may be expanded or reduced according to the container adjustment strategy.

[0203] Specifically, statistical analysis or machine learning methods can be used to predict future resource demands, that is, to predict the workload that microservices or containers may face. Given the current performance pattern, a scaling-up or scaling-down strategy for containers can be formulated by combining the predicted workload with the current performance pattern. For example, if a significant increase in workload is predicted in the future, and the current performance pattern shows that existing containers cannot effectively handle this situation, it might be recommended to increase the number of containers to distribute the load and improve processing capacity. Conversely, if a decrease in workload is predicted, and there is currently resource redundancy, it might be recommended to reduce the number of containers to save resources and costs.

[0204] The data processing method provided in the embodiments of this specification, based on the above-mentioned container adjustment strategy, performs expansion or reduction operations through container orchestration tools (such as Kubernetes) to dynamically adjust the number of containers, thereby achieving precise elastic scaling. This ensures that while meeting performance requirements, resources are utilized to the maximum extent, thereby improving the overall performance and stability of microservices.

[0205] The data processing method provided in the embodiments of this specification focuses on the performance pattern drift problem caused by large-scale microservice upgrades in production scenarios, and quantifies the impact of this performance pattern drift on CPU usage and capacity planning. It realizes fine-grained and lightweight detection of microservice upgrades and performance drift, and automatically identifies and locates the drift moment online in real time, which greatly reduces the operation and maintenance costs and ensures that cloud services continuously capture the latest performance patterns to achieve more accurate capacity estimation. It realizes effective modeling of performance pattern drift, including sliding window representation of performance patterns and LSDD-based quantification of performance pattern drift.

[0206] Referring to Figure 4, Figure 4 shows a schematic diagram of a mode drift detection structure provided in one embodiment of this specification.

[0207] In practical applications, microservices undergo frequent upgrades to adapt to ever-changing project requirements, leading to changes in CPU usage patterns, commonly referred to as pattern drift. To ensure accurate capacity estimation based on the latest CPU usage patterns, this specification proposes an online detector for identifying CPU usage pattern drift.

[0208] First, a mathematical formulation representing the pattern drift phenomenon is established, and a sliding window method is introduced to quantify these changes. Second, a mechanism for detecting pattern drift is used to effectively monitor and promptly detect pattern changes caused by version upgrades. The performance pattern after the most recent drift moment is the latest performance pattern that the current resilient framework needs to accurately represent. A highly interpretable and accurate model is used to represent the latest performance pattern.

[0209] Specifically, the scheduler 406 in Figure 4 is responsible for scheduling resources within the data center 408 to meet workload requirements; the data center 408 contains multiple containers, which can be allocated by the scheduler 406 to different workloads; the container is the basic resource unit in the data center 408 and can be allocated by the scheduler 406 to workloads.

[0210] Workload: Requires the resources of data center 408 to run; Detector 402: A tool or system for detecting pattern drift, used to detect whether performance patterns have drifted; LSDD-based pattern drift detection metrics: Used to assess whether performance patterns have drifted; Sliding window drift: A method for detecting pattern drift, which effectively measures the distance difference between adjacent time windows by sliding a time window, thereby determining whether drift exists.

[0211] Automatic Resilience Framework 404: Automatically adjusts resource allocation in data center 408 based on changes in workload demand; Workload Forecaster: Predicts future workload demand for capacity planning; Capacity Planner: Plans data center capacity based on predicted workload demand and current performance.

[0212] Specifically, scheduler 406 first selects suitable containers from data center 408 based on workload requirements and allocates them to the workload. During workload operation, detector 402 continuously monitors performance patterns based on workload and resource utilization to check for drift. Specifically, it uses a sliding window drift detection method and uses an LSDD-based pattern drift detection metric to evaluate the degree of difference in CPU usage patterns (i.e., the target difference distance in the above embodiment). If the degree of difference is greater than a preset threshold (i.e., the difference distance threshold in the above embodiment), it is considered that pattern drift has been detected. Based on the detected drift, it locates data within the time window range that matches the latest CPU usage pattern, and uses this data to perform pattern modeling to obtain the current performance pattern. The capacity planner in the automatic resilience framework 404 determines the adjustment strategy for the corresponding containers based on the workload predicted by the workload predictor and the current performance pattern, and sends it to scheduler 406, so that scheduler 406 schedules containers in data center 408 according to the adjustment strategy.

[0213] The data processing method provided in the embodiments of this specification optimizes data center resource scheduling by accurately and automatically detecting mode drift, achieving precise container adjustment and helping to improve work efficiency.

[0214] Corresponding to the above method embodiments, this specification also provides a data processing system embodiment. Figure 5 shows a schematic diagram of the structure of a data processing system 500 provided in one embodiment of this specification. As shown in Figure 5, the system includes a detection unit 502 and a scheduling unit 504, wherein...

[0215] The detection unit 502 is used to apply the above-described data processing method to obtain performance detection results, and to determine the current performance mode when the performance detection result is the target detection result.

[0216] The scheduling unit 504 is used to determine a container adjustment strategy for the container based on the predicted workload and the current performance mode, and to expand or shrink the container according to the container adjustment strategy.

[0217] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0218] Figure 6 shows a structural block diagram of a computing device 600 according to one embodiment of this specification. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0219] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0220] In one embodiment of this specification, the aforementioned components of the computing device 600, as well as other components not shown in FIG. 6, may be interconnected, for example, via a bus. It should be understood that the block diagram of the computing device shown in FIG. 6 is merely illustrative and not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0221] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0222] The processor 620 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0223] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0224] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0225] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.

[0226] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0227] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0228] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0229] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0230] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0231] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0232] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Obtain the work performance sequence under the target performance mode, wherein the work performance sequence includes work performance pairs consisting of the workload at each performance acquisition time point and the resource utilization rate corresponding to the workload; The work performance sequence is sequentially divided according to a preset time window and a preset time interval to determine multiple sub-work performance sequences; From the plurality of sub-work performance sequences, a target sequence and the previous reference sequence adjacent to the target sequence are determined, and the target difference distance between the target sequence and the reference sequence is determined based on the target sequence and the reference sequence. Based on the target difference distance, the target performance pattern is detected to obtain the performance detection result.

2. The data processing method according to claim 1, wherein determining the target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence comprises: Based on the performance pairs in the target sequence, determine the target association distribution result, and based on the performance pairs in the reference sequence, determine the reference association distribution result; Based on the target association distribution results and the reference association distribution results, a first target difference distance is determined between the target sequence and the reference sequence.

3. The data processing method according to claim 2, wherein determining the target association distribution result based on the job performance pairs in the target sequence, and determining the reference association distribution result based on the job performance pairs in the reference sequence, comprises: Based on the work performance pairs in the target sequence, determine the target correlation distribution results between workload and resource utilization in the target sequence; as well as Based on the work performance pairs in the reference sequence, determine the reference correlation distribution results between workload and resource utilization in the reference sequence.

4. The data processing method according to claim 1, wherein determining the target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence comprises: Based on the target sequence and the reference sequence, a second target difference distance between the target sequence and the reference sequence is obtained using a target metric model.

5. The data processing method according to claim 1, 2, or 4, before detecting the target performance pattern based on the target difference distance and obtaining the performance detection result, further comprising: Determine the preset confidence level and the preset number of tests; The difference distance threshold is determined based on the target sequence, the reference sequence, the preset confidence level, and the preset number of tests.

6. The data processing method according to claim 5, wherein detecting the target performance pattern based on the target difference distance to obtain a performance detection result includes: Based on the target difference distance and the difference distance threshold, the target performance pattern is detected to obtain the performance detection result.

7. The data processing method according to claim 6, wherein detecting the target performance pattern based on the target difference distance and the difference distance threshold to obtain a performance detection result includes: If the target difference distance is determined to be greater than the difference distance threshold, the performance detection result of the target performance mode is determined to be a change in performance mode; If the target difference distance is less than or equal to the difference distance threshold, the performance detection result of the target performance mode is determined to be that the performance mode has not changed.

8. The data processing method according to claim 7, further comprising, after determining that the performance detection result of the target performance mode is a change in performance mode: Obtain the predicted sub-operating performance sequence following the target sequence in the operating performance sequence; The current performance mode is determined based on the performance pairs in the predicted sub-performance sequence.

9. The data processing method according to claim 1, wherein obtaining the working performance sequence under the target performance mode includes: Determine the time points for collecting performance data under the target performance mode within the preset time period; Determine the container corresponding to each performance acquisition time point, as well as the initial workload and initial resource utilization of the container; Based on the initial workload of the container and the initial resource utilization rate, determine the workload and the corresponding resource utilization rate at each performance acquisition time point; Work performance pairs are formed based on the workload and the resource utilization rate corresponding to the workload, and the work performance sequence is determined based on the work performance pairs.

10. The data processing method according to claim 9, wherein determining the workload and resource utilization corresponding to each performance acquisition time point based on the initial workload of the container and the initial resource utilization rate includes: Based on the number of containers in the sampled container, the initial workload of the container, and the initial resource utilization, determine the average workload and the average resource utilization corresponding to the average workload at each performance acquisition time point: Based on the number of containers, the average workload, and the average resource utilization, the workload and the corresponding resource utilization at each performance acquisition time point are determined.

11. The data processing method according to claim 1, further comprising, after determining the target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence: The target sequence is labeled, and it is determined whether there are any unlabeled sub-operating performance sequences among the plurality of sub-operating performance sequences. If so, then continue with the steps of determining the target sequence and the previous reference sequence adjacent to the target sequence from the plurality of sub-working performance sequences, and determining the target difference distance between the target sequence and the reference sequence based on the target sequence and the reference sequence.

12. The data processing method according to claim 11, wherein detecting the target performance pattern based on the target difference distance to obtain a performance detection result includes: If it is determined that there are no unlabeled sub-work performance sequences among the plurality of sub-work performance sequences, the target performance pattern is detected based on the target difference distance to obtain each performance detection result, wherein each performance detection result includes performance pattern change or performance pattern no change; If, among the performance test results indicating a change in the performance mode, there exists a preset number of consecutive performance test results, then the performance test results for the target performance mode are determined to indicate a change in the performance mode.

13. The data processing method according to claim 12, further comprising, after determining that the performance detection result of the target performance mode is a change in performance mode: A target performance test result is determined from the preset number of consecutive performance test results, wherein the target performance test result is any one of the preset number of consecutive performance test results; Determine the target sequence corresponding to the target performance detection result, and determine the predicted sub-working performance sequence following the target sequence in the working performance sequence; The current performance mode is determined based on the performance pairs in the predicted sub-performance sequence.

14. The data processing method according to claim 8 or 13, wherein determining the current performance mode based on the performance pairs in the predicted sub-job performance sequence includes: Based on the job performance pairs in the predicted sub-job performance sequence, a statistical learning model is used to obtain the split points of the job performance pairs and the target parameters corresponding to each segment, wherein each segment is determined by dividing the job performance pairs in the predicted sub-job performance sequence through the split points; The current performance mode is determined based on the target parameters corresponding to each segment.

15. The data processing method according to claim 8 or 13, after determining the current performance mode based on the performance pairs in the predicted sub-job performance sequence, further comprising: Based on the predicted workload and the current performance pattern, determine the container tuning strategy for the container; The container may be expanded or reduced according to the container adjustment strategy.

16. A data processing system, comprising a detection unit and a scheduling unit, wherein, The detection unit is used to apply the data processing method according to any one of claims 1-15 to obtain a performance detection result, and to determine the current performance mode when the performance detection result is the target detection result. The scheduling unit is configured to determine a container adjustment strategy for the container based on the predicted workload and the current performance mode, and to expand or shrink the container according to the container adjustment strategy.

17. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 15.

18. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 15.

Images