Method, apparatus, and computer readable storage medium for system optimization

Abstract

This application provides a system optimization method applied to an operating system. This method collects data from various applications to observe changes in the system characteristics of different business types throughout their lifecycles, enabling the perception of performance bottleneck types for each business type. By classifying and modeling system characteristics, it identifies business behaviors running on the OS, achieving non-intrusive perception of business types. Even without prior training, the method can identify application performance bottleneck types based on existing models through resource usage and system characteristic similarity classification, and then automatically provide optimization directions and strategies for corresponding scenarios based on the application's system characteristics.

Description

Technical Field

[0001] This application relates to the field of computers, specifically to a method for system optimization. Background Technology

[0002] An operating system (OS) is system software that manages computer hardware and software resources; it is also the kernel and foundation of a computer system. For example... Figure 1 The diagram shows the logical architecture of a computer system. A computer system includes a hardware layer, an operating system, and an application layer. The operating system manages various hardware resources in the hardware layer and provides resources for applications in the application layer. Ensuring optimal performance is always the primary goal of the operating system. The core of achieving this goal is to enable the operating system to understand the status of applications running at the upper layers and perform system optimizations to improve system performance.

[0003] In existing technologies, operating systems identify application operations, such as read and write operations, based on the application's historical runtime data; and adjust the application's parameter configuration accordingly to improve system performance. However, with thousands of open applications, it is difficult for operating systems to perform customized system optimizations for all applications in advance, making it difficult to effectively maximize system performance. Summary of the Invention

[0004] This application provides a system optimization method that can identify the business type of any application running on a system, and then perform targeted system optimization. Furthermore, by first clustering and then classifying, it can also specifically identify the business types of applications under different performance bottleneck types, improving the accuracy of identifying the business types of applications running on the operating system, and enabling more directional system optimization.

[0005] In a first aspect, this application provides a method for system optimization, the method being applied in an operating system on which one or more first applications are running, the method comprising:

[0006] Collect first system feature data, which includes system feature data in multiple dimensions, and the system feature data is used to describe the system's resource usage or system behavior;

[0007] The first system feature data of each dimension is input into the clustering model corresponding to the dimension to obtain the performance bottleneck type of the application. The performance bottleneck type is used to describe the indicators that restrict the performance improvement of the application. The clustering model of each dimension is generated by a clustering algorithm with the second system feature data of the dimension as input. The second system feature data includes the system feature data collected by each of the multiple second applications during runtime.

[0008] A classification model corresponding to the performance bottleneck type is determined. This classification model is trained using second system feature data belonging to the performance bottleneck type as input and the business type of the second application as output. In other words, the system stores multiple clustering models, each with a corresponding classification model. Each classification model is trained using data under that performance bottleneck type, and the trained classification model can accurately identify the business type of the application under that performance bottleneck type.

[0009] The first system feature data is input into the classification model to obtain the business type of the first application, and the business type is used to describe the function of the first application.

[0010] Based on the business type, determine the corresponding system optimization strategy to optimize the system.

[0011] This application sets different optimization strategies for different business types, which can be used to optimize the system in a targeted manner.

[0012] In one implementation, the method can also determine the system optimization strategy based on the changes in the performance bottleneck points corresponding to the business type and performance bottleneck type.

[0013] In a second aspect, this application provides a computer system comprising a processor and a memory, the memory storing computer instructions, and the processor invoking the computer instructions to perform the method provided in the first aspect above.

[0014] Thirdly, this application provides a computer-readable storage medium including computer instructions that are invoked by a processor to perform the method described in the first aspect.

[0015] Fourthly, this application provides a system optimization method. The device includes a data acquisition unit, a service type perception unit, a decision-making unit, and an execution unit.

[0016] The acquisition unit is used to acquire first system feature data, which includes system feature data in multiple dimensions. The system feature data is used to describe the system's resource usage or system behavior.

[0017] The service type awareness unit is used for:

[0018] The first system feature data of each dimension is input into the clustering model corresponding to the dimension to obtain the performance bottleneck type of the application. The performance bottleneck type is used to describe the indicators that restrict the performance improvement of the application. The clustering model of each dimension is generated by a clustering algorithm with the second system feature data of the dimension as input. The second system feature data includes the system feature data collected by each of the multiple second applications during runtime.

[0019] Determine the classification model corresponding to the performance bottleneck type. The classification model is a model trained with second system feature data belonging to the performance bottleneck type as input and the business type of the second application as output.

[0020] The first system feature data is input into the classification model to obtain the business type of the first application, and the business type is used to describe the function of the first application.

[0021] The decision-making unit is used to determine the corresponding system optimization strategy based on the business type in order to optimize the system.

[0022] Fifthly, this application provides a model training or generation method, the method being applied to a training device, characterized in that the method includes:

[0023] Obtain second system feature data in multiple dimensions. The second system feature data includes system feature data collected by each application during runtime in multiple applications. The system feature data is used to describe the system's resource usage or system behavior.

[0024] The second system feature dataset for each dimension is input into the clustering algorithm to obtain the clustering model for each dimension and the performance bottleneck type for each application.

[0025] The system feature data of applications belonging to the same performance bottleneck type are used as input to the classification model, and the business type of the applications belonging to the same performance bottleneck type is used as output to train the classification model, thus obtaining the trained classification model.

[0026] In one implementation, the method further includes:

[0027] Collect raw system feature data, which includes system feature data in multiple dimensions;

[0028] Based on the training results of the classification model, system feature data with noise dimension is determined from the original system feature data, wherein the system feature data with noise dimension is data that is unrelated to the business type;

[0029] The important system feature data used to generate the clustering model are identified from the system feature data of the noise dimension; the data in the original system feature data other than the unimportant system feature data are the second system feature data of the multiple dimensions obtained, and the unimportant system feature data are the data in the system feature data of the noise dimension other than the important system feature data.

[0030] Sixthly, this application provides a computer system including a processor and a memory, the memory storing computer instructions, and the processor invoking the computer instructions to execute the methods provided in the fifth aspect and any implementation thereof.

[0031] In a seventh aspect, this application provides a computer-readable storage medium including computer instructions that are invoked by a processor to perform the method provided in the fifth aspect and any implementation thereof.

[0032] Eighthly, this application provides a training device, which includes a data acquisition unit and a training unit.

[0033] The acquisition unit is used to: obtain second system feature data in multiple dimensions, the second system feature data including system feature data collected by each application during runtime in multiple applications, the system feature data being used to describe the system's resource usage or system behavior;

[0034] The training unit is used to: input the second system feature dataset of each dimension into the clustering algorithm to obtain the clustering model of each dimension and the performance bottleneck type of each application;

[0035] The system feature data of applications belonging to the same performance bottleneck type are used as input to the classification model, and the business type of the applications belonging to the same performance bottleneck type is used as output to train the classification model, thus obtaining the trained classification model.

[0036] In one implementation, the device further includes a preprocessing unit.

[0037] The acquisition unit is also used to: acquire raw system feature data, which includes system feature data in multiple dimensions;

[0038] The preprocessing unit is used to: determine noisy system feature data from the original system feature data based on the training results of the classification model, wherein the noisy system feature data is data unrelated to the business type;

[0039] The important system feature data used to generate the clustering model are identified from the noise system feature data; the data in the original system feature data other than the unimportant system feature data are the obtained second system feature data of multiple dimensions, and the unimportant system feature data are the data in the noise system feature data other than the important system feature data. Attached Figure Description

[0040] Figure 1 A schematic diagram of the structure of a computer system provided in an embodiment of this application;

[0041] Figure 2 Logical architecture diagram of the computer system provided in the embodiments of this application;

[0042] Figure 3 A flowchart of the system optimization method provided in the embodiments of this application;

[0043] Figure 4 A flowchart of the system optimization method provided in the embodiments of this application;

[0044] Figure 5 Logical architecture diagram of the computer system provided in the embodiments of this application;

[0045] Figure 6 Flowcharts of the system optimization method and model training or generation method provided in the embodiments of this application;

[0046] Figure 7 A schematic diagram illustrating the raw data acquisition provided in an embodiment of this application;

[0047] Figure 8 A method for filtering system features provided in the embodiments of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0049] The terms "first" and "second" used in this application are used to distinguish objects of the same type and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, unless otherwise stated, "a plurality of" means two or more.

[0050] Figure 1This is a schematic diagram of the structure of a computer system 100, which includes a processor 101, a memory 102, and may also include an input / output system 103 and a network interface 104. Those skilled in the art will understand that the computer system 100 may include more or fewer components than shown, or combine certain components. These components can communicate via one or more buses 105 or signal lines, and the buses may be categorized as address buses, data buses, control buses, etc.

[0051] As described herein, computer system 100 may include hardware and software components for implementing the methods described in this application. Processor 101 may be configured to implement some or all of the methods described in this application, for example by executing program instructions stored on memory 102 (e.g., a non-volatile computer-readable storage medium).

[0052] Processor 101 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. Processor 101 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0053] Memory 102 is used to store computer programs, such as Figure 1 The application program 301 and operating system 202 are shown; the processor 101 can call computer programs stored in the memory 102 to implement the functions defined by the computer program. For example, the processor 101 executes the operating system to implement various functions of the operating system on the computer system 100. The operating system 202 can be... This application does not impose any restrictions on other operating systems, including those used in the embodiments of the present application.

[0054] Memory 102 also stores other data besides computer programs, such as data generated during the operation of operating system 202 and application 301. Memory 102 may include volatile memory, such as random-access memory (RAM); memory 102 may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 102 may also include combinations of the above types of memory. Figure 1 Only one memory is shown here. Of course, memory 102 can also be configured as multiple storage devices as needed. Memory 102 can also be the memory in processor 101, and there is no limitation here.

[0055] The input / output system 103, such as the Basic Input / Output System (BIOS), also known as ROM BIOS, System BIOS, or PC BIOS, is the firmware that performs hardware initialization during the power-on boot phase and provides runtime services for the operating system and programs.

[0056] Network interface 104 is used to connect other network devices, including wireless and wired connections. Specifically, it can be a network interface card (NIC) or a network adapter.

[0057] The above components are connected via bus 105. Bus 105 can be one or multiple. Bus 105 includes Advanced Microcontroller Bus Architecture (AMBA), Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Extended ISA bus, Video Electronics Standards Association (VESA) local area bus, and Peripheral Component Interconnect (PCI) bus, etc.

[0058] Furthermore, in one embodiment, such as Figure 2As shown, the computer system 100 can be logically divided into an application layer 201, an operating system 202, and a hardware layer 203. The application layer 201 includes one or more application programs 301 (which can be simply referred to as applications). The operating system 202 provides the hardware resources in the hardware layer 203 to the application programs 301 in the application layer 201. These hardware resources may include computing resources, storage resources, and network resources. The operating system 202, acting as software middleware between the hardware layer 203 and the application layer 201, is a computer program that manages and controls hardware and software resources.

[0059] Existing system optimization methods target a specific application, adjusting application parameters based on the identified operations being performed by the application. For example, collecting network traffic data and using classification algorithms to identify service types. However, network traffic data, as a behavioral characteristic, cannot be applied to identifying the service type of any application. Furthermore, different users run different applications on different operating systems, making it difficult for the operating system to perform customized system optimizations for all applications in advance. Therefore, the direction of system optimization by the operating system is unclear, and system performance is difficult to effectively achieve.

[0060] Although application behavior characteristics have limitations, applications can still be categorized into a limited number of types based on their behavior within the system. For example, speccpu (a tool for monitoring CPU performance) and redis (a database application) are different types of applications, but both exhibit the system characteristic of high CPU utilization. In this embodiment, based on the system characteristic of high CPU utilization, it can be determined that the bottleneck restricting the performance improvement of both speccpu and redis is CPU utilization. Therefore, in this embodiment, speccpu and redis can be classified as CPU-intensive, indicating that the factor restricting the performance improvement of these two applications is CPU usage. Subsequently, based on the classification model corresponding to the previously identified performance bottleneck types, the application's business type is precisely classified.

[0061] In the embodiments of this application, such as Figure 3 and Figure 4As shown, the operating system can identify the performance bottleneck type of an application using a pre-generated clustering model based on the collected system feature data, and further identify the business type of the application under that performance bottleneck type using a trained classification model; then, it can optimize the operating system using a system optimization strategy corresponding to the application type. Here, a performance bottleneck indicates that performance improvement is constrained, and the performance bottleneck type describes the factors that constrain or affect the application's performance improvement. These factors can be the application's usage of hardware resources, which can include computing resources, input / output (IO) resources, storage resources, or bandwidth resources. Optionally, as shown in Table 1, the performance bottleneck type can be any of the following: CPU-intensive, IO-intensive, storage-intensive, or network-intensive. The performance bottleneck type can also be a combination of any number of the above four types, such as CPU / IO-intensive, indicating that the application's performance bottleneck is CPU usage and IO request waiting.

[0062] Table 1 Explanation of Performance Bottleneck Types

[0063] Performance bottleneck types illustrate CPU intensive The performance bottleneck of an application is CPU usage. IO intensive This indicates that the application's performance bottleneck is the waiting time for I / O requests. Storage intensive This indicates that the application's performance bottleneck is storage space usage. Bandwidth intensive This indicates that the application's performance bottleneck is bandwidth usage.

[0064] Figure 5 for Figure 1 and Figure 2 The computer system 100 is shown in the following logical structure diagram. The operating system 202 may specifically include middleware 302, a basic library 303, a runtime environment 304, a kernel 305, and a virtualization layer 306. The hardware layer 203 may include various hardware components, such as a processor 101 and a BIOS 103. The middleware 302 provides a unified interface for upper-layer applications, shielding them from underlying details. The basic library 303 provides specific encapsulation for specific functions. It includes a model library and an intelligence library. The model library includes multiple machine learning models, such as the clustering and classification models mentioned in various embodiments of this application. The intelligence library includes multiple system configuration strategies that can be used for system optimization, each corresponding to a business type. The kernel 305 provides underlying system components and services; in other words, it provides hardware abstraction and management. The compiler 306 compiles high-level languages ​​into chip-executable instructions.

[0065] Programmable probes are used to collect system characteristics at their location; essentially, a programmable probe is a script for collecting data. The location of a programmable probe can be set as follows: Figure 4As shown, the programmable probes are located in application 301, middleware 302, virtualization layer 306, BIOS 103, and CPU 101. Specifically, the programmable probes collect system characteristic data from application 301, middleware 302, virtualization layer 306, BIOS 103, and CPU 101, such as resource utilization, read / write activity, bandwidth, and average instruction cycles. Of course, the location of the programmable probes in the computer system can be set by the developer or user.

[0066] The training or generation process of clustering and classification models in the model library included in the base library can be completed before the operating system software package is shipped.

[0067] The runtime environment 304 provides a runtime environment for developers and users. The system optimization device in this embodiment can be located within the runtime environment. Specifically, the system optimization device 400 includes an abstract perception module 401, a decision module 402, and an execution module 403.

[0068] The abstract perception module 401 can collect system feature data through various layers of the editable probe operating system 202, the application layer 201, and the hardware layer 203. It can also organize the collected system feature data. Furthermore, the abstract perception module 401 can call clustering and classification models in the basic library 203, inputting system feature data into the clustering model to obtain the system's performance bottleneck type; inputting system feature data belonging to the same performance bottleneck type into the classification module corresponding to that type to obtain the business type of the applications running on the system; and finally, passing the identified performance bottleneck type and business type to the decision module.

[0069] The decision module 402 receives the performance bottleneck type and service type sent by the abstract perception module 401, and searches for the corresponding system configuration policy in the intelligent library in the basic library 203 according to the service type. The system configuration policy can be a set of configuration items, which includes the values ​​of multiple configuration items. The configuration items can be kernel configuration parameters, BIOS parameters, core binding, or adjustment of OS behavior, etc.

[0070] The execution module 403 is used to perform system optimization according to the system configuration strategy determined by the decision module 402, that is, to adjust the values ​​of system configuration items according to the system configuration strategy. Specifically, it adjusts the values ​​of the corresponding parameters in the kernel to the values ​​of the parameters in the system configuration item set according to the system configuration item set; or it adjusts the values ​​of the configuration items in the compiler 307 through the optimizer FDO (Feedback-Directed Optimization) in the compiler 307.

[0071] Before detailing the system optimization method provided in this application, we will first introduce the generation or training process of the clustering and classification models used in this system optimization method. A flowchart can be found here. Figure 6 As shown.

[0072] Optionally, the device used to execute the model generation or training process and the device used to execute the system optimization method can be the same device or different devices.

[0073] This application provides a method for generating a clustering model, which includes the following steps:

[0074] Step 501: Obtain the system feature dataset;

[0075] The system feature dataset includes system feature data from multiple applications. System feature data describes the resource usage or behavior of a computer system. For each application, the system feature data consists of the values ​​of N dimensions of system features collected during the application's runtime, where N is a positive integer. Each dimension of the system feature can include multiple types. Dimensions can include multiple of the following: CPU dimension, I / O dimension, network dimension, or storage dimension, and can also include task dimension and interrupt dimension. Therefore, the system feature dataset can be divided into N categories according to dimensions, with each category including system feature data from multiple applications under the same dimension.

[0076] Optionally, the dimensions of the system feature data for each application in the system feature dataset can be one column as shown in Table 2, and the specific types of system features for each application can be one column as shown in Table 2.

[0077] Optionally, applications of different business types can run on the system, and data can be collected from the system through programmable probes during application runtime, namely, system characteristic data in N dimensions. These applications can be business types such as databases, big data applications, in-memory computing applications, or network servers. Specifically, the system characteristic data can include resource utilization, read / write activity, bandwidth, or average instruction cycles.

[0078] Table 2. Dimensions, Types, and Representations of System Features

[0079]

[0080]

[0081] Optionally, system feature data includes the values ​​of system features at multiple points in time during application operation. For example, for application A, the system feature data for application A could be the values ​​of each system feature in Table 2 at different points in time.

[0082] Optionally, before step 501, the collected raw data can be preprocessed and dimensionality reduced to filter out system feature data.

[0083] Step 502: Cluster the system feature dataset using a clustering algorithm to determine the performance bottleneck types and clustering models for multiple applications;

[0084] Clustering algorithms can group applications with similar system feature data into a single category. By inputting the system feature dataset into a clustering algorithm, the algorithm obtains classification results for multiple applications. Applications belonging to the same category in the classification results share similar system features.

[0085] The system feature data of N dimensions are input into the corresponding clustering algorithms to obtain the clustering results of the system feature data in the same dimension and N clustering models. The system feature data of each dimension are input into the clustering algorithm to obtain the clustering results of each dimension and the clustering model of each dimension.

[0086] For example, the system feature dataset includes system feature data for each of 20 applications: Application 1, Application 2... Application 20. Each application's system feature data includes four dimensions: CPU, I / O, network, and storage. These four dimensions are then input into a clustering algorithm, such as the K-meas algorithm. It's important to note that only one dimension's system feature data is input into the clustering algorithm at a time, yielding the clustering result for that dimension. Let's assume the CPU dimension's system feature is CPU utilization, the I / O dimension's is I / O utilization, the network dimension's is bandwidth utilization, and the storage dimension's is storage capacity utilization. The clustering results for each dimension are shown in Table 3. The clustering results indicate that the performance bottleneck for applications 1-5 is CPU utilization, and for applications 6-20, it's I / O utilization. Furthermore, the clustering models corresponding to each of the four dimensions are obtained: the clustering model for the CPU dimension, the clustering model for the I / O dimension, the clustering model for the network dimension, and the clustering model for the storage dimension.

[0087] Optionally, before step 502, important system feature data are evaluated and analyzed using the silhouette coefficient and elbow method to obtain the number of clusters.

[0088] Table 3 Clustering results across four dimensions

[0089]

[0090]

[0091] For example, if multiple applications have high CPU utilization, they can be clustered into the same category. The performance bottleneck of these applications in this category can be considered to be CPU utilization, thus classifying them as CPU-intensive. Similarly, by identifying the performance bottleneck type of each application among multiple applications, if most applications in a cluster have high memory usage, then the performance bottleneck type of this category is determined to be storage-intensive; if most applications in a cluster have high bandwidth usage, then the performance bottleneck type of this category is determined to be network-intensive. The performance bottleneck type can be any of the following: CPU-intensive, I / O-intensive, storage-intensive, or bandwidth-intensive. The performance bottleneck type can also be a combination of any of the above four types, such as CPU / I / O-intensive, indicating that the application's performance bottleneck is CPU usage and I / O request waiting. For example, if the clustering result shows that an application has both high CPU and I / O utilization, then the performance bottleneck type of this application is CPU / I / O-intensive.

[0092] For example, the clustering results of the system feature data of the MangoDB application show low CPU utilization, network bandwidth utilization, and memory utilization in the cluster category to which the application belongs, but long I / O process wait times, indicating that MangoDB is an I / O-intensive application and the performance bottleneck is in the I / O direction. The system feature data of the Spark application shows the same clustering results, reflecting the same performance bottleneck. The clustering results of the system feature data of the Nginx application show high CPU utilization and low network memory and I / O utilization, indicating that Nginx is a CPU-intensive application and the performance bottleneck is in the CPU direction.

[0093] Optionally, for each specific type of system feature under each dimension, the values ​​of that system feature at different times can be collected. This allows for further identification of the performance bottleneck of an application at different times. For example, the CPU utilization of applications 1-5 in Table 3 represents the CPU utilization of applications 1-5 at time A. Therefore, the clustering result indicates that the performance bottleneck of applications 1-5 at time A is the CPU utilization. After labeling the application's business type, the performance bottleneck of applications of different business types at different times can be obtained, i.e., the changes in the performance bottleneck of applications of different business types during operation.

[0094] This application provides a method for training a classification model, the method comprising the following steps:

[0095] Step 601: Construct a labeled system feature dataset;

[0096] Based on the system feature dataset obtained in step 501, each system feature data in the dataset can be labeled with a business type, resulting in a labeled system feature dataset. Several business types can be pre-defined based on experience, allowing each system feature data application to be categorized into one of these business types when the system feature dataset is obtained. Business types can be network-based, big data-based, or database-based, etc.

[0097] Optionally, preprocessing such as standardization and anomaly detection can be performed on the labeled system feature dataset.

[0098] Step 602: Using system feature data of multiple applications under the same performance bottleneck type as input and the business type of each application as output, train a classification model for applications belonging to this performance bottleneck type.

[0099] The labeled system feature datasets are grouped according to performance bottleneck type, with system feature data of the same performance bottleneck type grouped together. The model is then trained sequentially for each group of labeled system feature data, maintaining the corresponding classification model.

[0100] For example, if a set of system feature data has a performance bottleneck type of CPU intensive, and there are multiple CPU intensive applications, these applications may have different business types. Then, the system feature data can be used as the input of a classifier, and the business type can be used as the output of the classifier to obtain a trained classification model.

[0101] Optionally, the number of classification models trained corresponds to the number of performance bottleneck types. These trained classification models are then stored in the model library within the operating system's base library.

[0102] Optionally, multiple rounds of training can be conducted, and the input system feature data can be filtered and the data dimensions adjusted based on the accuracy and weight of each round of training feedback. This enables perceptual training for specific business types, reduces the impact of data fluctuations on classification results, and establishes a robust and accurate business category classification model.

[0103] Optionally, the classification model used can be a model provided by classification algorithms such as support vector machines (SVM), Xgboost, and deep neural networks (DNN).

[0104] This application provides a system optimization method, which is applied to a computer system, specifically within an operating system. One or more applications run on the computer system. A flowchart of the method is shown below. Figure 5 As shown, the method includes:

[0105] Step 701: Collect system feature data;

[0106] System characteristic data can be used to describe the resource usage or behavior of a computer system. System characteristic data is collected from multiple dimensions, each of which can be any number of the following: CPU, I / O, network, or storage. For example, system characteristic data may include one or more of the following: CPU usage, I / O read / write activity, storage space usage, or bandwidth usage. Furthermore, system characteristic data may also include average instruction cycles.

[0107] Optionally, the collected system feature data may be multi-dimensional system feature data collected from the runtime of only one application. For example, CPU utilization, IO utilization, bandwidth utilization, and storage space utilization of application A may be collected.

[0108] Step 702: Input the system feature data into the clustering model to obtain the application's performance bottleneck type;

[0109] A clustering model is a model obtained by clustering system feature data using an unsupervised clustering algorithm. The system feature data used to generate the clustering model is the system feature data collected when each application runs on the system. Unsupervised means that the input data is automatically classified or grouped without any given human training examples. Clustering is an unsupervised learning method that groups similar objects into different groups through static classification methods, so that objects in each subset have some similar attributes. The input to the clustering model is the collected system feature data.

[0110] There are multiple clustering models, each corresponding to a dimension of system feature data. Each clustering model can cluster the system feature data under its corresponding dimension to determine the type of application performance bottleneck. Each clustering model is the output model of a clustering algorithm that takes system feature data from multiple applications under one of the multiple dimensions as input.

[0111] Optionally, the clustering model is based on the clustering model generated in steps 601-602;

[0112] For example, clustering models include CPU-based, I / O-based, network-based, and storage-based clustering models. When the application's CPU utilization is input into the CPU-based clustering model, if the cluster belongs to Class 1 under the CPU dimension in Table 3 (i.e., most of the clusters show high CPU utilization (e.g., 90-99%)), then the application's performance bottleneck is CPU utilization, and the bottleneck type is CPU-intensive. Similarly, when the application's I / O utilization is input into the I / O-based clustering model, if most of the clusters show low I / O utilization (e.g., 10-20%), then the application's performance bottleneck is not in the I / O dimension.

[0113] Step 703: Determine the classification model corresponding to the performance bottleneck type;

[0114] Each performance bottleneck type corresponds to a classification model, which can further analyze the specific business type of the applications running on the system. The classification model for each performance bottleneck type is trained by taking the system feature data of multiple applications of that performance bottleneck type as input and the business type of each application as output.

[0115] Optionally, the training method for the classification model can be found in steps 601-602.

[0116] Step 704: Input the collected system feature data into the determined classification model to obtain the application's business type;

[0117] The application's business type can be specifically categorized as: network-based, big data-based, in-memory computing-based, or database-based, etc.

[0118] Step 705: Determine the corresponding system optimization strategy based on the business type;

[0119] The intelligent library in the basic library stores system optimization strategies for each of the various business types. Each business type corresponds to one system optimization strategy, and the system optimization strategy for each business type can be a pre-set value for multiple system configuration items.

[0120] Optionally, the system stores the performance bottlenecks of different business types of applications at different times, that is, the changes in the performance bottlenecks of different business types of applications during operation.

[0121] Optionally, corresponding system optimization strategies can be determined based on changes in business type and performance bottlenecks.

[0122] Step 706: Adjust system configuration according to system optimization strategy.

[0123] Adjust the value of the configuration item in the system according to the value recorded in the system optimization strategy.

[0124] For example, for network-intensive applications with a communication business type, the corresponding system optimization is to optimize the network interface card (NIC), specifically by enabling multiple queues and binding NIC interrupts to the nearest physical core. For CPU / storage-intensive applications with an in-memory computing business type, the corresponding system optimization can include one or more of the following: disabling memory prefetching; enabling static memory and static big pages; binding processes to NUMA. For I / O-intensive applications with a big data storage business type, the corresponding system optimization can include one or more of the following: increasing the size of disk prefetched data; optimizing disk mount options (noatime and nobarrier); adjusting the time for dirty page data to be written back to disk.

[0125] The system optimization method provided in this application can identify the business type of a running application in real time, and then take corresponding system optimization strategies. This method first identifies the performance bottleneck type of the application based on its system feature data, and then analyzes the application's business type based on the classification model corresponding to the identified performance bottleneck type. Clustering followed by classification means that applications with the same performance bottleneck type have similar characteristics. A classification model trained using system feature data of the same performance bottleneck type can more accurately identify the application's business type. However, if the classification model is directly trained using system feature data, the same system feature data may vary greatly at different times, leading to different business types identified by the classification model for the same application at different times, resulting in lower accuracy in identifying the business type.

[0126] Table 4 shows the performance improvement of the following applications before and after applying the system optimization method provided in the embodiments of this application.

[0127] Table 4. Performance comparison of several applications before and after applying this method.

[0128]

[0129] This application identifies bottleneck types and business types in the current system based on a pre-trained business feature classification model within a self-optimizing system. Table 4 shows that, through abstract identification and optimization of the operating system configuration via the self-optimizing system, performance improvements have been achieved in various storage, computing, big data, and network application scenarios, including MangoDB (database application), SPECjbb (server business testing tool used to test the performance level of Java business application servers), Hadoop (big data application), and Nginx (network server). This application constructs a one-to-one correspondence between business features by modeling a large amount of system feature data using unsupervised and supervised learning, identifying the relationship between business bottlenecks and resources, and thus optimizing specific resources to improve the performance of various business types.

[0130] To ensure the accuracy of the classification model obtained through training, it is necessary to provide correct training data, which means that the original system features collected need to be screened.

[0131] This application provides a method for filtering system features, the flowchart of which is shown below. Figure 7 As shown, the method includes:

[0132] Step 801: Collect original system features;

[0133] In order to ensure that the data dimensions of the original system features collected can meet the needs of business profiles in multiple scenarios and categories, this application strives to collect data from all scenarios as much as possible.

[0134] In various embodiments of this application, such as Figure 6 The raw system features collected can include system features in multiple dimensions, which can be: CPU dimension (CPU-related parameters), network dimension (network controller port-related parameters), IO dimension (IO controller-related parameters), task dimension (task-related parameters), system performance-related parameters, or storage dimension (storage capacity-related parameters). The specific raw dimensions of the collected data can be referred to Table 2 above. Based on the analysis of global software and hardware resources, the resources involved in system operation are identified, and data in multiple dimensions are collected for each resource, including utilization-related features, saturation, and performance-related dimensions, to achieve 360-degree panoramic data collection.

[0135] Step 802: Identify the feature data of the noisy system based on the training results of the classification model;

[0136] When training a classification model or generating a clustering model, the system features are re-screened based on the fluctuations and instability of noise features perceived by the parameter weights and recognition accuracy based on the model feedback. Step 801 aims to collect data from all scenarios, but not all system features are related to business types. For the same application, the data collected for the same system feature at different times should maintain consistency. If inconsistency exists, that dimension is considered a noisy system feature. Noisy system features are unrelated to business types and will affect the accuracy of the classification model. This application utilizes the classification results of a supervised training model to identify noisy system features. If, in the final classification results, an application is identified as belonging to more than two business types, then, in a certain order, each system feature is identified as causing the application to be classified into multiple categories; this system feature is the noise dimension.

[0137] Optionally, before step 802, the accuracy and weight information of the classification model fed back in this round of training are obtained.

[0138] Optionally, after step 802, the system features with the noise dimension are removed from the original system features. The original system features after removing the system features with the noise dimension are the system features used to train the classification model in step 602.

[0139] Step 803: Identify important system features from the noise system characteristics.

[0140] Based on the collected original system features, factor analysis and chi-square test are used to identify the important system features. For system features in the noise dimension, instead of simply removing them, we examine whether any important system features exist within the noise dimension. Features in the noise dimension other than the important ones can be considered unimportant system features.

[0141] Noise arises from two main causes: one is inappropriate data selection, rendering it unsuitable as a feature; this is genuine noise. The other is data fluctuation caused by changes in hardware or load pressure, which manifests as noise and cannot be simply categorized as noise. For this type of data, we first determine whether it is an important feature in the most critical scenarios. If it is an important feature, further preprocessing can be performed (e.g., CPU utilization balancing). If it is not an important feature, it can be removed.

[0142] Optionally, unimportant system features are removed from the original system features. The original system features after removing unimportant system features are the system features used in step 502 to generate the clustering model by inputting the clustering algorithm and the system features used in step 602 to train the classification model.

[0143] Optionally, it may also include step 804: preprocessing important system features in the noisy system features based on the feedback of the clustering results;

[0144] The main reason why critical features become noise is that changes in input load or hardware platform cause these values ​​to fluctuate and become noise. However, these critical features cannot be simply removed; they need to be preprocessed so that their values ​​remain essentially unchanged when faced with changes in load or hardware platform. Therefore, the main purpose of preprocessing is to eliminate the fluctuations in critical features caused by changes in load or hardware platform through certain methods.

[0145] By processing the original system features as described above, the impact of data fluctuations on classification results can be reduced, and robust and accurate business classification and clustering models can be established.

[0146] The following is a specific example to illustrate what the noise dimension is.

[0147] For example, among the dimensions of the raw system features collected, CPU utilization is a crucial dimension. This dimension plays a key role in unsupervised clustering, used to identify whether a business is CPU-intensive. In actual system feature data collection for applications, different business load scenarios must be considered. For example, with Nginx, the number of processes running on the server side might be considered, such as 32 or 128. Each Nginx process occupies a CPU core, causing the CPU utilization of that core to approach 100%, thus classifying the application as CPU-intensive. Since the initial calculation focuses on the CPU utilization of the entire system (128 CPU cores), the CPU utilization is close to 100% for a scenario with 128 processes, but less than 30% for a scenario with 32 processes. Therefore, the CPU utilization dimension leads to the Nginx application with 32 processes (workers) being classified as non-CPU-intensive; the CPU utilization dimension becomes a noise dimension. However, this dimension cannot be simply removed, otherwise we will not be able to identify CPU-intensive services. Therefore, we need to preprocess this dimension so that Nginx with 32 workers can still be identified as a CPU-intensive service. Here, we only count the utilization of the CPU cores running the Nginx service to replace the CPU utilization of the entire system to correct this noisy dimension and achieve the correct classification.

[0148] Existing technologies typically utilize AI algorithms (such as Lasso) to select important dimensions and then use these dimensions for classification. While this approach works for scenarios with fixed business types, if new business types emerge in the future, currently unimportant features may become important, leading to incorrect classification and identification of new businesses. This invention considers scalability in data dimension selection, filtering out only truly unimportant noisy dimensions and preprocessing important noisy dimensions to remove their noise attributes. This approach preserves as much comprehensive sampling data as possible, enabling the sampled data to support a wider range of future business types.

[0149] This application provides a system optimization device, which includes a data acquisition unit, a service type perception unit, and a decision-making unit. The data acquisition unit executes step 701, the service type perception unit executes steps 702-704, and the decision-making unit executes steps 705-706. The service type perception unit can be... Figure 5 The abstract perception module 401 in the middle, the decision unit is Figure 5 Module 402 of the decision-making module.

[0150] This application provides a model training device, which includes an acquisition unit and a training unit. The acquisition unit is used to execute steps 501 and 801, and the training unit is used to execute steps 502, 601 and 602. The device also includes a preprocessing unit, which is used to execute steps 802-804.

[0151] This application also provides a computer-readable storage medium storing computer instructions that, when invoked by a processor, execute the methods provided in any embodiment of this application.

[0152] This application also provides a computer program product, which includes computer instructions that, when invoked by a processor, execute the method provided in any embodiment of this application.

[0153] This application also provides a computer system including a processor and a memory, wherein the processor invokes computer memory stored in the memory to execute the method provided in any embodiment of this application.

[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0158] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A system optimization method, said method being applied in an operating system, wherein one or more first applications are running on the operating system, characterized in that, include: Collect first system feature data, which includes system feature data in multiple dimensions, and the system feature data is used to describe the system's resource usage or system behavior; The first system feature data of each dimension is input into the clustering model corresponding to the dimension to obtain the performance bottleneck type of the application. The performance bottleneck type is used to describe the indicators that restrict the performance improvement of the application. The clustering model of each dimension is generated by a clustering algorithm with the second system feature data of the dimension as input. The second system feature data includes the system feature data collected by each of the multiple second applications during runtime. Determine the classification model corresponding to the performance bottleneck type. The classification model is a model trained with second system feature data belonging to the performance bottleneck type as input and the business type of the second application as output. The first system feature data is input into the classification model to obtain the business type of the first application, and the business type is used to describe the function of the first application. Based on the business type, determine the corresponding system optimization strategy to optimize the system.

2. The method according to claim 1, characterized in that, The multiple dimensions include one or more of the following: CPU resource dimension, IO resource dimension, network resource dimension, or storage resource dimension.

3. The method according to claim 1, characterized in that, The performance bottleneck types include one or more of the following: CPU-intensive, I / O-intensive, network-intensive, or storage-intensive.

4. The method according to claim 1, characterized in that, The computer system stores the changes in the performance bottleneck point of each of the multiple performance bottleneck types during the application's runtime.

5. The method according to claim 4, characterized in that, The step of determining the corresponding system optimization strategy based on the business type includes: The system optimization strategy is determined based on the changes in the performance bottleneck points corresponding to the business type and the performance bottleneck type.

6. A computer system, characterized in that, The computer system includes a processor and a memory, the memory storing computer instructions, and the processor invoking the computer instructions to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that are invoked by a processor to perform the method as described in any one of claims 1-5.

8. A system optimization device, characterized in that, The device includes a data acquisition unit, a service type perception unit, a decision-making unit, and an execution unit. The acquisition unit is used to acquire first system feature data, which includes system feature data in multiple dimensions. The system feature data is used to describe the system's resource usage or system behavior. The service type awareness unit is used for: The first system feature data of each dimension is input into the clustering model corresponding to the dimension to obtain the performance bottleneck type of the application. The performance bottleneck type is used to describe the indicators that restrict the performance improvement of the application. The clustering model of each dimension is generated by a clustering algorithm with the second system feature data of the dimension as input. The second system feature data includes the system feature data collected by each of the multiple second applications during runtime. Determine the classification model corresponding to the performance bottleneck type. The classification model is a model trained with second system feature data belonging to the performance bottleneck type as input and the business type of the second application as output. The first system feature data is input into the classification model to obtain the business type of the first application, and the business type is used to describe the function of the first application. The decision-making unit is used to determine the corresponding system optimization strategy based on the business type in order to optimize the system.

9. A model training or generation method, said method being applied to a training device, characterized in that, The method includes: Obtain second system feature data in multiple dimensions. The second system feature data includes system feature data collected by each application during runtime in multiple applications. The system feature data is used to describe the system's resource usage or system behavior. The second system feature dataset for each dimension is input into the clustering algorithm to obtain the clustering model for each dimension and the performance bottleneck type for each application. The system feature data of applications belonging to the same performance bottleneck type are used as input to the classification model, and the business type of the applications belonging to the same performance bottleneck type is used as output to train the classification model, so as to obtain the trained classification model. The operating system inputs the first system feature data into the classification model to obtain the business type of the first application; determines the corresponding system optimization strategy based on the business type to optimize the system; the classification model is determined based on the performance bottleneck type; the business type is used to describe the function of the first application; the first system feature data includes system feature data of multiple dimensions, and the system feature data is used to describe the system's resource usage or system behavior.

10. The method according to claim 9, characterized in that, The method further includes: Collect raw system feature data, which includes system feature data in multiple dimensions; Based on the training results of the classification model, system feature data with noise dimension is determined from the original system feature data, wherein the system feature data with noise dimension is data that is unrelated to the business type; The important system feature data used to generate the clustering model are identified from the system feature data of the noise dimension; the data in the original system feature data other than the unimportant system feature data are the second system feature data of the multiple dimensions obtained, and the unimportant system feature data are the data in the system feature data of the noise dimension other than the important system feature data.

11. A computer system, characterized in that, The computer system includes a processor and a memory, the memory storing computer instructions, and the processor invoking the computer instructions to perform the method as described in any one of claims 9-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that are invoked by a processor to perform the method as described in any one of claims 9-10.

13. A training device, characterized in that, The device includes a data acquisition unit and a training unit. The acquisition unit is used to: obtain second system feature data in multiple dimensions, the second system feature data including system feature data collected by each application during runtime in multiple applications, the system feature data being used to describe the system's resource usage or system behavior; The training unit is used to: input the second system feature dataset of each dimension into the clustering algorithm to obtain the clustering model of each dimension and the performance bottleneck type of each application; The system feature data of applications belonging to the same performance bottleneck type is used as input to the classification model, and the business type of the applications belonging to the same performance bottleneck type is used as output to train the classification model, resulting in a trained classification model. This allows the operating system to input the first system feature data into the classification model to obtain the business type of the first application. Based on the business type, a corresponding system optimization strategy is determined for system optimization. The classification model is determined based on the performance bottleneck type. The business type describes the function of the first application. The first system feature data includes system feature data in multiple dimensions, which describes the system's resource usage or system behavior.

14. The device according to claim 13, characterized in that, The device also includes a preprocessing unit. The acquisition unit is also used to: acquire raw system feature data, which includes system feature data in multiple dimensions; The preprocessing unit is used to: determine noisy system feature data from the original system feature data based on the training results of the classification model, wherein the noisy system feature data is data unrelated to the business type; The important system feature data used to generate the clustering model are identified from the noise system feature data; the data in the original system feature data other than the unimportant system feature data are the obtained second system feature data of multiple dimensions, and the unimportant system feature data are the data in the noise system feature data other than the important system feature data.

Images