A method for analyzing job resource consumption patterns based on BWO and clustering algorithm

Abstract

The present invention relates to a method for analyzing a job resource consumption pattern based on BWO and clustering algorithms, and belongs to the field of big data computing technology; the method comprises: (1) job operation and task scheduling; (2) monitoring of working node resource indicators; (3) data preprocessing and data set generation; (4) running a BWO / K-prototypes algorithm based on a White Whale optimization algorithm and a K-prototypes algorithm; (5) application of the optimal solution of the BWO / K-prototypes algorithm; (6) clustering result analysis; (7) acquisition and storage of the resource consumption pattern analysis results of a Flink job. The present invention can not only improve the execution efficiency of a Flink cluster and avoid exception throwing, but also reduce resource occupation and avoid unnecessary resource waste.

Description

Technical Field

[0001] The present invention relates to the field of big data computing technology, and more specifically, to a method for analyzing job resource consumption patterns based on BWO and clustering algorithms. Background Art

[0002] We are currently in an era of rapid development of big data, the Internet, cloud computing and artificial intelligence. The advancement of these technologies has driven a surge in the amount of global data, including massive amounts of real-time data. Therefore, we need to find more efficient ways to aggregate and analyze these massive amounts of real-time data in order to provide strong support for real-time business decisions and real-time data charts.

[0003] Apache Flink is an open source big data processing framework developed by the Apache Software Foundation and is widely used to process massive amounts of real-time data. In a Flink cluster, you can run a variety of Flink jobs. However, due to the heterogeneity of data sources and differences in processing operations, the resource consumption patterns of Flink jobs may vary, which poses a challenge to the resource configuration of the working nodes of the Flink cluster.

[0004] In actual applications, Flink clusters may face the situation where Flink jobs that consume a lot of memory resources are running in the cluster. If the memory resources of the working nodes are insufficient, the cluster will face performance bottlenecks and may even throw an OutOfMemoryError exception, causing the job to fail. Conversely, if the working nodes are configured with a large amount of CPU resources but are not fully utilized, the cluster will face resource waste.

[0005] Therefore, it is necessary to pay attention to the differences in resource consumption patterns between Flink jobs and explore methods to analyze the resource consumption patterns of Flink jobs. By using effective analysis methods to obtain the resource consumption patterns of each Flink job, when running Flink jobs in the future, more refined resource configuration can be performed on the Flink cluster based on the analysis results of the resource consumption patterns of the Flink jobs, thereby improving the execution efficiency of the cluster, avoiding exceptions while reducing resource usage and avoiding unnecessary resource waste. Among them, an effective method is to use the K-prototypes algorithm to perform cluster analysis on the monitoring data of the working nodes of the Flink cluster, thereby revealing the resource consumption patterns of each Flink job.

[0006] However, when analyzing the resource consumption pattern of Flink jobs based on the K-prototypes algorithm, there are still two challenges:

[0007] (1) Attribute weight adjustment: In the K-prototypes algorithm, there is a lack of the ability to adjust attribute weights. Although attribute weights can be set and adjusted by users based on their domain expertise and experience, this approach is subjective and may face challenges of inaccuracy.

[0008] (2) Initial center selection: Another challenge of the K-prototypes algorithm is the selection of the initial center. If the initial center is not selected properly, the clustering result may fall into the local optimum rather than the global optimum.

[0009] Therefore, in response to the above challenges, it is necessary to improve the traditional K-prototypes algorithm and optimize the attribute weights and initial centers to obtain better clustering results, thereby obtaining more accurate analysis results of Flink job resource consumption patterns and providing a more reliable basis for more refined resource allocation. Summary of the invention

[0010] In view of the deficiencies of the prior art, the present invention proposes a method for analyzing job resource consumption patterns based on BWO and clustering algorithm.

[0011] In this method, the present invention proposes a BWO / K-prototypes algorithm based on the White Whale optimization algorithm and the K-prototypes algorithm. By calculating the fitness function, the attribute weights and the initial center are gradually iterated and optimized, and the resource consumption pattern of the Flink job is analyzed based on the BWO / K-prototypes algorithm to reveal the resource consumption pattern of each job.

[0012] Terminology explanation:

[0013] 1. Apache Flink: Apache Flink is a framework and distributed processing engine for stateful computing on batch and streaming data. A Flink cluster is mainly composed of a Jobmanager node and multiple Taskmanager nodes. The Jobmanager, also known as the Master, is responsible for obtaining the Flink jobs to be run, converting the job processing into tasks, and dispatching them to the Taskmanager for execution; while the Taskmanager, also known as the Worker, is mainly responsible for executing the tasks scheduled by the Jobmanager.

[0014] 2. K-prototypes algorithm: K-prototypes algorithm is a traditional clustering algorithm, which combines the k-means algorithm for processing numerical attributes and the k-modes algorithm for processing categorical attributes, so that it can effectively process data sets containing numerical and categorical attributes.

[0015] 3. White Whale Optimization Algorithm: The White Whale Optimization Algorithm (BWO) is a meta-heuristic algorithm. The design of this algorithm is inspired by the behavioral characteristics of white whales. It simulates the swimming, hunting and falling behaviors of white whales to perform iterative search, thereby exploring the optimal solution in the optimization problem.

[0016] 4. Flink Metrics Reporter: Flink Metrics Reporter is a component in Apache Flink used to report and expose metrics data. It can help users monitor Flink clusters in real time. At the same time, users can configure it through the Flink configuration file (conf / flink-conf.yaml).

[0017] 5. Flink Dashboard: Flink Dashboard is a graphical user interface provided by Apache Flink, which can help users manage and monitor Flink clusters in an effective and intuitive way. Under the default configuration, the port number of Flink Dashborad is 8081.

[0018] 6. Worker node ID: In a Flink cluster, each worker node has a unique identifier, which is called the worker node ID. Its main function is to distinguish the worker nodes in the cluster.

[0019] The technical solution of the present invention is:

[0020] A method for analyzing job resource consumption patterns based on BWO and clustering algorithm, comprising:

[0021] (1) Job running and task scheduling: A task scheduling strategy is implemented for the Flink cluster. In this task scheduling strategy, all tasks belonging to the same Flink job are centrally scheduled to the same working node for execution. Therefore, during the monitoring period, each working node will only execute tasks belonging to the same Flink job, and will not mix and execute tasks belonging to different Flink jobs.

[0022] (2) Monitoring of working node resource indicators; working node resource indicators include dynamic resource indicators and static resource indicators;

[0023] (3) Data preprocessing and dataset generation;

[0024] (4) Run the BWO / K-prototypes algorithm based on the White Whale optimization algorithm and the K-prototypes algorithm; including:

[0025] Initialize the beluga population;

[0026] K-prototypes algorithm clustering;

[0027] Set the fitness function; for the data set X, the fitness function includes two indicators, namely the purity of the classification attribute and the compactness of the numerical attribute;

[0028] Through the White Whale optimization algorithm, the population is continuously updated, and the attribute weights and initial centers are optimized and adjusted;

[0029] (5) Application of the optimal solution of the BWO / K-prototypes algorithm; including: clustering the data set X using the K-prototypes algorithm, and the obtained clustering results will be used for subsequent clustering result analysis. Among them, the attribute weights and initial centers of this clustering are set according to the optimal solution of the BWO / K-prototypes algorithm;

[0030] (6) Clustering result analysis;

[0031] (7) Obtain and save the resource consumption pattern analysis results of Flink jobs.

[0032] Preferably, according to the present invention, in step (2), the working node resource indicator monitoring includes:

[0033] During the monitoring period, the monitoring and parsing module periodically collects and parses the indicators exposed by Flink MetricsReporter to obtain the two dynamic resource indicators of memory usage and CPU usage of each working node, and saves the dynamic resource indicators in real time.

[0034] When the Flink cluster is started, the two static resource indicators of Flink total memory and CPU core number of each working node are obtained through the Flink Dashboard and saved.

[0035] According to a preferred embodiment of the present invention, in step (3), data preprocessing and generating a data set include:

[0036] (3-1) Extract all data records within the monitoring period to form data set X;

[0037] (3-2) For each data record, calculate the memory usage; as shown in formula (1):

[0038]

[0039] in, is the data point X i The value of memory usage in is the data point X iThe corresponding data record stores the memory usage value. FM is the static resource indicator Flink total memory of the working node.

[0040] (3-3) The CPU usage value stored in each data record is used as the value of the CPU usage value attribute of each data point;

[0041] (3-4) According to the task scheduling strategy described in step (1), the working node ID is replaced with the Flink job classification identifier to identify each data record, wherein the Flink job classification identifier is used to distinguish each Flink job;

[0042] The Flink job classification identifier used to identify each data record will be used as the classification attribute of the dataset X. At the same time, for the classification attribute values, the values ​​with the same meaning are converted to the same integer;

[0043] (3-5) The data set X includes two numerical attributes, memory usage and CPU usage, and a classification attribute, the Flink job classification identifier; and the data set X includes n data points, which are clustered into a cluster set CL containing K clusters; as shown in formula (2) and formula (3):

[0044] X={X 1 ,X 2 ,…,X i-1 ,X i},where i∈n(2)

[0045] CL={CL 1 ,CL 2 ,…,CL k-1 ,CL k},where k∈K(3)

[0046] Preferably, according to the present invention, the beluga whale population is initialized; comprising:

[0047] At the iterative starting point of the Beluga optimization algorithm, attribute weights and initial centers are randomly generated to initialize the Beluga population; attribute weights are represented by W, as shown in formula (4); initial centers are represented by ICL, as shown in formula (5), and each initial center contains an attribute; the Beluga population is represented by x, and O represents the number of Beluga in the population, as shown in formula (6);

[0048]

[0049]

[0050] x={x 1 ,x 2 ,…,x i-1 ,x i},where i∈O(6)

[0051] Among them, m represents the number of attributes in the data set X, x i represents the i-th beluga whale in the beluga whale population, represents the beluga individual x i The weight of the numerical attribute in , represents the beluga individual x i The weight of the categorical attributes in represents the beluga individual x i The initial center of the jth attribute of the kth cluster in .

[0052] Preferably, the K-prototypes algorithm clustering according to the present invention comprises:

[0053] The distance between each data point and each cluster center is calculated. The distance metric combines the Euclidean distance of numerical attributes and the Hamming distance of categorical attributes; as shown in formula (7) and formula (8):

[0054]

[0055]

[0056] Among them, m num Indicates the number of numerical attributes in the data set X, m cat Indicates the number of classification attributes in the data set X, d(X i ,CL k ) represents the data point X i The distance metric to the center of cluster k, W num represents the weight of the numerical attribute, d num (X i ,CL k ) represents the Euclidean distance of the numerical attribute, W cat represents the weight of the classification attribute, d cat (X i ,CL k ) represents the Hamming distance of the categorical attribute, X i,j Represents data point X i The value of the jth attribute, CL k,j represents the central value of the jth attribute of cluster k;

[0057] Assign each data point to the cluster with the minimum distance metric;

[0058] Based on the data points assigned to the same cluster, the mean of the numerical attribute and the highest frequency value of the categorical attribute are updated as the center of the cluster;

[0059] In one iteration, if no data point moves between clusters, the K-prototypes algorithm clustering ends; otherwise, the K-prototypes algorithm clustering process is repeated.

[0060] According to the preferred embodiment of the present invention, the purity calculation of the classification attribute is as shown in formula (9), formula (10) and formula (11):

[0061]

[0062] R max =max(R k,c )(10)

[0063]

[0064] Among them, Purity is the purity of the classification attribute, R max YesR k,c The maximum value in nCL k is the number of data points in cluster k, R k,c is the proportion of data points belonging to category c in cluster k, nCL k,c is the number of data points in cluster k that belong to category c, where c is the category of the categorical attribute;

[0065] The closeness of a numerical attribute is represented by the silhouette coefficient, which is calculated as shown in formula (12) and formula (13):

[0066]

[0067]

[0068] Among them, s(X i ) is the data point X i The silhouette coefficient, a(X i ) is the data point X i The average distance to all other data points in the same cluster, b(X i ) is the data point X i The average distance to all data points in the nearest cluster; the silhouette coefficient S of the clustering result is the average of the silhouette coefficients of all data points, and the value range is [-1,1];

[0069] The fitness function is defined as the weighted sum of the purity of the categorical attributes and the compactness of the numerical attributes, as shown in formula (14):

[0070] Fitnessfunction=α 1 Purity+α 2 S(14)

[0071] Among them, Fitnessfunction is the fitness function, α 1 and α 2 is the weight, and α 1 +α 2 =1.

[0072] According to the preferred embodiment of the present invention, the population is continuously updated, and the attribute weights and the initial center are optimized and adjusted by the White Whale optimization algorithm; including:

[0073] In each iteration of the White Whale optimization algorithm,

[0074] First, calculate the balance factor and whale fall probability;

[0075] Then, in each iteration of the search process, the social behavior of beluga whales in nature is imitated, which includes three stages: exploration, exploitation, and whale fall;

[0076] Finally, update the solution set and the optimal solution; for each beluga in the population, if the new solution The fitness function value is x i The fitness function value of x is high, then i quilt Replacement, x i The fitness function value is also The fitness function value is replaced by, where x i Depend on and composition, Depend on and If the optimal fitness function value found in this iteration is higher than the known global optimal fitness function value, the known global optimal solution x is updated. best and the corresponding fitness function value, this iteration ends; otherwise, continue to keep the known global optimal solution x best And the corresponding fitness function value, this iteration ends;

[0077] The White Whale optimization algorithm iterates until the total number of iterations is reached and returns the optimal solution.

[0078] Further preferred, the White Whale optimization algorithm uses the balance factor B f transition from the exploration phase to the development phase;

[0079] When the balance factor B f >0.5, it enters the exploration stage; when the balance factor B f When ≤0.5, it enters the development stage. At the same time, the probability of entering the development stage gradually increases with the increase of the number of iterations; the balance factor B f As shown in formula (15):

[0080] Bf =B o (1-T / 2T max )(15)

[0081] Among them, T represents the current iteration, T max represents the total number of iterations, B o is a random number between (0,1);

[0082] Whale fall probability W f It decreases linearly with the increase of the number of iterations. The amount of reduction in each iteration is proportional to the number of iterations. When the balance factor B f <W f When , it enters the whale fall stage; otherwise, it skips this stage; the whale fall probability W f As shown in formula (16):

[0083] W f =0.1-0.05T / T max (16)

[0084] Further preferably, in the exploration stage, the white whale optimization algorithm imitates the swimming behavior of the white whale, and they swim in a synchronous or mirrored manner. In the exploration stage, the attribute weight and the initial center are updated as shown in formula (17) and formula (18):

[0085]

[0086]

[0087] where j∈m,di=m(k-1)+j

[0088] Where T represents the current iteration, and represents the beluga individual x i The new weight of attribute y and the white whale individual x i The new initial center of the jth attribute of the kth cluster in , and represents the beluga individual x i The current weight of attribute y and the beluga individual x i The current initial center of the jth attribute of the kth cluster in ; and Both represent random white whale x r The variable value of the first dimension in the variable, the total dimension of the variable is mK+2; r 1 ,r 2 is a random number between (0,1), di is used to determine the choice of sin(2πr 2 ) or cos(2πr 2 );

[0089] In the development phase, the White Whale Optimization Algorithm imitates the predation behavior of white whales and focuses on the known global optimal solution x best At the same time, the Levy flight strategy is introduced in the development stage to enhance the convergence ability of the algorithm; in the development stage, the attribute weights and initial center updates are shown in formulas (19)-(23):

[0090]

[0091]

[0092] C 1 =2r 4 (1-T / T max )(twenty one)

[0093]

[0094]

[0095] Where T represents the current iteration, and represents the beluga individual x i The new weights of the attributes and the white whale individual x i The new initial centers of the K clusters in and represents the beluga individual x i The current weights of the attributes and the beluga individual x i The current initial centers of the K clusters in x; r is a beluga whale selected at random, and composition, and represents a random white whale x r The current weight of the attribute and the random white whale x r The current initial centers of the K clusters in x; best is a known global optimal solution, given by and composition, and Indicates that the global optimal solution x is known best The current weights of the attributes and the known global optimal solution x best The current initial centers of the K clusters in r 3 ,r 4 is a random number between (0,1), C 1 is the adaptive factor measuring the intensity of Levy flight, L F is the Levy flight function, u and v are normally distributed random numbers, β is a default constant equal to 1.5; σ is the scale parameter;

[0096] In the whale fall phase, the attribute weights and initial center updates are shown in formulas (24)-(28):

[0097]

[0098] W step =(u b -l b )exp(-C 2 T / T max )(25)

[0099]

[0100] ICL step =(u b -l b )exp(-C 2 T / T max )(27)

[0101] C 2 =2W f ×O(28)

[0102] Where T represents the current iteration, and represents the beluga individual x i The new weights of the attributes and the white whale individual x i The new initial centers of the K clusters in r 5 ,r 6 and r 7 is a random number between (0,1); W step and ICL step Both represent the whale fall step length, u b and l b are the upper and lower limits of the variable value range in Beluga, C 2 is a step factor related to whale fall probability and population size, W f is the probability of whale fall.

[0103] Preferably, in step (6), clustering result analysis includes:

[0104] Set thresholds H and N, and resource consumption score L; at the same time, according to the thresholds H and N, divide the resource consumption score into three levels: L1, L2 and L3;

[0105] Based on the clustering results of the K-prototypes algorithm, identify the cluster in which each classification category appears most frequently, and then use the most frequently appearing cluster to analyze the resource consumption pattern of the Flink job corresponding to the category;

[0106] According to the cluster center, the analysis is first performed. The value of the numerical attribute memory usage is compared with the threshold. If it is greater than the threshold H, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L3. If it is greater than the threshold N and less than the threshold H, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L2. If it is less than the threshold N, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L1. Then, the value of the numerical attribute CPU usage is compared with the threshold. If it is greater than the threshold H, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L3. If it is greater than the threshold N and less than the threshold H, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L2. If it is less than the threshold N, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L1.

[0107] Preferably, in step (7), the resource consumption pattern analysis result of the Flink job is obtained and saved; including:

[0108] Through step (6), the resource consumption pattern analysis results of each Flink job are obtained; at the same time, the resource consumption pattern analysis results of the Flink job are saved in a specific table of the MySQL database, and the specific table includes three columns: "Flink job classification identifier", "memory resource consumption score" and "CPU resource consumption score".

[0109] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0110] 1. Refined resource allocation: Through in-depth analysis of the resource consumption pattern of Flink jobs, when running Flink jobs in the future, we can optimize the resource allocation of the working nodes in the cluster based on the analysis results of the resource consumption pattern of Flink jobs, so that tasks can be executed on working nodes that better meet their resource requirements, improving the matching degree between tasks and working node resources. This can not only improve the execution efficiency of the Flink cluster and avoid exceptions; it can also reduce resource usage and avoid unnecessary resource waste.

[0111] 2. Based on the White Whale Optimization Algorithm and K-prototypes Algorithm: The BWO / K-prototypes algorithm is designed to optimize the attribute weights and initial centers step by step through iterative optimization by calculating the fitness function. Compared with the traditional K-prototypes algorithm, the present invention can effectively solve the challenges of subjectivity and inaccuracy in attribute weight adjustment. At the same time, the present invention optimizes the initial center, effectively avoiding the problem of clustering results falling into local optimality due to improper selection of the initial center.

[0112] 3. Fitness function design: In the BWO / K-prototypes algorithm, the purity of classification attributes and the compactness of numerical attributes are combined as the fitness function. The main idea is that the higher the fitness function value, the better the clustering result, and the more accurate the analysis result of the resource consumption pattern of the Flink job, thus providing a more reliable basis for more refined resource allocation.

[0113] 4. Wide applicability: The resource consumption pattern analysis results of Flink jobs are not only applicable to the refined resource configuration of working nodes in the Flink cluster, but can also be extended to the refined task scheduling of the Flink cluster. At the same time, the BWO / K-prototypes algorithm is not only applicable to the resource consumption pattern analysis of Flink jobs, but can also be extended to clustering tasks in other fields, such as image analysis, bioinformatics, machine learning, and pattern recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 The flowchart of the job resource consumption pattern analysis method based on BWO and clustering algorithm is shown in the figure.

[0115] Figure 2 This is a diagram of the work node resource indicator monitoring architecture.

[0116] Figure 3 Schematic diagram of the BWO / K-prototypes algorithm research framework.

[0117] Figure 4 Schematic diagram of the K-prototypes algorithm clustering process flow.

[0118] Figure 5 Schematic diagram of the fitness function calculation process.

[0119] Figure 6 Schematic diagram of the optimal solution application. DETAILED DESCRIPTION

[0120] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0121] Example 1

[0122] A method for analyzing job resource consumption patterns based on BWO and clustering algorithms, such as Figure 1 As shown, including:

[0123] (1) Job running and task scheduling: A task scheduling strategy is implemented for the Flink cluster. In this task scheduling strategy, all tasks belonging to the same Flink job are centrally scheduled to the same working node for execution. Therefore, during the monitoring period, each working node will only execute tasks belonging to the same Flink job, and will not mix and execute tasks belonging to different Flink jobs.

[0124] (2) Monitoring of working node resource indicators; working node resource indicators include dynamic resource indicators and static resource indicators;

[0125] (3) Data preprocessing and dataset generation;

[0126] (4) Run the BWO / K-prototypes algorithm based on the White Whale optimization algorithm and the K-prototypes algorithm; including:

[0127] Initialize the beluga population;

[0128] K-prototypes algorithm clustering;

[0129] Set the fitness function; for the data set X, the fitness function includes two indicators, namely the purity of the classification attribute and the compactness of the numerical attribute;

[0130] Through the White Whale optimization algorithm, the population is continuously updated, and the attribute weights and initial centers are optimized and adjusted;

[0131] (5) Application of the optimal solution of the BWO / K-prototypes algorithm; including: clustering the data set X using the K-prototypes algorithm, and the obtained clustering results will be used for subsequent clustering result analysis. Among them, the attribute weights and initial centers of this clustering are set according to the optimal solution of the BWO / K-prototypes algorithm;

[0132] (6) Clustering result analysis;

[0133] (7) Obtain and save the resource consumption pattern analysis results of Flink jobs.

[0134] Example 2

[0135] The method for analyzing the job resource consumption pattern based on BWO and clustering algorithm described in Example 1 is different in that:

[0136] In step (2), the working node resource indicators are monitored; Figure 2 As shown, including:

[0137] During the monitoring period, the monitoring and parsing module periodically collects and parses the indicators disclosed by Flink MetricsReporter, and obtains the two dynamic resource indicators of memory usage and CPU usage of each working node, and saves the above dynamic resource indicators in real time to Redis or other suitable databases. In the Redis database, each data record is stored using a hash structure, which contains two fields: memory usage and CPU usage, and the key of each data record consists of the working node ID and the collection timestamp, so that each data record uses the key to clearly identify the working node ID and collection timestamp of its source.

[0138] At the same time, when the Flink cluster is started, you can obtain the two static resource indicators of Flink total memory and CPU core number of each working node through Flink Dashboard, and save the above static resource indicators to Redis or other suitable databases. In the Redis database, each data record is stored using a hash structure, which contains two fields: Flink total memory and CPU core number, and the key of each data record consists of the static resource indicator identifier and the working node ID.

[0139] In step (3), data preprocessing and data set generation include:

[0140] (3-1) Extract all data records within the monitoring period from the Redis database to form data set X;

[0141] (3-2) For each data record, calculate the memory usage; as shown in formula (1), use the result as the value of the memory usage of the numerical attribute of each data point.

[0142]

[0143] in, is the data point X i The value of memory usage in is the data point X i The corresponding data record stores the memory usage value. FM is the static resource indicator Flink total memory of the working node.

[0144] (3-3) The CPU usage value stored in each data record is used as the value of the CPU usage value attribute of each data point;

[0145] (3-4) According to the task scheduling strategy described in step (1), the Flink job classification identifier can be used to replace the work node ID to identify each data record, wherein the Flink job classification identifier is used to distinguish each Flink job. For example, if all tasks executed on the work node with the work node ID 192.168.136.11:46477-d2a2d3 belong to Flink job 1, then all data records identified by the work node ID can be identified by Flink job 1; if all tasks executed on the work node with the work node ID 192.168.136.12:34685-451322 belong to Flink job 2, then all data records identified by the work node ID can be identified by Flink job 2.

[0146] The Flink job classification identifier used to identify each data record will be used as the classification attribute of the dataset X. At the same time, for the classification attribute value, the values ​​with the same meaning are converted to the same integer. For example, Flink job 1 is converted to 1, and Flink job 2 is converted to 2;

[0147] (3-5) The data set X includes two numerical attributes, memory usage and CPU usage, and a classification attribute, the Flink job classification identifier; and the data set X includes n data points, which are clustered into a cluster set CL containing K clusters; as shown in formula (2) and formula (3):

[0148] X={X 1 ,X 2 ,…,X i-1 ,X i},where i∈n(2)

[0149] CL={CL 1 ,CL 2 ,…,CL k-1 ,CL k},where k∈K(3)

[0150] like Figure 3 As shown, initialize the beluga population; including:

[0151] The present invention applies the white whale optimization algorithm to optimize the attribute weights and initial centers of the K-prototypes algorithm. At the iterative starting point of the white whale optimization algorithm, attribute weights and initial centers are randomly generated to initialize the white whale population; the attribute weights are represented by W, as shown in formula (4); the initial centers are represented by ICL, as shown in formula (5), and each initial center contains an attribute; the white whale population is represented by x, and O represents the number of white whales in the population, as shown in formula (6);

[0152]

[0153]

[0154] x={x 1 ,x 2 ,…,x i-1 ,x i},where i∈O(6)

[0155] Among them, m represents the number of attributes in the data set X, x i represents the i-th beluga whale in the beluga whale population, represents the beluga individual x i The weight of the numerical attribute in , represents the beluga individual x i The weight of the categorical attributes in represents the beluga individual x i The initial center of the jth attribute of the kth cluster in .

[0156] K-prototypes algorithm clustering; such as Figure 4 Shown; including:

[0157] After the beluga population is initialized, a beluga in the population is a solution. The K-prototypes algorithm will apply each solution to the data set X for clustering. The clustering process is described as follows:

[0158] The distance between each data point and each cluster center is calculated. The distance metric combines the Euclidean distance of numerical attributes and the Hamming distance of categorical attributes; as shown in formula (7) and formula (8):

[0159]

[0160]

[0161] Among them, m num Indicates the number of numerical attributes in the data set X, m cat Indicates the number of classification attributes in the data set X, d(X i ,CL k ) represents the data point X i The distance metric to the center of cluster k, W num represents the weight of the numerical attribute, d num (X i ,CL k ) represents the Euclidean distance of the numerical attribute, W cat represents the weight of the classification attribute, d cat (X i ,CL k ) represents the Hamming distance of the categorical attribute, X i,j Represents data point Xi The value of the jth attribute, CL k,j represents the central value of the jth attribute of cluster k;

[0162] Assign each data point to the cluster with the minimum distance metric;

[0163] Based on the data points assigned to the same cluster, the mean of the numerical attribute and the highest frequency value of the categorical attribute are updated as the center of the cluster;

[0164] In one iteration, if no data point moves between clusters, clustering ends; otherwise, the above steps are repeated.

[0165] The purity calculation of classification attributes is shown in formula (9), formula (10) and formula (11):

[0166]

[0167] R max =max(R k,c )(10)

[0168]

[0169] Among them, Purity is the purity of the classification attribute, R max YesR k,c The maximum value in nCL k is the number of data points in cluster k, R k,c is the proportion of data points belonging to category c in cluster k, nCL k,c is the number of data points in cluster k that belong to category c, where c is the category of the categorical attribute; the higher the purity value, the higher the consistency of the categorical attributes of the data points in the cluster;

[0170] The closeness of a numerical attribute is represented by the silhouette coefficient, which is calculated as shown in formula (12) and formula (13):

[0171]

[0172] Among them, s(X i ) is the data point X i The silhouette coefficient, a(X i ) is the data point X i The average distance to all other data points in the same cluster, b(X i ) is the data point X iThe average distance to all data points in the nearest cluster; the silhouette coefficient S of the clustering result is the average of the silhouette coefficients of all data points, and its value range is [-1,1]; the closer the silhouette coefficient S is to 1, the higher the compactness of the data points in the cluster in terms of numerical attributes, and the greater the separation of the data points between clusters in terms of numerical attributes.

[0173] like Figure 5 As shown, the fitness function is defined as the weighted sum of the purity of the categorical attributes and the compactness of the numerical attributes, as shown in formula (14):

[0174] Fitnessfunction=α 1 Purity+α 2 S(14)

[0175] Among them, Fitnessfunction is the fitness function, α 1 and α 2 is the weight, and α 1 +α 2 = 1. The higher the fitness function value, the better the solution.

[0176] Through the White Whale optimization algorithm, the population is continuously updated, and the attribute weights and initial centers are optimized and adjusted; including:

[0177] In each iteration of the White Whale optimization algorithm,

[0178] First, calculate the balance factor and whale fall probability;

[0179] Then, in each iteration of the search process, the social behavior of beluga whales in nature is imitated, which includes three stages: exploration, exploitation, and whale fall;

[0180] Finally, update the solution set and the optimal solution; for each beluga in the population, if the new solution The fitness function value is x i The fitness function value of x is high, then i quilt Replacement, x i The fitness function value is also The fitness function value is replaced by, where x i Depend on and composition, Depend on and If the optimal fitness function value found in this iteration is higher than the known global optimal fitness function value, the known global optimal solution x is updated. best and the corresponding fitness function value, this iteration ends; otherwise, continue to keep the known global optimal solution x bestAnd the corresponding fitness function value, this iteration ends;

[0181] The White Whale optimization algorithm iterates until the total number of iterations is reached and returns the optimal solution.

[0182] The White Whale optimization algorithm uses the balance factor B f transition from the exploration phase to the development phase;

[0183] When the balance factor B f >0.5, it enters the exploration stage; when the balance factor B f When ≤0.5, it enters the development stage. At the same time, the probability of entering the development stage gradually increases with the increase of the number of iterations; the balance factor B f As shown in formula (15):

[0184] B f =B o (1-T / 2T max )(15)

[0185] Among them, T represents the current iteration, T max represents the total number of iterations, B o is a random number between (0,1);

[0186] Whale fall probability W f It decreases linearly with the increase of the number of iterations. The amount of reduction in each iteration is proportional to the number of iterations. When the balance factor B f <W f When , it enters the whale fall stage; otherwise, it skips this stage; the whale fall probability W f As shown in formula (16):

[0187] W f =0.1-0.05T / T max (16)

[0188] In the exploration phase, the White Whale Optimization Algorithm imitates the swimming behavior of white whales. They swim in a synchronous or mirrored manner. In the exploration phase, the attribute weights and initial center updates are shown in formulas (17) and (18):

[0189]

[0190]

[0191] where j∈m,di=m(k-1)+j

[0192] Where T represents the current iteration, and represents the beluga individual x i The new weight of attribute y and the white whale individual xi The new initial center of the jth attribute of the kth cluster in , and represents the beluga individual x i The current weight of attribute y and the beluga individual x i The current initial center of the jth attribute of the kth cluster in ; and Both represent random white whale x r The variable value of the first dimension in the variable, the total dimension of the variable is mK+2; r 1 ,r 2 is a random number between (0,1), di is used to determine the choice of sin(2πr 2 ) or cos(2πr 2 );

[0193] In the development phase, the White Whale Optimization Algorithm imitates the predation behavior of white whales and focuses on the known global optimal solution x best At the same time, the Levy flight strategy is introduced in the development stage to enhance the convergence ability of the algorithm; in the development stage, the attribute weights and initial center updates are shown in formulas (19)-(23):

[0194]

[0195]

[0196] C 1 =2r 4 (1-T / T max )(twenty one)

[0197]

[0198]

[0199] Where T represents the current iteration, and represents the beluga individual x i The new weights of the attributes and the white whale individual x i The new initial centers of the K clusters in and represents the beluga individual x i The current weights of the attributes and the beluga individual x i The current initial centers of the K clusters in x; r is a beluga whale selected at random, and composition, and represents a random white whale x r The current weight of the attribute and the random white whale xr The current initial centers of the K clusters in x; best is a known global optimal solution, given by and composition, and Indicates that the global optimal solution x is known best The current weights of the attributes and the known global optimal solution x best The current initial centers of the K clusters in r 3 ,r 4 is a random number between (0,1), C 1 is the adaptive factor measuring the intensity of Levy flight, L F is the Levy flight function, u and v are normally distributed random numbers, β is a default constant equal to 1.5; σ is the scale parameter;

[0200] In the whale fall stage, the whale fall stage is a strategy for the algorithm to jump out of the local optimum. In the whale fall stage, the attribute weights and initial center updates are shown in formulas (24)-(28):

[0201]

[0202] W step =(u b -l b )exp(-C 2 T / T max )(25)

[0203]

[0204] ICL step =(u b -l b )exp(-C 2 T / T max )(27)

[0205] C 2 =2W f ×O(28)

[0206] Where T represents the current iteration, and represents the beluga individual x i The new weights of the attributes and the white whale individual x i The new initial centers of the K clusters in r 5 ,r 6 and r 7 is a random number between (0,1); W step and ICL step Both represent the whale fall step length, u b and l bare the upper and lower limits of the variable value range in Beluga, C 2 is a step factor related to whale fall probability and population size, W f is the probability of whale fall.

[0207] The optimal solution is applied as Figure 6 shown.

[0208] In step (6), clustering result analysis includes:

[0209] Set thresholds H and N, and resource consumption score L; at the same time, according to the thresholds H and N, divide the resource consumption score into three levels: L1, L2 and L3;

[0210] Table 1 Flink job resource consumption pattern analysis parameter setting table; Flink job resource consumption pattern analysis parameter settings are shown in Table 1:

[0211]

[0212] Based on the clustering results of the K-prototypes algorithm, identify the cluster in which each classification category appears most frequently, and then use the most frequently appearing cluster to analyze the resource consumption pattern of the Flink job corresponding to the category;

[0213] According to the cluster center, the analysis is first performed. The value of the numerical attribute memory usage is compared with the threshold. If it is greater than the threshold H, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L3. If it is greater than the threshold N and less than the threshold H, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L2. If it is less than the threshold N, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L1. Then, the value of the numerical attribute CPU usage is compared with the threshold. If it is greater than the threshold H, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L3. If it is greater than the threshold N and less than the threshold H, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L2. If it is less than the threshold N, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L1.

[0214] In step (7), the resource consumption pattern analysis results of the Flink job are obtained and saved; including:

[0215] Through step (6), the resource consumption pattern analysis results of each Flink job can be obtained. At the same time, the resource consumption pattern analysis results of the Flink job are saved in a specific table in the MySQL database, which contains three columns: "Flink job classification identification", "memory resource consumption score" and "CPU resource consumption score". For example, if the resource consumption pattern analysis result of a Flink job is a memory resource consumption score of L1 and a CPU resource consumption score of L2, then when running the job in the cluster in the future, you can consider reducing the memory resource configuration of the working node to avoid resource waste due to excessive resource allocation.

[0216] Table 2 Flink job resource consumption pattern analysis results; The Flink job resource consumption pattern analysis results are shown in Table 2:

[0217]

[0218] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for analyzing job resource consumption patterns based on BWO and clustering algorithm, characterized in that: include: (1) Job running and task scheduling: Implement a task scheduling strategy for the Flink cluster. In this task scheduling strategy, all tasks belonging to the same Flink job are centrally scheduled to the same working node for execution; (2) Monitoring of working node resource indicators; working node resource indicators include dynamic resource indicators and static resource indicators; (3) Data preprocessing and dataset generation; (4) Run the BWO / K-prototypes algorithm based on the White Whale optimization algorithm and the K-prototypes algorithm; including: Initialize the beluga population; K-prototypes algorithm clustering; Set the fitness function; for the data set X, the fitness function includes two indicators, namely the purity of the classification attribute and the compactness of the numerical attribute; Through the White Whale optimization algorithm, the population is continuously updated, and the attribute weights and initial centers are optimized and adjusted; (5) Application of the optimal solution of the BWO / K-prototypes algorithm; including: clustering the data set X using the K-prototypes algorithm, wherein the attribute weights and initial centers of this clustering are set according to the optimal solution of the BWO / K-prototypes algorithm; (6) Clustering result analysis; (7) Obtaining and saving the analysis results of resource consumption patterns of Flink jobs; In step (2), the working node resource indicator monitoring includes: During the monitoring period, the monitoring and parsing module periodically collects and parses the indicators disclosed by Flink Metrics Reporter to obtain the two dynamic resource indicators of memory usage and CPU usage of each working node, and saves the dynamic resource indicators in real time; In step (3), data preprocessing and data set generation include: (3-1) Extract all data records within the monitoring period to form data set X; (3-2) For each data record, calculate the memory usage; as shown in formula (1): in, is the data point X i The value of memory usage in is the data point X i The corresponding data record stores the memory usage value. FM is the static resource indicator Flink total memory of the working node. (3-3) The CPU usage value stored in each data record is used as the value of the CPU usage value attribute of each data point; (3-4) According to the task scheduling strategy described in step (1), the working node ID is replaced with the Flink job classification identifier to identify each data record, wherein the Flink job classification identifier is used to distinguish each Flink job; The Flink job classification identifier used to identify each data record will be used as the classification attribute of the dataset X. At the same time, for the classification attribute values, the values ​​with the same meaning are converted to the same integer; (3-5) The data set X includes two numerical attributes, memory usage and CPU usage, and a classification attribute, the Flink job classification identifier; and the data set X includes n data points, which are clustered into a cluster set CL containing K clusters; as shown in formula (2) and formula (3): X={X1,X2,…,X i-1 ,X i },where i∈n (2) CL={CL1,CL2,…,CL k-1 ,CL k },where k∈K (3); In step (6), clustering result analysis includes: Set thresholds H and N, and resource consumption score L; at the same time, according to the thresholds H and N, divide the resource consumption score into three levels: L1, L2 and L3; Based on the clustering results of the K-prototypes algorithm, identify the cluster in which each classification category appears most frequently, and then use the most frequently appearing cluster to analyze the resource consumption pattern of the Flink job corresponding to the category; According to the cluster center, the analysis is first performed. The value of the numerical attribute memory usage is compared with the threshold. If it is greater than the threshold H, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L3. If it is greater than the threshold N and less than the threshold H, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L2. If it is less than the threshold N, the memory resource consumption score of the Flink job analyzed using the cluster is considered to be L1. Then, the value of the numerical attribute CPU usage is compared with the threshold. If it is greater than the threshold H, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L3. If it is greater than the threshold N and less than the threshold H, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L2. If it is less than the threshold N, the CPU resource consumption score of the Flink job analyzed using the cluster is considered to be L1.

2. The method for analyzing job resource consumption patterns based on BWO and clustering algorithm according to claim 1, characterized in that: Initialize the beluga population; including: At the iterative starting point of the Beluga optimization algorithm, attribute weights and initial centers are randomly generated to initialize the Beluga population; attribute weights are represented by W, as shown in formula (4); initial centers are represented by ICL, as shown in formula (5), and each initial center contains an attribute; the Beluga population is represented by x, and O represents the number of Beluga in the population, as shown in formula (6); x={x1,x2,…,x i-1 ,x i },where i∈O(6) Among them, m represents the number of attributes in the data set X, x i represents the i-th beluga whale in the beluga whale population, represents the beluga individual x i The weight of the numerical attribute in , represents the beluga individual x i The weight of the categorical attributes in , represents the beluga individual x i The initial center of the jth attribute of the kth cluster in .

3. The method for analyzing job resource consumption patterns based on BWO and clustering algorithm according to claim 1, characterized in that: K-prototypes algorithm clustering; including: The distance between each data point and each cluster center is calculated. The distance metric combines the Euclidean distance of numerical attributes and the Hamming distance of categorical attributes; as shown in formula (7) and formula (8): Among them, m num Indicates the number of numerical attributes in the data set X, m cat Indicates the number of classification attributes in the data set X, d(X i ,CL k ) represents the data point X i The distance metric to the center of cluster k, W num represents the weight of the numerical attribute, d num (X i ,CL k ) represents the Euclidean distance of the numerical attribute, W cat represents the weight of the classification attribute, d cat (X i ,CL k ) represents the Hamming distance of the categorical attribute, X i,j Represents data point X i The value of the jth attribute, CL k,j represents the central value of the jth attribute of cluster k; Assign each data point to the cluster with the minimum distance metric; Based on the data points assigned to the same cluster, the mean of the numerical attribute and the highest frequency value of the categorical attribute are updated as the center of the cluster; In one iteration, if no data point moves between clusters, the K-prototypes algorithm clustering ends; otherwise, the K-prototypes algorithm clustering process is repeated.

4. The method for analyzing job resource consumption patterns based on BWO and clustering algorithm according to claim 1, characterized in that: The purity calculation of classification attributes is shown in formula (9), formula (10) and formula (11): R max =max(R k,c ) (10) Among them, Purity is the purity of the classification attribute, R max YesR k,c The maximum value in nCL k is the number of data points in cluster k, R k,c is the proportion of data points belonging to category c in cluster k, nCL k,c is the number of data points in cluster k that belong to category c, where c is the category of the categorical attribute; The closeness of a numerical attribute is represented by the silhouette coefficient, which is calculated as shown in formula (12) and formula (13): Among them, s(X i ) is the data point X i The silhouette coefficient, a(X i ) is the data point X i The average distance to all other data points in the same cluster, b(X i ) is the data point X i The average distance to all data points in the nearest cluster; the silhouette coefficient S of the clustering result is the average of the silhouette coefficients of all data points, and the value range is [-1,1]; The fitness function is defined as the weighted sum of the purity of the categorical attributes and the compactness of the numerical attributes, as shown in formula (14): Fitnessfunction=α1Purity+α2S (14) Among them, Fitnessfunction is the fitness function, α1 and α2 are weights, and α1+α2=1.

5. The method for analyzing job resource consumption patterns based on BWO and clustering algorithm according to claim 1, characterized in that: Through the White Whale optimization algorithm, the population is continuously updated, and the attribute weights and initial centers are optimized and adjusted; including: In each iteration of the White Whale optimization algorithm, First, calculate the balance factor and whale fall probability; Then, in each iteration of the search process, the social behavior of beluga whales in nature is imitated, which includes three stages: exploration, exploitation, and whale fall; Finally, update the solution set and the optimal solution; for each beluga in the population, if the new solution The fitness function value is x i The fitness function value of x is high, then i quilt Replacement, x i The fitness function value is also The fitness function value is replaced by, where x i Depend on and composition, Depend on and If the optimal fitness function value found in this iteration is higher than the known global optimal fitness function value, the known global optimal solution x is updated. best and the corresponding fitness function value, this iteration ends; otherwise, continue to keep the known global optimal solution x best And the corresponding fitness function value, this iteration ends; The White Whale optimization algorithm iterates until the total number of iterations is reached and returns the optimal solution.

6. The method for analyzing job resource consumption patterns based on BWO and clustering algorithm according to claim 5, characterized in that: The White Whale optimization algorithm uses the balance factor B f transition from the exploration phase to the development phase; When the balance factor B f >0.5, it enters the exploration stage; when the balance factor B f When ≤0.5, it enters the development stage. At the same time, the probability of entering the development stage gradually increases with the increase of the number of iterations; the balance factor B f As shown in formula (15): B f =B o (1-T / 2T max )(15) Among them, T represents the current iteration, T max represents the total number of iterations, B o is a random number between (0,1); Whale fall probability W f It decreases linearly with the increase of the number of iterations. The amount of reduction in each iteration is proportional to the number of iterations. When the balance factor B f <W f When , it enters the whale fall stage; otherwise, it skips this stage; the whale fall probability W f As shown in formula (16): W f =0.1-0.05T / T max (16)。 7. The method for analyzing job resource consumption patterns based on BWO and clustering algorithm according to claim 5, characterized in that: In the exploration phase, the White Whale Optimization Algorithm imitates the swimming behavior of white whales. They swim in a synchronous or mirrored manner. In the exploration phase, the attribute weights and initial center updates are shown in formulas (17) and (18): Where T represents the current iteration, and represents the beluga individual x i The new weight of attribute y and the white whale individual x i The new initial center of the jth attribute of the kth cluster in , and represents the beluga individual x i The current weight of attribute y and the beluga individual x i The current initial center of the jth attribute of the kth cluster in ; and Both represent random white whale x r The variable value of the first dimension in , the total dimension of the variable is mK+2; r1, r2 are random numbers between (0, 1), and di is used to decide whether to choose sin(2πr2) or cos(2πr2) in the initial center update formula; In the development phase, the White Whale Optimization Algorithm imitates the predation behavior of white whales and focuses on the known global optimal solution x best At the same time, the Levy flight strategy is introduced in the development stage to enhance the convergence ability of the algorithm; in the development stage, the attribute weights and initial center updates are shown in formulas (19)-(23): C1=2r4(1-T / T max )(21) Where T represents the current iteration, and represents the beluga individual x i The new weights of the attributes and the white whale individual x i The new initial centers of the K clusters in and represents the beluga individual x i The current weights of the attributes and the beluga individual x i The current initial centers of the K clusters in x; r is a beluga whale selected at random, and composition, and represents a random white whale x r The current weight of the attribute and the random white whale x r The current initial centers of the K clusters in x; best is a known global optimal solution, given by and composition, and Indicates that the global optimal solution x is known best The current weights of the attributes and the known global optimal solution x best The current initial centers of the K clusters in the cluster; r3, r4 are random numbers between (0, 1), C1 is an adaptive factor to measure the intensity of Levy flight, L F is the Levy flight function, u and v are normally distributed random numbers, β is a default constant equal to 1.5; σ is the scale parameter; In the whale fall phase, the attribute weights and initial center updates are shown in formulas (24)-(28): W step =(u b -l b )exp(-C2T / T max )(25) ICL step =(u b -l b )exp(-C2T / T max )(27) C2=2W f ×O(28) Where T represents the current iteration, and represents the beluga individual x i The new weights of the attributes and the white whale individual x i The new initial centers of the K clusters in the equation are r5, r6 and r7, which are random numbers between (0, 1); W step and ICL step Both represent the whale fall step length, u b and l b are the upper and lower limits of the range of variables in beluga whales, C2 is a step factor related to the probability of whale fall and population size, and W f is the probability of whale fall.

8. A method for analyzing job resource consumption patterns based on BWO and clustering algorithm according to any one of claims 1 to 7, characterized in that: In step (7), the resource consumption pattern analysis results of the Flink job are obtained and saved; including: The resource consumption pattern analysis results of the Flink job are saved in a specific table of the MySQL database. The specific table includes three columns: "Flink job classification identifier", "memory resource consumption score", and "CPU resource consumption score".

Images