Dynamic adjustment method of YARN cluster node resources based on dual-mode perception

Abstract

The present invention provides a YARN cluster node resource dynamic adjustment method and system based on dual-mode perception, which relates to the field of data processing technology, including collecting system indicator data and application program indicator data, respectively constructing two deep neural network models for fusion evaluation, calculating resource dynamic adjustment coefficients, using a sliding time window mechanism to update node resources in real time, and using a genetic algorithm to construct a multi-dimensional resource allocation optimization model to achieve optimal resource allocation. The present invention can improve cluster resource utilization, reduce resource competition, and realize intelligent dynamic adjustment of node resources.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a YARN cluster node resource dynamic adjustment method based on dual-mode perception. Background Art

[0002] With the rapid development of big data and distributed computing technologies, YARN has been widely used in enterprise-level big data processing systems as a mainstream resource management and scheduling platform. YARN clusters achieve centralized management and flexible allocation of computing resources through a unified resource management and task scheduling mechanism, providing a reliable operating environment for various distributed applications. In traditional YARN cluster resource management, node resource configuration usually adopts a static allocation method, and system administrators need to set fixed resource limits for each node in advance, including resource quotas such as CPU, memory, and network bandwidth.

[0003] However, the existing YARN cluster resource management solutions have the following main problems:

[0004] First, the traditional static resource allocation method cannot effectively adapt to the dynamically changing resource requirements of applications. When the workload fluctuates, it may lead to unreasonable resource allocation, resulting in excessive waste of resources on some nodes and tight resources on other nodes. Due to the lack of accurate perception and prediction capabilities of the real-time resource usage of nodes, it is difficult to achieve on-demand resource allocation and dynamic adjustment.

[0005] Secondly, existing resource management solutions often only focus on single-dimensional resource indicators, such as CPU utilization or memory usage, ignoring the mutual influence and correlation between various resources. This single resource monitoring method cannot fully reflect the actual load status of the node, which can easily cause deviations in resource allocation decisions and affect the overall performance and resource utilization efficiency of the cluster.

[0006] Finally, the current resource allocation strategy lacks an intelligent optimization mechanism, and it is difficult to find the optimal resource allocation solution when dealing with complex multi-dimensional resource allocation problems. Especially in high-concurrency scenarios, simple resource allocation algorithms cannot effectively balance resource competition among multiple applications, which can easily lead to low task execution efficiency and system performance degradation. These problems seriously restrict the application effect of YARN clusters in large-scale production environments. Summary of the invention

[0007] The embodiment of the present invention provides a YARN cluster node resource dynamic adjustment method based on dual-mode perception, which can solve the problems in the prior art.

[0008] According to a first aspect of the embodiments of the present invention,

[0009] Provides a YARN cluster node resource dynamic adjustment method based on dual-mode perception, including:

[0010] Collect system indicator data and application indicator data of YARN cluster nodes, the system indicator data including node CPU utilization, memory usage, network bandwidth occupancy and disk I / O load, and the application indicator data including application execution time, resource application amount, task completion rate and data processing throughput; construct a first deep neural network model based on the system indicator data, and construct a second deep neural network model based on the application indicator data; fuse the output results of the first deep neural network model and the second deep neural network model to obtain a node resource utilization status evaluation result;

[0011] According to the node resource utilization status evaluation result, a resource dynamic adjustment coefficient is calculated for each node in the YARN cluster, and the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient, and a network bandwidth adjustment coefficient; based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit of the allocatable resources of each node in real time; when the node resource utilization rate exceeds the preset utilization rate threshold, a resource expansion operation is triggered, and the node resource quota is increased through an elastic scaling algorithm; when the node resource utilization rate is lower than the preset utilization rate threshold, a resource recovery operation is triggered, and idle resources are released through a resource recovery algorithm;

[0012] Receive a resource application request submitted by an application, wherein the resource application request includes the required number of CPU cores, memory capacity, and network bandwidth requirements; construct a multi-dimensional resource allocation optimization model according to the real-time resource status of each node and the resource application request; solve the multi-dimensional resource allocation optimization model using a genetic algorithm to obtain an optimal resource allocation plan; according to the optimal resource allocation plan, allocate the application's tasks to corresponding nodes for execution, monitor the resource usage during task execution in real time, and trigger a task reallocation mechanism when resource competition is detected.

[0013] A first deep neural network model is constructed based on the system indicator data, and a second deep neural network model is constructed based on the application indicator data; the output results of the first deep neural network model and the second deep neural network model are fused to obtain a node resource utilization status evaluation result including:

[0014] Normalizing the system indicator data to obtain normalized system indicator data; inputting the normalized system indicator data into a first deep neural network, wherein the first deep neural network includes an input layer, three hidden layers, and an output layer, wherein the input layer includes four neurons, the number of neurons in the three hidden layers is sixteen, eight, and four, respectively, the output layer includes one neuron, and the output of the first deep neural network is a system resource load score;

[0015] Performing feature engineering processing on the application indicator data, inputting the application indicator data after the feature engineering processing into a second deep neural network, the second deep neural network includes an input layer, four hidden layers and an output layer, the input layer includes six neurons, the number of neurons in the four hidden layers is twenty-four, sixteen, eight and four respectively, a batch normalization layer is set after each of the hidden layers, the output layer includes one neuron, and the output of the second deep neural network is an application performance score;

[0016] Based on the system resource load score and the application performance score, an attention mechanism is used to calculate a weight coefficient; the weight coefficient is combined with the system resource load score and the application performance score to construct a fused input vector; the fused input vector is input into a multilayer perceptron, wherein the multilayer perceptron includes two fully connected layers, and the number of neurons in the two fully connected layers is eight and four, respectively; the output of the multilayer perceptron is subjected to a sigmoid activation function to obtain a final evaluation score; the final evaluation score is subjected to exponential moving average smoothing to obtain a node resource utilization status evaluation result.

[0017] According to the node resource utilization status evaluation result, a resource dynamic adjustment coefficient is calculated for each node in the YARN cluster; based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit of the allocatable resources of each node in real time, including:

[0018] Collecting node resource historical usage data, calculating resource usage variance based on the node resource historical usage data, and calculating resource usage fluctuation coefficient according to the resource usage variance; inputting the node resource historical usage data into a preset evaluation model to obtain a node resource utilization status evaluation result; calculating a resource dynamic adjustment coefficient according to the node resource utilization status evaluation result and the resource usage fluctuation coefficient, wherein the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient, and a network bandwidth adjustment coefficient; wherein the CPU adjustment coefficient is calculated according to the ratio of the node resource utilization status evaluation result to a preset evaluation threshold, the memory adjustment coefficient is calculated according to the product of the node resource utilization status evaluation result and the memory real-time utilization rate, and the network bandwidth adjustment coefficient is calculated according to the product of the node resource utilization status evaluation result and the bandwidth real-time utilization rate;

[0019] Construct a multi-layer sliding time window, wherein the multi-layer sliding time window includes a main window and a sub-window, wherein the main window size is sixty minutes, the sub-window size is five minutes, and the sliding step length of the main window and the sub-window is one minute; divide the node resource historical usage data according to the sub-windows, and perform time-weighted aggregation on the resource usage data in each of the divided sub-windows to obtain a sub-window aggregation value; combine the sub-window aggregation values ​​according to the main window, and calculate the resource usage trend value based on the combined multiple sub-window aggregation values;

[0020] The node benchmark resource upper limit is calculated according to the resource dynamic adjustment coefficient, the resource usage fluctuation coefficient, and the resource usage trend value, wherein the node benchmark resource upper limit includes the CPU core number upper limit, the memory capacity upper limit, and the bandwidth upper limit; the node benchmark resource upper limit is updated by exponential smoothing to obtain a smoothed resource upper limit; the smoothed resource upper limit is constrained according to a preset minimum reserved resource ratio and a maximum adjustment range to obtain a constrained resource upper limit; the resource allocation compensation factor is calculated based on historical resource allocation data, and the resource allocation compensation factor and the constrained resource upper limit are weighted to obtain a final resource upper limit, and the final resource upper limit is used to determine a final allocatable resource upper limit.

[0021] When the node resource utilization exceeds the preset utilization threshold, the resource expansion operation is triggered. The node resource quota is increased through the elastic scaling algorithm, including:

[0022] Calculating the comprehensive resource utilization rate based on the node resource utilization rate data, and calculating the upper threshold value of resource utilization rate and the lower threshold value of resource utilization rate according to the mean and standard deviation of the historical comprehensive resource utilization rate; collecting the comprehensive resource utilization rate data within a preset observation time window, and calculating the resource pressure score based on the comprehensive resource utilization rate data, and the resource pressure score uses a time decay method to perform weighted calculation on the utilization rate exceeding the upper threshold value of resource utilization rate;

[0023] When the comprehensive resource utilization is greater than the upper resource utilization threshold, the basic expansion capacity is calculated according to the difference between the comprehensive resource utilization and the upper resource utilization threshold;

[0024] Based on the load growth trend, the basic expansion capacity is corrected to obtain the predicted expansion capacity; the predicted expansion capacity is compared with the preset maximum single-step expansion capacity to determine the step size of the step-by-step expansion; based on the resource pressure score, the time interval between adjacent expansion operations is calculated; and the resource expansion operation is performed according to the step-by-step expansion step size and the time interval.

[0025] The basic expansion capacity is corrected based on the load growth trend to obtain the predicted expansion capacity; the predicted expansion capacity is compared with the preset maximum single-step expansion capacity to determine the step size of the step expansion; the time interval between adjacent expansion operations is calculated based on the resource pressure score; and the resource expansion operation is performed according to the step size of the step expansion and the time interval, including:

[0026] Collect system load time series data, and calculate the load growth rate based on the load data at adjacent moments; perform periodic analysis on the load time series data, and extract the periodic amplitude coefficient and phase offset parameter; perform weighted calculation on the historical load growth rate using an exponential decay method to obtain a load trend value, and simultaneously calculate the second-order derivative value of the load;

[0027] Calculate a correction factor according to the load trend value and the second-order derivative value, and multiply the correction factor by the basic expansion capacity to obtain the predicted expansion capacity; calculate a relative expansion upper limit based on the current resource amount, and compare the relative expansion upper limit with the absolute expansion upper limit to obtain the maximum single-step expansion capacity; calculate the optimal expansion step number according to the predicted expansion capacity, the maximum single-step expansion capacity, and the load trend value;

[0028] An exponential distribution function is used to calculate the expansion capacity of each expansion step, wherein the input parameters of the exponential distribution function include the expansion step sequence number, the predicted expansion capacity, and the optimal number of expansion steps; a basic expansion time interval is calculated based on the resource pressure score; and the basic expansion time interval is dynamically adjusted according to the expansion step sequence number to obtain an execution time interval of each expansion step;

[0029] Performing an exponential decay weighted calculation on the load change amplitude within a preset evaluation time window to obtain a stability score; when the stability score meets a preset threshold condition, calculating the resource reservation amount based on the predicted expansion capacity; and performing a step-by-step expansion operation according to the expansion capacity and the execution time interval;

[0030] Collect the target resource utilization and the actual resource utilization, calculate the mean square error between the target resource utilization and the actual resource utilization to obtain the expansion effect score; according to the difference between the expansion effect score and the preset effect threshold, adaptively adjust the expansion control parameters; the adaptively adjusted expansion control parameters are used to guide the execution of the next round of expansion operations.

[0031] When the node resource utilization rate is lower than the preset utilization rate threshold, the resource recovery operation is triggered. The resource recovery algorithm releases idle resources including:

[0032] When the comprehensive resource utilization rate is less than the lower threshold of the resource utilization rate, a basic recovery amount is calculated according to the difference between the comprehensive resource utilization rate and the lower threshold of the resource utilization rate; the basic recovery amount is constrained based on the minimum resource guarantee amount and the safety margin to obtain a safe recovery amount; the safe recovery amount is divided into multiple recovery stages, and the recovery step length of each recovery stage is calculated; the time delay of each recovery stage is calculated based on the recovery stage sequence number; and the resource recovery operation is performed according to the recovery step length and the time delay;

[0033] The operation cooling time is calculated based on the change amplitude of resource utilization; the state migration probability of the new and old resource states is calculated; the operation priority is calculated based on resource pressure, cost savings, and operation risk; and the resource expansion operation and the resource recovery operation are coordinated and controlled according to the operation cooling time, the state migration probability, and the operation priority.

[0034] According to the real-time resource status of each node and the resource application request, a multi-dimensional resource allocation optimization model is constructed; the multi-dimensional resource allocation optimization model is solved by a genetic algorithm to obtain an optimal resource allocation solution including:

[0035] Collecting real-time resource status data of each node, the real-time resource status data includes computing resource utilization, storage resource utilization, and network resource utilization; collecting resource application requests, the resource application requests include computing resource requirements, storage resource requirements, and network resource requirements; constructing a comprehensive objective function based on the real-time resource status data, the comprehensive objective function includes a resource utilization balance objective function, a response time minimization objective function, and an energy efficiency optimization objective function; setting resource capacity constraints based on the resource application requests and requesting to satisfy the constraints to obtain a resource allocation optimization model;

[0036] The computing resource requirements, storage resource allocation, and network resource allocation in the resource allocation optimization model are encoded as chromosome gene sequences; an initial population that satisfies constraint conditions is randomly generated according to the chromosome gene sequences; a comprehensive evaluation function is constructed by weighted combination of the comprehensive objective function and adding constraint violation penalty items; and a fitness value is obtained by calculating the comprehensive evaluation function value corresponding to each individual in the initial population;

[0037] Calculating the selection probability of an individual based on the fitness ranking of the individual in the population; performing a tournament selection operation according to the selection probability to obtain a high-quality individual; calculating the difference ratio between the fitness value of the high-quality individual and the maximum fitness value; calculating the adaptive crossover rate according to the difference ratio, and performing a crossover operation on the high-quality individual based on the adaptive crossover rate to obtain an offspring individual;

[0038] Calculate the dynamic mutation rate according to the current iteration generation; perform mutation operation on the offspring individuals based on the dynamic mutation rate to obtain mutant individuals; select individuals whose fitness values ​​are greater than a preset elite threshold from the mutant individuals as elite individuals; calculate the population diversity index of the mutant individuals; when the population diversity index is less than the preset diversity threshold, determine the neighborhood search range based on the initial search radius and the attenuation coefficient, and perform local search within the neighborhood search range to obtain the local optimal individual;

[0039] Calculate the degree of violation of the constraint conditions by the local optimal individual to obtain the violation degree; perform feasibility correction on the local optimal individual according to the violation degree to obtain a corrected individual; smooth the corrected individual and the historical optimal solution to obtain a smoothed individual; calculate the quality metric value and the stability metric value of the solution of the smoothed individual; calculate the evaluation index of the solution based on the quality metric value and the stability metric value; determine whether to update the optimal solution according to the evaluation index of the solution; when the number of iterations reaches a preset termination condition, output the optimal solution as an optimal resource allocation plan, and the optimal resource allocation plan includes the computing resource allocation amount, storage resource allocation amount, and network resource allocation amount of each node.

[0040] A second aspect of an embodiment of the present invention provides a YARN cluster node resource dynamic adjustment system based on dual-mode perception, including:

[0041] The first unit is used to collect system indicator data and application indicator data of YARN cluster nodes, wherein the system indicator data includes node CPU utilization, memory usage, network bandwidth occupancy and disk I / O load, and the application indicator data includes application execution time, resource application amount, task completion rate and data processing throughput; construct a first deep neural network model based on the system indicator data, and construct a second deep neural network model based on the application indicator data; and fuse the output results of the first deep neural network model and the second deep neural network model to obtain a node resource utilization status evaluation result;

[0042] The second unit is used to calculate the resource dynamic adjustment coefficient for each node in the YARN cluster according to the node resource utilization status evaluation result, and the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient and a network bandwidth adjustment coefficient; based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit of the allocatable resources of each node in real time; when the node resource utilization rate exceeds the preset utilization rate threshold, a resource expansion operation is triggered, and the node resource quota is increased through an elastic scaling algorithm; when the node resource utilization rate is lower than the preset utilization rate threshold, a resource recovery operation is triggered, and idle resources are released through a resource recovery algorithm;

[0043] The third unit is used to receive a resource application request submitted by an application, wherein the resource application request includes the required number of CPU cores, memory capacity and network bandwidth requirements; construct a multi-dimensional resource allocation optimization model according to the real-time resource status of each node and the resource application request; solve the multi-dimensional resource allocation optimization model using a genetic algorithm to obtain an optimal resource allocation plan; according to the optimal resource allocation plan, allocate the application's tasks to the corresponding nodes for execution, and monitor the resource usage during the task execution in real time, and trigger a task reallocation mechanism when resource competition is detected.

[0044] A third aspect of the embodiments of the present invention

[0045] An electronic device is provided, comprising:

[0046] processor;

[0047] a memory for storing processor-executable instructions;

[0048] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0049] A fourth aspect of the embodiments of the present invention is:

[0050] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0051] The present invention provides a method for dynamically adjusting YARN cluster node resources through dual-mode perception, which has the following beneficial effects:

[0052] By constructing a dual deep neural network model of system indicators and application indicators and fusing the output results of the two models, it is possible to comprehensively and accurately evaluate the node resource utilization status, avoid the limitations of single-dimensional evaluation, and improve the accuracy and reliability of resource evaluation.

[0053] By adopting a dynamic resource adjustment mechanism based on a sliding time window, combined with elastic scaling and resource recovery algorithms, adaptive adjustment of node resources is achieved, which can not only respond to the growth of resource demand in a timely manner, but also effectively recover idle resources, significantly improving the utilization efficiency of cluster resources.

[0054] By building a multi-dimensional resource allocation optimization model and using genetic algorithms to solve it, the optimal allocation of computing resources is achieved. At the same time, the introduction of a task reallocation mechanism can effectively deal with resource competition, ensure the operating performance and service quality of the application, and improve the overall throughput of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a process of dynamically adjusting YARN cluster node resources based on dual-mode perception according to an embodiment of the present invention;

[0056] Figure 2 It is a structural diagram of a YARN cluster node resource dynamic adjustment system based on dual-mode perception according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0059] Figure 1FIG. 1 is a flow chart of a method for dynamically adjusting YARN cluster node resources based on dual-mode perception according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] S101. Collect system indicator data and application indicator data of YARN cluster nodes, wherein the system indicator data includes node CPU utilization, memory usage, network bandwidth occupancy and disk I / O load, and the application indicator data includes application execution time, resource application amount, task completion rate and data processing throughput; construct a first deep neural network model based on the system indicator data, and construct a second deep neural network model based on the application indicator data; fuse the output results of the first deep neural network model and the second deep neural network model to obtain a node resource utilization status evaluation result;

[0061] S102. According to the node resource utilization status evaluation result, a resource dynamic adjustment coefficient is calculated for each node in the YARN cluster, and the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient, and a network bandwidth adjustment coefficient; based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit of the allocatable resources of each node in real time; when the node resource utilization rate exceeds the preset utilization rate threshold, a resource expansion operation is triggered, and the node resource quota is increased through an elastic scaling algorithm; when the node resource utilization rate is lower than the preset utilization rate threshold, a resource recovery operation is triggered, and idle resources are released through a resource recovery algorithm;

[0062] S103. Receive a resource application request submitted by an application, wherein the resource application request includes the required number of CPU cores, memory capacity, and network bandwidth requirements; construct a multi-dimensional resource allocation optimization model based on the real-time resource status of each node and the resource application request; use a genetic algorithm to solve the multi-dimensional resource allocation optimization model to obtain an optimal resource allocation plan; according to the optimal resource allocation plan, allocate the application's tasks to the corresponding nodes for execution, and monitor the resource usage during the task execution in real time, and trigger a task reallocation mechanism when resource competition is detected.

[0063] In an optional implementation, a first deep neural network model is constructed based on the system indicator data, and a second deep neural network model is constructed based on the application indicator data; the output results of the first deep neural network model and the second deep neural network model are fused to obtain a node resource utilization status evaluation result including:

[0064] Normalizing the system indicator data to obtain normalized system indicator data; inputting the normalized system indicator data into a first deep neural network, wherein the first deep neural network includes an input layer, three hidden layers, and an output layer, wherein the input layer includes four neurons, the number of neurons in the three hidden layers is sixteen, eight, and four, respectively, the output layer includes one neuron, and the output of the first deep neural network is a system resource load score;

[0065] Performing feature engineering processing on the application indicator data, inputting the application indicator data after the feature engineering processing into a second deep neural network, the second deep neural network includes an input layer, four hidden layers and an output layer, the input layer includes six neurons, the number of neurons in the four hidden layers is twenty-four, sixteen, eight and four respectively, a batch normalization layer is set after each of the hidden layers, the output layer includes one neuron, and the output of the second deep neural network is an application performance score;

[0066] Based on the system resource load score and the application performance score, an attention mechanism is used to calculate a weight coefficient; the weight coefficient is combined with the system resource load score and the application performance score to construct a fused input vector; the fused input vector is input into a multilayer perceptron, wherein the multilayer perceptron includes two fully connected layers, and the number of neurons in the two fully connected layers is eight and four, respectively; the output of the multilayer perceptron is subjected to a sigmoid activation function to obtain a final evaluation score; the final evaluation score is subjected to exponential moving average smoothing to obtain a node resource utilization status evaluation result.

[0067] First, the system indicator data is processed. The system indicator data includes four indicators: CPU usage, memory usage, disk IO usage, and network bandwidth usage. These indicator data are uniformly mapped to the range of zero to one through the minimum and maximum normalization method to obtain the normalized system indicator data. For example, the original CPU usage is 85%, the minimum value is 0%, and the maximum value is 100%. The normalized value is 0.85.

[0068] Next, we build the first deep neural network model for system resource load evaluation. The network adopts a four-layer structure, and the input layer receives four normalized system indicators. The three hidden layers contain sixteen, eight, and four neurons respectively, and the ReLU activation function is used to increase the nonlinear expression ability of the network. The output layer uses one neuron, and the output is mapped to the range of zero to one through the sigmoid function to obtain the system resource load score.

[0069] For the processing of application indicator data, feature engineering is first performed. Application indicators include six indicators: response time, throughput, error rate, concurrency, queue length, and resource occupancy. These indicators are standardized and statistical features such as mean and variance are extracted.

[0070] A second deep neural network model is built to evaluate application performance. The network contains six input neurons, corresponding to the six processed application metrics. The four hidden layers contain twenty-four, sixteen, eight, and four neurons, respectively. A batch normalization layer is added after each hidden layer to speed up the training process and improve model stability. The output layer outputs the application performance score through one neuron.

[0071] When fusing the outputs of the two networks, the attention mechanism is used to dynamically calculate the weights. Different weight coefficients are assigned to the system resource load score and the application performance score according to the current system status and application performance. For example, when system resources are tight, the weight of the system load score will increase accordingly.

[0072] The calculated weight coefficients are combined with the two scores to construct a fusion input vector. This vector is input into a two-layer multilayer perceptron, where the two fully connected layers contain eight and four neurons respectively. Finally, the evaluation score between zero and one is obtained through the sigmoid function.

[0073] In order to eliminate the fluctuation of the evaluation results, the final evaluation score is smoothed by exponential moving average. The smoothing coefficient is set to 0.2, that is, the new evaluation result accounts for 20% of the weight and the historical evaluation result accounts for 80% of the weight, and the smoothed node resource utilization status evaluation result is obtained.

[0074] This application can achieve:

[0075] Through the dual evaluation mechanism of system indicators and application indicators, the node resource utilization status is fully and accurately reflected, avoiding the one-sidedness caused by single-dimensional evaluation and improving the reliability of the evaluation results. The fusion solution of deep neural network model and attention mechanism can adaptively adjust the importance of different indicators, so that the evaluation results are more in line with the actual operation situation, and improve the accuracy and adaptability of the evaluation. Through optimization methods such as batch normalization and exponential moving average, the training efficiency of the model and the stability of the evaluation results are significantly improved, making the system have better practical value and promotion prospects.

[0076] In an optional implementation, according to the node resource utilization status evaluation result, a resource dynamic adjustment coefficient is calculated for each node in the YARN cluster; based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit value of the allocatable resources of each node in real time, including:

[0077] Collecting node resource historical usage data, calculating resource usage variance based on the node resource historical usage data, and calculating resource usage fluctuation coefficient according to the resource usage variance; inputting the node resource historical usage data into a preset evaluation model to obtain a node resource utilization status evaluation result; calculating a resource dynamic adjustment coefficient according to the node resource utilization status evaluation result and the resource usage fluctuation coefficient, wherein the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient, and a network bandwidth adjustment coefficient; wherein the CPU adjustment coefficient is calculated according to the ratio of the node resource utilization status evaluation result to a preset evaluation threshold, the memory adjustment coefficient is calculated according to the product of the node resource utilization status evaluation result and the memory real-time utilization rate, and the network bandwidth adjustment coefficient is calculated according to the product of the node resource utilization status evaluation result and the bandwidth real-time utilization rate;

[0078] Construct a multi-layer sliding time window, wherein the multi-layer sliding time window includes a main window and a sub-window, wherein the main window size is sixty minutes, the sub-window size is five minutes, and the sliding step length of the main window and the sub-window is one minute; divide the node resource historical usage data according to the sub-windows, and perform time-weighted aggregation on the resource usage data in each of the divided sub-windows to obtain a sub-window aggregation value; combine the sub-window aggregation values ​​according to the main window, and calculate the resource usage trend value based on the combined multiple sub-window aggregation values;

[0079] The node benchmark resource upper limit is calculated according to the resource dynamic adjustment coefficient, the resource usage fluctuation coefficient, and the resource usage trend value, wherein the node benchmark resource upper limit includes the CPU core number upper limit, the memory capacity upper limit, and the bandwidth upper limit; the node benchmark resource upper limit is updated by exponential smoothing to obtain a smoothed resource upper limit; the smoothed resource upper limit is constrained according to a preset minimum reserved resource ratio and a maximum adjustment range to obtain a constrained resource upper limit; the resource allocation compensation factor is calculated based on historical resource allocation data, and the resource allocation compensation factor and the constrained resource upper limit are weighted to obtain a final resource upper limit, and the final resource upper limit is used to determine a final allocatable resource upper limit.

[0080] First, collect the historical usage data of node resources in the YARN cluster, including CPU usage, memory usage, network bandwidth usage and other indicators. The collected data records the resource usage of each node at intervals of one minute. Based on the collected historical data, calculate the usage variance of each resource indicator. Taking CPU usage as an example, the fluctuation of resource usage is reflected by calculating the dispersion of CPU usage over a period of time. When the CPU usage variance is large, it means that the resource usage fluctuates greatly; when the variance is small, it means that the resource usage is relatively stable.

[0081] Then, the node resource historical usage data is input into the pre-trained evaluation model, which uses the random forest algorithm to evaluate the resource utilization status of the node by analyzing the resource usage pattern. The evaluation results are divided into five levels: severe shortage, mild shortage, normal, mild surplus, and severe surplus. The input features of the evaluation model include statistical features such as the mean, peak, and valley of resource utilization.

[0082] The resource dynamic adjustment coefficient is calculated based on the evaluation results and the resource usage fluctuation coefficient. For the CPU adjustment coefficient, when the evaluation result is severely insufficient, the coefficient value is 80% of the preset evaluation threshold; when the evaluation result is normal, the coefficient value is 100% of the preset evaluation threshold. The memory adjustment coefficient is calculated based on the evaluation result and the real-time memory utilization. For example, when the evaluation result is slightly excessive and the memory utilization is 70%, the adjustment coefficient value is 49%. The calculation method of the network bandwidth adjustment coefficient is similar to the memory adjustment coefficient.

[0083] A multi-layer sliding time window mechanism is constructed. The main window of 60 minutes is used to capture long-term resource usage trends, and the sub-window of 5 minutes is used to respond to resource usage changes in real time. For the resource usage data in each sub-window, a time-weighted method is used for aggregation. The closer the data is to the current moment, the greater the weight. For example, the weight of the data in the last minute is 0.3, the weight of the data in the previous minute is 0.25, and so on.

[0084] Combine the sub-window aggregate values ​​in the main window to calculate the resource usage trend value. By analyzing the changing trend of the aggregate values ​​of the twelve sub-windows, determine whether the resource usage is in an upward or downward trend. For example, when the aggregate values ​​of three consecutive sub-windows show an upward trend, it means that the resource demand is increasing.

[0085] The node benchmark resource upper limit is calculated based on the resource dynamic adjustment coefficient, fluctuation coefficient and usage trend value. The upper limit of the benchmark resource is updated using exponential smoothing, and the smoothing coefficient is set to 0.3. At the same time, the constraints of the minimum reserved resource ratio of 20% and the maximum adjustment range of 30% are considered. Finally, the compensation factor is calculated based on the historical resource allocation data, and the final allocable resource upper limit is obtained in combination with the constraint resource upper limit.

[0086] This application can achieve:

[0087] The multi-layer sliding time window mechanism achieves accurate perception of resource usage status, ensuring rapid response to short-term resource fluctuations while maintaining the overall stability of resource allocation. The dynamic resource adjustment coefficient and multiple constraint mechanisms are used to avoid drastic fluctuations in resource allocation, improve cluster resource utilization efficiency, and reduce resource fragmentation. The compensation mechanism and smoothing method based on historical data make resource allocation more reasonable, improve the stability and reliability of task operation, and reduce system operation and maintenance costs.

[0088] In an optional implementation, when the node resource utilization exceeds a preset utilization threshold, a resource expansion operation is triggered, and increasing the node resource quota through an elastic scaling algorithm includes:

[0089] Calculating the comprehensive resource utilization rate based on the node resource utilization rate data, and calculating the upper threshold value of resource utilization rate and the lower threshold value of resource utilization rate according to the mean and standard deviation of the historical comprehensive resource utilization rate; collecting the comprehensive resource utilization rate data within a preset observation time window, and calculating the resource pressure score based on the comprehensive resource utilization rate data, and the resource pressure score uses a time decay method to perform weighted calculation on the utilization rate exceeding the upper threshold value of resource utilization rate;

[0090] When the comprehensive resource utilization is greater than the upper resource utilization threshold, the basic expansion capacity is calculated according to the difference between the comprehensive resource utilization and the upper resource utilization threshold;

[0091] Based on the load growth trend, the basic expansion capacity is corrected to obtain the predicted expansion capacity; the predicted expansion capacity is compared with the preset maximum single-step expansion capacity to determine the step size of the step-by-step expansion; based on the resource pressure score, the time interval between adjacent expansion operations is calculated; and the resource expansion operation is performed according to the step-by-step expansion step size and the time interval.

[0092] In the resource elastic scaling scenario, you first need to collect basic resource indicator data such as the node's CPU usage, memory usage, and disk IO usage. For CPU usage, collect usage data within the past 5 minutes; for memory usage, collect usage data within the past 10 minutes; for disk IO usage, obtain data within the past 15 minutes. Normalize these resource usage data of different dimensions and calculate the comprehensive resource utilization using a weighted average method. The CPU usage weight is 0.5, the memory usage weight is 0.3, and the disk IO usage weight is 0.2.

[0093] When calculating resource utilization thresholds based on historical data, collect comprehensive resource utilization data for the past 30 days. Calculate the average and standard deviation of these data, and use the average plus 1.5 times the standard deviation as the upper resource utilization threshold, and the average minus 1.5 times the standard deviation as the lower resource utilization threshold. For example, if the average comprehensive resource utilization of a node in the past 30 days is 65% and the standard deviation is 10%, the upper resource utilization threshold is 80% and the lower resource utilization threshold is 50%.

[0094] When calculating the resource stress score, set a 15-minute observation time window and collect comprehensive resource utilization data every minute. When it is detected that the resource utilization exceeds the upper threshold, calculate the value of the excess part and perform time decay processing based on the distance from the current time. The closer the data is to the current time, the greater the weight is. Specifically, a linear decay method is used, with the weight of the nearest time point being 1 and the weight of the farthest time point being 0.2. The weighted excess values ​​are accumulated to obtain the resource stress score.

[0095] When the comprehensive resource utilization exceeds the upper threshold, the percentage of the excess is calculated as the basic expansion capacity. For example, if the current comprehensive resource utilization is 90% and the upper threshold is 80%, the basic expansion capacity is 12.5% ​​(i.e., the excess 10% divided by the current threshold 80%).

[0096] When determining the predicted capacity expansion, analyze the load growth trend in the last hour. If it shows a continuous upward trend, increase the basic capacity expansion by 20%; if it shows a fluctuating upward trend, increase it by 10%; if it shows a stable trend, maintain the basic capacity expansion unchanged.

[0097] To avoid overly aggressive expansion operations, the maximum single-step expansion capacity is set to 30% of the current resource quota. If the predicted expansion capacity exceeds the maximum single-step expansion capacity, the expansion operation is divided into multiple steps, and each step of expansion capacity does not exceed the maximum single-step expansion capacity.

[0098] The time interval between adjacent expansion operations is determined based on the resource pressure score. The higher the pressure score, the shorter the time interval. When the pressure score is greater than 100, the time interval is set to 5 minutes; when the pressure score is between 50 and 100, the time interval is set to 10 minutes; when the pressure score is less than 50, the time interval is set to 15 minutes.

[0099] This application can achieve:

[0100] By real-time monitoring of multi-dimensional resource indicators and calculating comprehensive resource utilization, the overall load status of the system can be accurately reflected, avoiding misjudgment caused by a single indicator, and improving the accuracy and comprehensiveness of resource monitoring.

[0101] The dynamic threshold calculation method based on historical data, combined with the time-decayed pressure score calculation mechanism, can effectively identify resource pressure conditions, ensuring the timeliness of capacity expansion and avoiding frequent capacity expansion caused by short-term fluctuations in resource utilization.

[0102] By analyzing the load growth trend to adjust the expansion capacity and setting the maximum single-step expansion limit, the predictability and stability of the expansion operation are achieved, which not only meets the needs of business growth, but also avoids resource waste and improves resource utilization efficiency.

[0103] In an optional implementation, the basic expansion capacity is corrected based on the load growth trend to obtain a predicted expansion capacity; the predicted expansion capacity is compared with a preset maximum single-step expansion capacity to determine a step size of the step expansion; the time interval between adjacent expansion operations is calculated based on the resource pressure score; and the resource expansion operation is performed according to the step size of the step expansion and the time interval, including:

[0104] Collect system load time series data, and calculate the load growth rate based on the load data at adjacent moments; perform periodic analysis on the load time series data, and extract the periodic amplitude coefficient and phase offset parameter; perform weighted calculation on the historical load growth rate using an exponential decay method to obtain a load trend value, and simultaneously calculate the second-order derivative value of the load;

[0105] Calculate a correction factor according to the load trend value and the second-order derivative value, and multiply the correction factor by the basic expansion capacity to obtain the predicted expansion capacity; calculate a relative expansion upper limit based on the current resource amount, and compare the relative expansion upper limit with the absolute expansion upper limit to obtain the maximum single-step expansion capacity; calculate the optimal expansion step number according to the predicted expansion capacity, the maximum single-step expansion capacity, and the load trend value;

[0106] An exponential distribution function is used to calculate the expansion capacity of each expansion step, wherein the input parameters of the exponential distribution function include the expansion step sequence number, the predicted expansion capacity, and the optimal number of expansion steps; a basic expansion time interval is calculated based on the resource pressure score; and the basic expansion time interval is dynamically adjusted according to the expansion step sequence number to obtain an execution time interval of each expansion step;

[0107] Performing an exponential decay weighted calculation on the load change amplitude within a preset evaluation time window to obtain a stability score; when the stability score meets a preset threshold condition, calculating the resource reservation amount based on the predicted expansion capacity; and performing a step-by-step expansion operation according to the expansion capacity and the execution time interval;

[0108] Collect the target resource utilization and the actual resource utilization, calculate the mean square error between the target resource utilization and the actual resource utilization to obtain the expansion effect score; according to the difference between the expansion effect score and the preset effect threshold, adaptively adjust the expansion control parameters; the adaptively adjusted expansion control parameters are used to guide the execution of the next round of expansion operations.

[0109] The resource expansion system first collects the time series data of the system load and calculates the load growth rate by comparing the load values ​​at adjacent moments. Taking the load data collected every 5 minutes as an example, if the current load is 85% and the previous load is 80%, the load growth rate is 1.0625. Perform periodic analysis on the collected load time series data to identify the periodic characteristics of load changes. For example, if the load of a business system increases significantly between 9 and 11 every day, periodic parameters such as amplitude coefficient 1.2 and phase shift 2 hours can be extracted.

[0110] The historical load growth rate is weighted by exponential decay, with recent data having a larger weight and long-term data having a smaller weight. Assuming that the weight of the load growth rate in the most recent hour is 0.6, the weight of the previous hour is 0.3, and the weight of the hour before that is 0.1, the load trend value is obtained by weighted calculation. At the same time, the change in the load change rate, that is, the second-order derivative value, is calculated to determine whether the load growth is accelerating or decelerating.

[0111] The correction factor is calculated based on the load trend value and the second-order derivative value. When the load shows an accelerating growth trend, the correction factor is greater than 1, such as 1.2; when the load growth trend slows down, the correction factor is less than 1, such as 0.8. Multiply the correction factor by the basic expansion capacity to get the predicted expansion capacity. Assuming the basic expansion capacity is 10 resource units and the correction factor is 1.2, the predicted expansion capacity is 12 resource units.

[0112] Calculate the upper limit of relative expansion based on the current total amount of resources, which generally does not exceed 50% of the current amount of resources. Also consider the upper limit of absolute expansion, such as a maximum expansion of 20 resource units at a time, and take the smaller value of the two as the maximum single-step expansion capacity. Calculate the optimal number of expansion steps based on the predicted expansion capacity, the maximum single-step expansion capacity, and the load trend value. When the load grows faster, appropriately reduce the number of expansion steps, such as setting it to 3 steps; when the load grows slowly, increase the number of expansion steps, such as setting it to 5 steps.

[0113] The specific expansion capacity of each expansion step is calculated using an exponential distribution method. A larger expansion capacity is allocated in the early stage of expansion, and a smaller expansion capacity is allocated in the later stage. For example, if the total expansion capacity is 12 resource units and the expansion is divided into 3 steps, the first step is to expand 6 units, the second step is to expand 4 units, and the third step is to expand 2 units. The expansion basic time interval is calculated based on the current resource pressure score of the system. The greater the pressure, the shorter the interval. The time interval is then dynamically adjusted according to the expansion step sequence number, and the intervals of subsequent steps are extended accordingly.

[0114] The load changes within the preset time window are analyzed and the stability score is calculated. When the stability score is higher than the preset threshold, a certain percentage of resource reservation is additionally calculated. Then, the step-by-step expansion operation is performed according to the determined expansion capacity and time interval.

[0115] During the expansion process, the target resource utilization rate and actual resource utilization rate data are continuously collected, and the mean square error between the two is calculated to obtain the expansion effect score. The expansion effect score is compared with the preset effect threshold, and the expansion control parameters are dynamically adjusted according to the difference to optimize the next round of expansion operations.

[0116] This application can achieve:

[0117] By conducting multi-dimensional analysis of load data, including growth trends, cycle characteristics, and stability, we can accurately predict the system's resource requirements, improve the accuracy of capacity expansion, and avoid resource waste.

[0118] The adoption of a step-by-step expansion strategy and dynamic adjustment of expansion parameters not only ensures the timeliness of expansion, but also avoids the impact of expansion operations on the system, improving the smoothness of the expansion process and system stability.

[0119] By introducing expansion effect evaluation and parameter adaptive adjustment mechanisms, the expansion strategy is continuously optimized through closed-loop feedback, which improves the intelligence level of the expansion process and enhances the system's adaptability to various load scenarios.

[0120] In an optional implementation, when the node resource utilization is lower than a preset utilization threshold, a resource recovery operation is triggered, and releasing idle resources through a resource recovery algorithm includes:

[0121] When the comprehensive resource utilization rate is less than the lower threshold of the resource utilization rate, a basic recovery amount is calculated according to the difference between the comprehensive resource utilization rate and the lower threshold of the resource utilization rate; the basic recovery amount is constrained based on the minimum resource guarantee amount and the safety margin to obtain a safe recovery amount; the safe recovery amount is divided into multiple recovery stages, and the recovery step length of each recovery stage is calculated; the time delay of each recovery stage is calculated based on the recovery stage sequence number; and the resource recovery operation is performed according to the recovery step length and the time delay;

[0122] The operation cooling time is calculated based on the change amplitude of resource utilization; the state migration probability of the new and old resource states is calculated; the operation priority is calculated based on resource pressure, cost savings, and operation risk; and the resource expansion operation and the resource recovery operation are coordinated and controlled according to the operation cooling time, the state migration probability, and the operation priority.

[0123] The resource recycling trigger mechanism is based on real-time monitoring of node resource utilization. When it is detected that the comprehensive resource utilization of the node is lower than the preset resource utilization threshold, the system will start the resource recycling process. The comprehensive resource utilization is obtained by weighted calculation of multiple indicators such as CPU utilization, memory utilization, network bandwidth utilization, etc. The weight of each indicator can be adjusted according to the actual business scenario.

[0124] First, calculate the basic recycling amount by multiplying the difference between the current comprehensive resource utilization rate and the lower threshold of resource utilization rate by the current total amount of allocated resources. For example, when the comprehensive resource utilization rate is 30% and the lower threshold of resource utilization rate is 50%, the difference is 20%. If the current total amount of allocated resources is 100 resource units, the basic recycling amount is 20 resource units.

[0125] Next, the safety constraint calculation is performed to ensure that the remaining resources after recycling are not less than the minimum resource guarantee amount, and a safety margin is reserved. The minimum resource guarantee amount is determined based on the historical load fluctuations, usually 50% of the peak load; the safety margin is 20% of the basic recycling amount. The safe recycling amount obtained after the constraint is the final recyclable resource amount.

[0126] In order to avoid the impact of resource recycling on the business, the safe recycling volume is divided into multiple recycling stages. The specific division method is to divide the safe recycling volume into three stages evenly, and the recycling step length of each stage is one-third of the safe recycling volume. For example, if the safe recycling volume is 15 resource units, the recycling step length of each stage is 5 resource units.

[0127] The execution time of each recycling stage needs to be set with different delays, and the delay time increases with the stage number. The first stage is delayed for five minutes, the second stage is delayed for ten minutes, and the third stage is delayed for fifteen minutes. This progressive delay setting allows the system to gradually adapt to resource changes.

[0128] When performing resource recovery operations, you also need to consider the operation cooldown time to avoid frequent triggering of resource adjustments. The cooldown time is dynamically calculated based on the change in resource utilization. The greater the change, the longer the cooldown time. For example, when the resource utilization changes by more than 30%, set a cooldown time of 30 minutes.

[0129] At the same time, the migration probability of the new and old resource states is calculated, which reflects the feasibility of resource adjustment. The migration probability is calculated by analyzing the historical resource adjustment records and the current system load status. For example, if the historical adjustment success rate is 90% and the current system load is stable, the migration probability is high.

[0130] Finally, the operational priority is calculated based on three dimensions: resource pressure, cost savings, and operational risk. Resource pressure reflects the current resource tension of the system, cost savings reflects the economic benefits brought by resource recovery, and operational risk indicates the negative impact that resource adjustments may bring. The weights of these three factors can be adjusted according to actual needs.

[0131] This application can achieve:

[0132] Through a multi-stage progressive resource recovery solution, resources can be adjusted smoothly, effectively avoiding the impact of resource recovery on the business and ensuring the stability and reliability of the system.

[0133] Based on comprehensive resource utilization, safety constraints and multi-dimensional decision-making mechanisms, accurate resource recovery is achieved, which not only ensures resource utilization efficiency but also ensures the quality of business operations, achieving the optimal balance between resource utilization and business needs.

[0134] The coordinated control mechanism of dynamic cooling time and state migration probability effectively prevents resource shocks, improves the accuracy and reliability of resource scheduling, reduces operation and maintenance costs, and improves the overall operation efficiency of the system.

[0135] In an optional implementation, a multi-dimensional resource allocation optimization model is constructed according to the real-time resource status of each node and the resource application request; and a genetic algorithm is used to solve the multi-dimensional resource allocation optimization model to obtain an optimal resource allocation solution, including:

[0136] Collecting real-time resource status data of each node, the real-time resource status data includes computing resource utilization, storage resource utilization, and network resource utilization; collecting resource application requests, the resource application requests include computing resource requirements, storage resource requirements, and network resource requirements; constructing a comprehensive objective function based on the real-time resource status data, the comprehensive objective function includes a resource utilization balance objective function, a response time minimization objective function, and an energy efficiency optimization objective function; setting resource capacity constraints based on the resource application requests and requesting to satisfy the constraints to obtain a resource allocation optimization model;

[0137] The computing resource requirements, storage resource allocation, and network resource allocation in the resource allocation optimization model are encoded as chromosome gene sequences; an initial population that satisfies constraint conditions is randomly generated according to the chromosome gene sequences; a comprehensive evaluation function is constructed by weighted combination of the comprehensive objective function and adding constraint violation penalty items; and a fitness value is obtained by calculating the comprehensive evaluation function value corresponding to each individual in the initial population;

[0138] Calculating the selection probability of an individual based on the fitness ranking of the individual in the population; performing a tournament selection operation according to the selection probability to obtain a high-quality individual; calculating the difference ratio between the fitness value of the high-quality individual and the maximum fitness value; calculating the adaptive crossover rate according to the difference ratio, and performing a crossover operation on the high-quality individual based on the adaptive crossover rate to obtain an offspring individual;

[0139] Calculate the dynamic mutation rate according to the current iteration generation; perform mutation operation on the offspring individuals based on the dynamic mutation rate to obtain mutant individuals; select individuals whose fitness values ​​are greater than a preset elite threshold from the mutant individuals as elite individuals; calculate the population diversity index of the mutant individuals; when the population diversity index is less than the preset diversity threshold, determine the neighborhood search range based on the initial search radius and the attenuation coefficient, and perform local search within the neighborhood search range to obtain the local optimal individual;

[0140] Calculate the degree of violation of the constraint conditions by the local optimal individual to obtain the violation degree; perform feasibility correction on the local optimal individual according to the violation degree to obtain a corrected individual; smooth the corrected individual and the historical optimal solution to obtain a smoothed individual; calculate the quality metric value and the stability metric value of the solution of the smoothed individual; calculate the evaluation index of the solution based on the quality metric value and the stability metric value; determine whether to update the optimal solution according to the evaluation index of the solution; when the number of iterations reaches a preset termination condition, output the optimal solution as an optimal resource allocation plan, and the optimal resource allocation plan includes the computing resource allocation amount, storage resource allocation amount, and network resource allocation amount of each node.

[0141] First, the real-time resource status of each node is collected, including computing resource utilization, storage resource utilization, and network resource utilization. The collection method is implemented by deploying a resource monitoring agent on each node, collecting data every 10 seconds. For example, the computing resource utilization of a node is 75%, the storage resource utilization is 60%, and the network resource utilization is 45%.

[0142] At the same time, the resource application requests submitted by the user are collected, including the computing resource requirements, storage resource requirements, and network resource requirements. For example, the computing resources requested by the user are 4-core CPUs, storage resources are 100GB, and network bandwidth is 100Mbps.

[0143] Based on the collected real-time resource status data, a comprehensive objective function is constructed. This function considers three aspects: resource utilization balance, response time, and energy efficiency. Resource utilization balance is measured by calculating the standard deviation of each resource utilization rate. The smaller the standard deviation, the more balanced the resource utilization. The response time is estimated by establishing a prediction model based on historical statistical data. The energy efficiency is calculated based on the energy consumption characteristic curve of each node.

[0144] Constraints are set in conjunction with resource application requests: the total amount of resource allocation for each node cannot exceed its maximum capacity; the minimum resource requirement requested by the user must be met. In this way, a complete resource allocation optimization model is constructed.

[0145] Next, we use the improved genetic algorithm to solve the model. First, we encode the allocation scheme of computing resources, storage resources, and network resources into chromosomes using real number encoding. For example, the chromosome [4,100,100] indicates the allocation of 4-core CPUs, 100GB storage, and 100Mbps bandwidth.

[0146] Randomly generate 50 initial solutions that meet the constraints as the initial population. Calculate the fitness value of each solution, which consists of the comprehensive objective function value and the constraint violation penalty value. Calculate the selection probability based on the ranking of the fitness value, and use the tournament selection method to select high-quality individuals. For example, if the selection probability is set to 0.8, it means that there is an 80% probability of selecting individuals with higher fitness.

[0147] Calculate the fitness difference ratio between high-quality individuals and optimal individuals, and dynamically adjust the crossover rate based on this ratio. For example, when the difference ratio is 0.3, set the crossover rate to 0.7. Perform a crossover operation to generate offspring individuals. Dynamically adjust the mutation rate according to the number of iterations. The mutation rate is larger in the early stage of iteration to increase the search range, and smaller in the later stage to improve convergence. Perform mutation operations on offspring individuals.

[0148] From the mutated individuals, select individuals whose fitness values ​​exceed 90% of the optimal value as elite individuals. Calculate the diversity index of the population. When the index is less than 0.2, perform a local search near the current optimal solution. Perform feasibility correction on the solution obtained by the local search to ensure that all constraints are met. Then smooth it with the historical optimal solution to improve the stability of the solution. Finally, evaluate the quality and stability of the solution and decide whether to update the optimal solution. Stop the iteration when the iteration reaches 1000 times or there is no improvement in the optimal solution for 50 consecutive times, and output the final resource allocation plan.

[0149] This application can achieve:

[0150] Through real-time monitoring and dynamic optimization, the coordinated allocation of computing resources, storage resources and network resources is achieved, which significantly improves resource utilization efficiency, reduces resource waste, and increases the utilization rate of various resources by more than 25%.

[0151] An improved genetic algorithm is used for multi-objective optimization solution, which improves the convergence speed and stability of the algorithm while ensuring the quality of the solution. Compared with the traditional method, the calculation time is shortened by 40% and a better resource allocation plan can be obtained.

[0152] A closed-loop feedback mechanism for resource allocation has been established, which adapts to dynamically changing resource demands by adjusting optimization strategies in real time, making the system more adaptable and robust, and improving the stability of the resource allocation scheme by 35%.

[0153] Figure 2 FIG. 1 is a schematic diagram of a YARN cluster node resource dynamic adjustment system based on dual-mode perception according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0154] The first unit is used to collect system indicator data and application indicator data of YARN cluster nodes, wherein the system indicator data includes node CPU utilization, memory usage, network bandwidth occupancy and disk I / O load, and the application indicator data includes application execution time, resource application amount, task completion rate and data processing throughput; construct a first deep neural network model based on the system indicator data, and construct a second deep neural network model based on the application indicator data; and fuse the output results of the first deep neural network model and the second deep neural network model to obtain a node resource utilization status evaluation result;

[0155] The second unit is used to calculate the resource dynamic adjustment coefficient for each node in the YARN cluster according to the node resource utilization status evaluation result, and the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient and a network bandwidth adjustment coefficient; based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit of the allocatable resources of each node in real time; when the node resource utilization rate exceeds the preset utilization rate threshold, a resource expansion operation is triggered, and the node resource quota is increased through an elastic scaling algorithm; when the node resource utilization rate is lower than the preset utilization rate threshold, a resource recovery operation is triggered, and idle resources are released through a resource recovery algorithm;

[0156] The third unit is used to receive a resource application request submitted by an application, wherein the resource application request includes the required number of CPU cores, memory capacity and network bandwidth requirements; construct a multi-dimensional resource allocation optimization model according to the real-time resource status of each node and the resource application request; solve the multi-dimensional resource allocation optimization model using a genetic algorithm to obtain an optimal resource allocation plan; according to the optimal resource allocation plan, allocate the application's tasks to the corresponding nodes for execution, and monitor the resource usage during the task execution in real time, and trigger a task reallocation mechanism when resource competition is detected.

[0157] According to a third aspect of the embodiments of the present invention,

[0158] An electronic device is provided, comprising:

[0159] processor;

[0160] a memory for storing processor-executable instructions;

[0161] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0162] A fourth aspect of the embodiments of the present invention is:

[0163] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0164] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A YARN cluster node resource dynamic adjustment method based on dual-mode perception, characterized in that: include: Collect system indicator data and application indicator data of YARN cluster nodes. The system indicator data includes node CPU utilization, memory usage, network bandwidth occupancy and disk I / O load. The application indicator data includes application execution time, resource application amount, task completion rate and data processing throughput. Constructing a first deep neural network model based on the system indicator data, and constructing a second deep neural network model based on the application indicator data; fusing the output results of the first deep neural network model and the second deep neural network model to obtain a node resource utilization status evaluation result; According to the node resource utilization status evaluation result, a resource dynamic adjustment coefficient is calculated for each node in the YARN cluster, and the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient, and a network bandwidth adjustment coefficient; based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit of the allocatable resources of each node in real time; when the node resource utilization rate exceeds the preset utilization rate threshold, a resource expansion operation is triggered, and the node resource quota is increased through an elastic scaling algorithm; when the node resource utilization rate is lower than the preset utilization rate threshold, a resource recovery operation is triggered, and idle resources are released through a resource recovery algorithm; Receive a resource application request submitted by an application, the resource application request including the required number of CPU cores, memory capacity and network bandwidth requirements; construct a multi-dimensional resource allocation optimization model according to the real-time resource status of each node and the resource application request; solve the multi-dimensional resource allocation optimization model using a genetic algorithm to obtain an optimal resource allocation solution; According to the optimal resource allocation scheme, the application's tasks are allocated to corresponding nodes for execution, and the resource usage during task execution is monitored in real time. When resource competition is detected, the task reallocation mechanism is triggered.

2. The method according to claim 1, characterized in that A first deep neural network model is constructed based on the system indicator data, and a second deep neural network model is constructed based on the application indicator data; the output results of the first deep neural network model and the second deep neural network model are fused to obtain a node resource utilization status evaluation result including: Normalizing the system indicator data to obtain normalized system indicator data; inputting the normalized system indicator data into a first deep neural network, wherein the first deep neural network includes an input layer, three hidden layers, and an output layer, wherein the input layer includes four neurons, the number of neurons in the three hidden layers is sixteen, eight, and four, respectively, the output layer includes one neuron, and the output of the first deep neural network is a system resource load score; Performing feature engineering processing on the application indicator data, inputting the application indicator data after the feature engineering processing into a second deep neural network, the second deep neural network includes an input layer, four hidden layers and an output layer, the input layer includes six neurons, the number of neurons in the four hidden layers is twenty-four, sixteen, eight and four respectively, a batch normalization layer is set after each of the hidden layers, the output layer includes one neuron, and the output of the second deep neural network is an application performance score; Based on the system resource load score and the application performance score, an attention mechanism is used to calculate a weight coefficient; the weight coefficient is combined with the system resource load score and the application performance score to construct a fused input vector; the fused input vector is input into a multilayer perceptron, wherein the multilayer perceptron includes two fully connected layers, and the number of neurons in the two fully connected layers is eight and four, respectively; the output of the multilayer perceptron is subjected to a sigmoid activation function to obtain a final evaluation score; the final evaluation score is subjected to exponential moving average smoothing to obtain a node resource utilization status evaluation result.

3. The method according to claim 1, characterized in that Calculate the resource dynamic adjustment coefficient for each node in the YARN cluster according to the node resource utilization status evaluation result; Based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit of the allocatable resources of each node in real time, including: Collecting node resource historical usage data, calculating resource usage variance based on the node resource historical usage data, and calculating resource usage fluctuation coefficient according to the resource usage variance; inputting the node resource historical usage data into a preset evaluation model to obtain a node resource utilization status evaluation result; calculating a resource dynamic adjustment coefficient according to the node resource utilization status evaluation result and the resource usage fluctuation coefficient, wherein the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient, and a network bandwidth adjustment coefficient; wherein the CPU adjustment coefficient is calculated according to the ratio of the node resource utilization status evaluation result to a preset evaluation threshold, the memory adjustment coefficient is calculated according to the product of the node resource utilization status evaluation result and the memory real-time utilization rate, and the network bandwidth adjustment coefficient is calculated according to the product of the node resource utilization status evaluation result and the bandwidth real-time utilization rate; Construct a multi-layer sliding time window, wherein the multi-layer sliding time window includes a main window and a sub-window, wherein the main window size is sixty minutes, the sub-window size is five minutes, and the sliding step length of the main window and the sub-window is one minute; divide the node resource historical usage data according to the sub-windows, and perform time-weighted aggregation on the resource usage data in each of the divided sub-windows to obtain a sub-window aggregation value; combine the sub-window aggregation values ​​according to the main window, and calculate the resource usage trend value based on the combined multiple sub-window aggregation values; The node benchmark resource upper limit is calculated according to the resource dynamic adjustment coefficient, the resource usage fluctuation coefficient, and the resource usage trend value, wherein the node benchmark resource upper limit includes the CPU core number upper limit, the memory capacity upper limit, and the bandwidth upper limit; the node benchmark resource upper limit is updated by exponential smoothing to obtain a smoothed resource upper limit; the smoothed resource upper limit is constrained according to a preset minimum reserved resource ratio and a maximum adjustment range to obtain a constrained resource upper limit; the resource allocation compensation factor is calculated based on historical resource allocation data, and the resource allocation compensation factor and the constrained resource upper limit are weighted to obtain a final resource upper limit, and the final resource upper limit is used to determine a final allocatable resource upper limit.

4. The method according to claim 1, characterized in that: When the node resource utilization exceeds the preset utilization threshold, the resource expansion operation is triggered. The node resource quota is increased through the elastic scaling algorithm, including: Calculating the comprehensive resource utilization rate based on the node resource utilization rate data, and calculating the upper threshold value of resource utilization rate and the lower threshold value of resource utilization rate according to the mean and standard deviation of the historical comprehensive resource utilization rate; collecting the comprehensive resource utilization rate data within a preset observation time window, and calculating the resource pressure score based on the comprehensive resource utilization rate data, and the resource pressure score uses a time decay method to perform weighted calculation on the utilization rate exceeding the upper threshold value of resource utilization rate; When the comprehensive resource utilization is greater than the upper resource utilization threshold, the basic expansion capacity is calculated according to the difference between the comprehensive resource utilization and the upper resource utilization threshold; Based on the load growth trend, the basic expansion capacity is corrected to obtain the predicted expansion capacity; the predicted expansion capacity is compared with the preset maximum single-step expansion capacity to determine the step size of the step-by-step expansion; based on the resource pressure score, the time interval between adjacent expansion operations is calculated; and the resource expansion operation is performed according to the step-by-step expansion step size and the time interval.

5. The method according to claim 4, characterized in that The basic expansion capacity is modified based on the load growth trend to obtain the predicted expansion capacity; the predicted expansion capacity is compared with the preset maximum single-step expansion capacity to determine the step length of the step-by-step expansion; Calculate the time interval between adjacent capacity expansion operations based on the resource pressure score; Performing the resource expansion operation according to the step-by-step expansion step and the time interval includes: Collect system load time series data, and calculate the load growth rate based on the load data at adjacent moments; perform periodic analysis on the load time series data, and extract the periodic amplitude coefficient and phase offset parameter; perform weighted calculation on the historical load growth rate using an exponential decay method to obtain a load trend value, and simultaneously calculate the second-order derivative value of the load; Calculate a correction factor according to the load trend value and the second-order derivative value, and multiply the correction factor by the basic expansion capacity to obtain the predicted expansion capacity; calculate a relative expansion upper limit based on the current resource amount, and compare the relative expansion upper limit with the absolute expansion upper limit to obtain the maximum single-step expansion capacity; calculate the optimal expansion step number according to the predicted expansion capacity, the maximum single-step expansion capacity, and the load trend value; An exponential distribution function is used to calculate the expansion capacity of each expansion step, wherein the input parameters of the exponential distribution function include the expansion step sequence number, the predicted expansion capacity, and the optimal number of expansion steps; a basic expansion time interval is calculated based on the resource pressure score; and the basic expansion time interval is dynamically adjusted according to the expansion step sequence number to obtain an execution time interval of each expansion step; Performing an exponential decay weighted calculation on the load change amplitude within a preset evaluation time window to obtain a stability score; when the stability score meets a preset threshold condition, calculating the resource reservation amount based on the predicted expansion capacity; and performing a step-by-step expansion operation according to the expansion capacity and the execution time interval; Collect the target resource utilization and the actual resource utilization, calculate the mean square error between the target resource utilization and the actual resource utilization to obtain the expansion effect score; according to the difference between the expansion effect score and the preset effect threshold, adaptively adjust the expansion control parameters; the adaptively adjusted expansion control parameters are used to guide the execution of the next round of expansion operations.

6. The method according to claim 4, characterized in that When the node resource utilization rate is lower than the preset utilization rate threshold, the resource recovery operation is triggered. The resource recovery algorithm releases idle resources including: When the comprehensive resource utilization rate is less than the lower threshold of the resource utilization rate, a basic recovery amount is calculated according to the difference between the comprehensive resource utilization rate and the lower threshold of the resource utilization rate; the basic recovery amount is constrained based on the minimum resource guarantee amount and the safety margin to obtain a safe recovery amount; the safe recovery amount is divided into multiple recovery stages, and the recovery step length of each recovery stage is calculated; the time delay of each recovery stage is calculated based on the recovery stage sequence number; and the resource recovery operation is performed according to the recovery step length and the time delay; The operation cooling time is calculated based on the change amplitude of resource utilization; the state migration probability of the new and old resource states is calculated; the operation priority is calculated based on resource pressure, cost savings, and operation risk; and the resource expansion operation and the resource recovery operation are coordinated and controlled according to the operation cooling time, the state migration probability, and the operation priority.

7. The method according to claim 1, characterized in that According to the real-time resource status of each node and the resource application request, a multi-dimensional resource allocation optimization model is constructed; the multi-dimensional resource allocation optimization model is solved by a genetic algorithm to obtain an optimal resource allocation solution including: Collecting real-time resource status data of each node, the real-time resource status data includes computing resource utilization, storage resource utilization, and network resource utilization; collecting resource application requests, the resource application requests include computing resource requirements, storage resource requirements, and network resource requirements; constructing a comprehensive objective function based on the real-time resource status data, the comprehensive objective function includes a resource utilization balance objective function, a response time minimization objective function, and an energy efficiency optimization objective function; setting resource capacity constraints based on the resource application requests and requesting to satisfy the constraints to obtain a resource allocation optimization model; The computing resource requirements, storage resource allocation, and network resource allocation in the resource allocation optimization model are encoded as chromosome gene sequences; an initial population that satisfies constraint conditions is randomly generated according to the chromosome gene sequences; a comprehensive evaluation function is constructed by weighted combination of the comprehensive objective function and adding constraint violation penalty items; and a fitness value is obtained by calculating the comprehensive evaluation function value corresponding to each individual in the initial population; Calculating the selection probability of an individual based on the fitness ranking of the individual in the population; performing a tournament selection operation according to the selection probability to obtain a high-quality individual; calculating the difference ratio between the fitness value of the high-quality individual and the maximum fitness value; calculating the adaptive crossover rate according to the difference ratio, and performing a crossover operation on the high-quality individual based on the adaptive crossover rate to obtain an offspring individual; Calculate the dynamic mutation rate according to the current iteration generation; perform mutation operation on the offspring individuals based on the dynamic mutation rate to obtain mutant individuals; select individuals whose fitness values ​​are greater than a preset elite threshold from the mutant individuals as elite individuals; calculate the population diversity index of the mutant individuals; when the population diversity index is less than the preset diversity threshold, determine the neighborhood search range based on the initial search radius and the attenuation coefficient, and perform local search within the neighborhood search range to obtain the local optimal individual; Calculate the degree of violation of the constraint conditions by the local optimal individual to obtain the violation degree; perform feasibility correction on the local optimal individual according to the violation degree to obtain a corrected individual; smooth the corrected individual and the historical optimal solution to obtain a smoothed individual; calculate the quality metric value and the stability metric value of the solution of the smoothed individual; calculate the evaluation index of the solution based on the quality metric value and the stability metric value; determine whether to update the optimal solution according to the evaluation index of the solution; when the number of iterations reaches a preset termination condition, output the optimal solution as an optimal resource allocation plan, and the optimal resource allocation plan includes the computing resource allocation amount, storage resource allocation amount, and network resource allocation amount of each node.

8. A YARN cluster node resource dynamic adjustment system based on dual-mode perception, used to implement the method as described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect system indicator data and application indicator data of YARN cluster nodes, wherein the system indicator data includes node CPU utilization, memory usage, network bandwidth occupancy and disk I / O load, and the application indicator data includes application execution time, resource application amount, task completion rate and data processing throughput; Constructing a first deep neural network model based on the system indicator data, and constructing a second deep neural network model based on the application indicator data; fusing the output results of the first deep neural network model and the second deep neural network model to obtain a node resource utilization status evaluation result; The second unit is used to calculate the resource dynamic adjustment coefficient for each node in the YARN cluster according to the node resource utilization status evaluation result, and the resource dynamic adjustment coefficient includes a CPU adjustment coefficient, a memory adjustment coefficient and a network bandwidth adjustment coefficient; based on the resource dynamic adjustment coefficient, a sliding time window mechanism is adopted to update the upper limit of the allocatable resources of each node in real time; when the node resource utilization exceeds the preset utilization threshold, a resource expansion operation is triggered, and the node resource quota is increased through an elastic scaling algorithm; When the node resource utilization rate is lower than the preset utilization rate threshold, the resource recovery operation is triggered and the idle resources are released through the resource recovery algorithm; The third unit is used to receive a resource application request submitted by an application program, wherein the resource application request includes the required number of CPU cores, memory capacity, and network bandwidth requirements; construct a multi-dimensional resource allocation optimization model according to the real-time resource status of each node and the resource application request; and solve the multi-dimensional resource allocation optimization model using a genetic algorithm to obtain an optimal resource allocation solution; According to the optimal resource allocation scheme, the application's tasks are allocated to corresponding nodes for execution, and the resource usage during task execution is monitored in real time. When resource competition is detected, the task reallocation mechanism is triggered.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images