Machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method and device, equipment and storage medium

Abstract

The application discloses a dynamic self-adaptive heterogeneous cluster resource scheduling method and device based on machine learning, equipment and storage medium, relates to the technical field of computer resource scheduling, and comprises the following steps: obtaining static hardware indexes of each node in a cluster, normalizing the static hardware indexes, and obtaining a node static image; performing aggregation processing based on the node static image to obtain a resource pool computing power score; extracting resource consumption characteristics of a job according to historical operation data to obtain a job image score; and determining a target resource pool of the job according to the resource pool computing power score and the job image score, so as to realize intelligent matching and scheduling between the job and the resource pool, and improve the matching precision of the job and the node computing power and the resource utilization efficiency in the heterogeneous cluster environment.

Description

Technical Field

[0001] This application relates to the field of computer resource scheduling technology, and in particular to a method, apparatus, device and storage medium for dynamic adaptive heterogeneous cluster resource scheduling based on machine learning. Background Technology

[0002] With the widespread application of large-scale distributed computing platforms in enterprise-level data processing scenarios, the cluster size continues to expand, and the node hardware configuration exhibits obvious heterogeneous characteristics, which puts forward higher requirements for the fineness of resource scheduling.

[0003] Existing cluster scheduling technologies typically allocate jobs based on a unified resource template or static queue partitioning. Scheduling decisions are mainly based on resource availability, lacking quantitative modeling of differences in node computing power and the actual resource requirements of jobs. Resource usage data generated during job execution is mostly used for monitoring or alarms, without forming an effective feedback mechanism for subsequent scheduling decisions, making it difficult to adapt to the fine-grained scheduling needs in heterogeneous node environments.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a method, apparatus, device and storage medium for dynamic adaptive heterogeneous cluster resource scheduling based on machine learning, which aims to solve the technical problem of how to accurately match the job resource requirements and node computing power in heterogeneous clusters.

[0006] To achieve the above objectives, this application proposes a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method, which includes: Obtain the static hardware metrics of each node in the cluster, and normalize the static hardware metrics to obtain a static profile of the node. Based on the static profiles of the nodes, an aggregation process is performed to obtain the resource pool computing power score. Resource consumption characteristics of the operation are extracted based on historical operational data to obtain an operation profile score; Based on the resource pool computing power score and the job profile score, the target resource pool for the job is determined to achieve intelligent matching and scheduling between the job and the resource pool.

[0007] In one embodiment, the step of obtaining static hardware metrics of each node in the cluster and normalizing the static hardware metrics to obtain a static profile of the node includes: The number of CPU cores, memory capacity, and input / output bandwidth of the acquisition node; Based on the number of CPU cores and the maximum number of cores in the cluster, determine the standardized value of the CPU; Based on the aforementioned memory capacity and the maximum cluster memory, a standardized memory value is determined; The input and output bandwidth capabilities are standardized and mapped based on a preset bandwidth level mapping table to obtain standardized input and output values. A static profile of a node is constructed based on the normalized values ​​of the central processing unit, the normalized values ​​of the memory, and the normalized values ​​of the input and output.

[0008] In one embodiment, the step of aggregating the node static profile to obtain the resource pool computing power score includes: Determine the CPU normalization value, memory normalization value, and I / O normalization value for each node in the resource pool; The weighted value of the central processing unit is determined based on the standardized value of the central processing unit and the weight coefficient of the central processing unit; Based on the standardized memory value and memory weight coefficient, the memory weighting value is determined; Based on the standardized input and output values ​​and the input and output weight coefficients, the input and output weight values ​​are determined. The single-node computing power value is obtained by summing the weighted values ​​of the central processing unit, the memory, and the input / output weighted values. The single-node computing power value of all nodes in the pool is averaged to obtain the resource pool computing power score.

[0009] In one embodiment, the step of extracting resource consumption features of the job based on historical operation data to obtain a job profile score includes: Extract CPU usage, execution time, data shuffling volume, and memory usage from historical job execution data; The CPU requirement of the job is determined based on the CPU utilization rate. Determine the memory requirement of the job based on the memory usage; The input and output intensity scores are determined based on the amount of data mixed and washed. The estimated execution time value is determined based on the execution time. The job profile score is obtained by weighting and summing the job's CPU requirements, memory requirements, input / output intensity score, and estimated execution time based on weighted coefficients. The weighted coefficients are determined by a regression model of historical execution data.

[0010] In one embodiment, the step of determining the target resource pool for a job based on the resource pool computing power score and the job profile score, so as to achieve intelligent matching and scheduling between the job and the resource pool, includes: Obtain the resource pool computing power score and job profile score of each resource pool; Calculate the matching degree between the job profile score and the resource pool computing power score of each resource pool; The resource pool with the highest matching degree is determined as the target resource pool for the operation; A job scheduling instruction is sent to the target resource pool so that the target resource pool executes the job.

[0011] In one embodiment, the method further includes: During job execution, the actual operation metrics of the job are collected through the monitoring module; The actual operating indicators are compared with the job profile scores to obtain the deviation analysis results; The weighting coefficients of the job profile score are adjusted based on the deviation analysis results. The weighting coefficients of the resource pool computing power score are adjusted based on the adjusted weighting coefficients. An adjustment strategy for job submission parameters is generated based on the revised weighting coefficients, and the adjustment strategy is applied to subsequent job scheduling.

[0012] In one embodiment, the step of correcting the weight coefficients of the resource pool computing power score based on the adjusted weight coefficients includes: Summarize and analyze the actual operational indicators after the task is completed, and generate a task execution report; Identify the type of deviation between the resource request amount and the actual consumption amount in the job execution report; When the deviation type is CPU request overload, reduce the CPU weight coefficient; When the deviation type is memory allocation excess, reduce the memory weight coefficient; When the deviation type is insufficient input / output bandwidth, increase the input / output weighting coefficient; An adjustment strategy for job submission parameters is generated based on the corrected CPU weight coefficient, memory weight coefficient, and input / output weight coefficient, so as to apply the adjustment strategy for job submission parameters to the resource pool computing power score calculation and job scheduling of subsequent jobs.

[0013] Furthermore, to achieve the above objectives, this application also proposes a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device, the device comprising: The node profile building module is used to obtain the static hardware indicators of each node in the cluster and to normalize the static hardware indicators to obtain the node static profile. The resource pool scoring module is used to perform aggregation processing based on the static profile of the node to obtain the resource pool computing power score. The job profile building module is used to extract resource consumption characteristics of jobs based on historical operation data and obtain job profile scores. The pool selection and matching module is used to determine the target resource pool for a job based on the resource pool's computing power score and the job profile score, so as to achieve intelligent matching and scheduling between jobs and resource pools.

[0014] Furthermore, to achieve the above objectives, this application also proposes a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: By constructing a profiling model of nodes, resource pools, and jobs, the computing power characteristics of nodes and the resource requirements of jobs are described in a structured and quantitative manner. Intelligent pool selection is introduced during the job submission phase to achieve dynamic matching between resource pools and job requirements. Simultaneously, job execution data is incorporated into the scheduling decision-making closed loop to continuously refine job profiles and resource allocation strategies. This shifts scheduling behavior from static configuration to data-driven dynamic optimization, significantly differentiating it from existing scheduling methods that rely on experience templates and static allocation. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method of this application. Figure 3 This is a schematic diagram of the module structure of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device according to an embodiment of this application; Figure 4 This is a schematic diagram of the device structure of the hardware operating environment involved in the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device. The following description uses a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device as an example to illustrate this embodiment and the subsequent embodiments.

[0025] Based on this, embodiments of this application provide a dynamic adaptive heterogeneous cluster resource scheduling method based on machine learning, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device method of this application.

[0026] In this embodiment, the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device method includes steps S10~S40: Step S10: Obtain the static hardware metrics of each node in the cluster, and normalize the static hardware metrics to obtain the static profile of the node. This solution aims to address the problem that existing job scheduling and resource allocation methods in heterogeneous computing cluster environments struggle to accurately match node computing power with job resource requirements. Specifically, due to significant differences in node hardware specifications, a uniform resource template and static allocation strategy result in high-performance nodes being idle while low-performance nodes are overloaded, leading to overall low resource utilization. Furthermore, resource usage data generated during job execution is not effectively used in scheduling decisions, preventing dynamic adjustments to resource allocation based on job characteristics and hindering continuous optimization capabilities.

[0027] It's important to clarify that a cluster refers to a collection of distributed computing resources formed by multiple computing nodes connected through a network; that is, the hardware infrastructure used to perform large-scale data processing tasks. Furthermore, a node refers to a single computing unit within the cluster, i.e., a physical server or virtual instance with independent central processing unit (CPU), memory, and storage resources. Further, static hardware metrics refer to fixed hardware configuration parameters in cluster nodes that do not change with their operating state; these include physical attributes such as the number of CPU cores, memory capacity, and I / O bandwidth. The number of CPU cores refers to the number of independent computing cores in the CPU of a node, a hardware metric that determines the node's parallel computing capability. Memory capacity refers to the total capacity of random access memory (RAM) provided by the node, a measure of hardware resources used for temporary storage of running data and program code. I / O bandwidth refers to the upper limit of the data transfer rate between the node and external storage or network, a hardware metric reflecting the node's data throughput performance. Finally, normalization refers to a standardized calculation process that converts hardware parameters with different dimensions and value ranges into a unified numerical range; that is, it eliminates dimensional differences through proportional calculations to make different metrics comparable. Additionally, a node static profile refers to a structured data set that characterizes the relative computing power level of a node in the entire cluster, i.e., a feature vector containing multiple standardized hardware metrics.

[0028] Understandably, step S10 obtains the static hardware metrics of each node in the cluster and normalizes the static hardware metrics to obtain a static profile of the node. This realizes a unified quantitative representation of the computing power of heterogeneous nodes, provides basic data for subsequent resource pool-level computing power evaluation, and solves the problem of incomparability of heterogeneous hardware.

[0029] In one feasible implementation, step S10 may include steps A11 to A15: Step A11: The number of CPU cores, memory capacity, and input / output bandwidth of the acquisition node; Understandably, step A11 collects the number of CPU cores, memory capacity, and input / output bandwidth of the node, obtaining the raw hardware data required to construct a static profile of the node, and establishing the data foundation for the computational power of the node.

[0030] Step A12: Determine the standardized value of the CPU based on the number of CPU cores and the maximum number of cores in the cluster; It should be noted that the maximum number of cores in the cluster refers to the maximum number of CPU cores across all nodes in the cluster, serving as the baseline and reference standard for standardized calculations. Additionally, the standardized CPU value is the ratio obtained by dividing the number of CPU cores on a node by the maximum number of cores in the cluster; it is a normalized metric representing the relative position of a node's CPU capability within the cluster, with a value ranging from 0 to 1.

[0031] Understandably, step A12 determines the standardized value of the central processing unit (CPU) based on the number of CPU cores and the maximum number of cores in the cluster, converting the CPU capability of a node into a proportional value relative to the highest capability of the cluster, thus achieving the comparability and quantification of CPU capabilities.

[0032] Step A13: Determine the memory normalization value based on the memory capacity and the maximum memory of the cluster; It should be noted that the maximum cluster memory refers to the maximum memory capacity of all nodes in the cluster, which is the baseline and reference standard used for standardized calculations. Additionally, the memory normalization value is the ratio obtained by dividing the node's memory capacity by the maximum cluster memory; it is a normalized index representing the relative position of a node's memory capacity within the cluster, and its value ranges from 0 to 1.

[0033] Understandably, step A13 determines the memory standardization value based on memory capacity and the maximum memory of the cluster, converting the memory capacity of a node into a proportional value relative to the highest capacity of the cluster, thus achieving the comparability and quantification of memory capacity.

[0034] Step A14: Standardize the input and output bandwidth capabilities based on a preset bandwidth level mapping table to obtain standardized input and output values; It should be noted that the preset bandwidth level mapping table refers to a correspondence table that maps different levels of input / output bandwidth capabilities to standardized values, i.e., a predefined conversion rule between discrete bandwidth values ​​and continuous standardized values. In this embodiment, 10 gigabits per second bandwidth is mapped to 0.3, 25 gigabits per second bandwidth is mapped to 0.7, and 40 gigabits per second bandwidth is mapped to 1.0. Furthermore, the input / output standardized value refers to a normalized index representing the relative level of a node's input / output capabilities, obtained through table lookup mapping; that is, a quantified value reflecting the relative position of a node's data transmission performance within the cluster, with a value range between 0 and 1.

[0035] Understandably, step A14 standardizes the input and output bandwidth capabilities based on a preset bandwidth level mapping table to obtain standardized input and output values, converting the node's input and output capabilities into standardized values ​​with unified dimensions, thus achieving comparable quantification of input and output capabilities.

[0036] Step A15: Construct a static profile of the node based on the CPU normalized value, memory normalized value, and I / O normalized value.

[0037] Understandably, step A15 constructs a static profile of the node based on the standardized values ​​of the central processing unit, memory, and input / output, forming a structured data set containing three standardized indicators to characterize the relative computing power level of the node and provide standardized input for resource pool-level aggregate computing.

[0038] In its implementation, to quantify the differences in computing power between heterogeneous nodes, this invention performs standardized modeling of the hardware capabilities of cluster nodes.

[0039] By collecting static hardware metrics such as the number of CPU cores, memory capacity, and I / O capabilities of nodes, a node computing power feature vector is constructed to characterize the relative computing power level of the node in the entire cluster.

[0040] To achieve comparability between different nodes, this invention normalizes the above indicators and defines a static node profile, the mathematical expression of which is shown below.

[0041]

[0042] in, The number of node cores divided by the maximum number of cores in the cluster; Node memory ÷ maximum cluster memory; Normalized values ​​for node I / O bandwidth (10 / 25 / 40Gbps → 0.3 / 0.7 / 1.0).

[0043] Step S20: Aggregate the data based on the static profiles of the nodes to obtain the resource pool computing power score; It's important to clarify that aggregation processing refers to the computational process of integrating individual node profile data into a resource pool-level overall assessment. This involves summing the capabilities of all nodes within the pool using methods such as weighted summation and averaging to form a comprehensive score. Furthermore, a resource pool refers to a collection of computing resources formed by dividing multiple nodes according to logical affiliation; it's a resource management unit used to handle specific types of job scheduling. More specifically, the resource pool computing power score is a comprehensive quantitative indicator characterizing the relative computing power of a resource pool in the current cluster environment; it's a numerical result obtained through weighted calculations used to compare the carrying capacity of different resource pools.

[0044] Understandably, step S20 performs aggregation processing based on the static profile of the node to obtain the resource pool computing power score, realizing the aggregation evaluation of pool-level computing power capabilities, providing a decision basis for job pool selection, and making it possible to compare computing power between different resource pools.

[0045] In one feasible implementation, step S20 may include steps A21 to A25: Step A21: Determine the CPU normalization value, memory normalization value, and I / O normalization value for each node in the resource pool; Understandably, step A21 determines the CPU normalization value, memory normalization value, and input / output normalization value of each node in the resource pool, and obtains normalized profile data of all nodes in the pool to provide input for subsequent weighted aggregation calculations.

[0046] Step A22: Determine the CPU weighting value based on the CPU normalization value and CPU weighting coefficient; It should be noted that the CPU weight coefficient is a proportional parameter reflecting the importance of the CPU indicator in the overall computing power assessment. In other words, it is a configuration value used to adjust the CPU's contribution, and its priority in overall computing power is reflected by its relative magnitude with other weight coefficients. Additionally, the CPU weighted value is the weighted result obtained by multiplying the standardized CPU value by the CPU weight coefficient; that is, it measures the contribution of the CPU dimension to the overall computing power of a single node.

[0047] Understandably, step A22 determines the CPU weighting value based on the CPU standardized value and CPU weighting coefficient, calculates the CPU dimension's weighted contribution to the computing power of a single node, and reflects the weighting proportion of CPU capability in the comprehensive evaluation.

[0048] Step A23: Determine the memory weighting value based on the memory normalization value and the memory weighting coefficient; It should be noted that the memory weight coefficient is a proportional parameter reflecting the importance of memory metrics in the overall computing power assessment; that is, a configuration value used to adjust the contribution of memory. Its relative magnitude with other weight coefficients reflects the priority of memory in the overall computing power. Additionally, the memory weighted value is the weighted result obtained by multiplying the standardized memory value by the memory weight coefficient; that is, it measures the contribution of the memory dimension to the overall computing power of a single node.

[0049] Understandably, step A23 determines the memory weighting value based on the memory standardization value and memory weighting coefficient, calculates the weighted contribution of the memory dimension to the computing power of a single node, and reflects the weight ratio of memory capability in the comprehensive evaluation.

[0050] Step A24: Determine the input-output weighted values ​​based on the input-output standardized values ​​and input-output weight coefficients; It should be noted that the input / output weight coefficient refers to a proportional parameter reflecting the importance of input / output indicators in the overall computing power assessment. In other words, it is a configuration value used to adjust the contribution of input and output, and its relative magnitude with other weight coefficients reflects the priority of input and output in the overall computing power. Additionally, the input / output weighted value is the weighted result obtained by multiplying the standardized input / output values ​​by the input / output weight coefficient; that is, it measures the contribution of the input / output dimension to the overall computing power of a single node.

[0051] Understandably, step A24 determines the input and output weight values ​​based on the input and output standardized values ​​and input and output weight coefficients, calculates the weighted contribution of the input and output dimensions to the computing power of a single node, and reflects the weight ratio of input and output capabilities in the comprehensive evaluation.

[0052] Step A25: Sum the weighted values ​​of the central processing unit, memory, and input / output to obtain the single-node computing power value. Average the single-node computing power values ​​of all nodes in the pool to obtain the resource pool computing power score.

[0053] It should be noted that the single-node computing power value refers to the comprehensive evaluation value obtained by summing the weighted values ​​of the central processing unit, memory, and input / output, which is a quantitative result representing the overall computing power level of a single node. Conversely, the resource pool computing power score refers to the pool-level comprehensive evaluation value obtained by averaging the single-node computing power values ​​of all nodes in the pool, which is a standardized indicator representing the overall carrying capacity of the resource pool.

[0054] Understandably, step A25 sums the weighted values ​​of the central processing unit, memory, and input / output to obtain the single-node computing power value. The single-node computing power values ​​of all nodes in the pool are averaged to obtain the resource pool computing power score. The pool-level comprehensive computing power evaluation result is formed by weighted summation and averaging, realizing the quantitative comparison of computing power capabilities among resource pools.

[0055] In its specific implementation, based on the construction of node profiles, this invention aggregates node profiles according to the logical resource pool to which the node belongs, forming a resource pool-level computing power profile.

[0056] To comprehensively reflect the overall carrying capacity of the resource pool in terms of CPU, memory, and I / O, this invention performs weighted calculation on the computing power characteristics of nodes within the pool to obtain a resource pool computing power score, the calculation method of which is shown below.

[0057]

[0058] in, The overall computing power score for the resource pool; This represents the number of nodes in the resource pool. , These represent the standardized scores for CPU, memory, and disk I / O of the i-th node, respectively. , , These are weighting coefficients used to reflect the importance of each indicator in the overall computing power assessment.

[0059] The resource pool computing power score is used to characterize the relative computing power of different resource pools in the current cluster environment, providing a basis for decision-making in subsequent job pool selection.

[0060] Step S30: Extract resource consumption characteristics of the operation based on historical operation data to obtain an operation profile score; It should be noted that historical execution data refers to performance records generated during the previous execution of a job, including job execution trajectory information such as CPU utilization, execution duration, data shuffling volume, and memory usage. Furthermore, a job refers to a computational task unit submitted to the cluster for execution, i.e., a program instance with specific resource requirements and execution logic. Further, resource consumption characteristics refer to a quantitative description of the actual occupancy of various hardware resources during job execution, i.e., a set of indicators reflecting the job's resource demand pattern. Additionally, CPU utilization refers to time-series data or statistical values ​​of the actual proportion of CPU utilization during job execution, i.e., a performance indicator reflecting the job's computational intensity. Further, execution duration refers to the length of time a job takes from start to finish, i.e., a measure of the job's time resource requirements. Additionally, data shuffling volume refers to the total amount of intermediate data transmitted across nodes during job execution, i.e., a performance indicator reflecting the job's input / output intensity, typically occurring during the data exchange phase of a distributed computing framework. Finally, memory usage refers to the actual amount of memory space occupied during job execution, i.e., a measure of the job's memory resource requirements. Additionally, job profile scoring refers to a comprehensive quantitative value that describes the characteristics of a job's demand for computing resources, i.e., a structured scoring result used to characterize the resource consumption pattern of a job.

[0061] Understandably, step S30 extracts the resource consumption characteristics of the job based on historical operation data to obtain the job profile score, thereby realizing the quantitative modeling of job resource requirements, providing a basis for matching and scheduling between jobs and resource pools, and enabling scheduling decisions to be based on the actual needs of the job rather than a fixed template.

[0062] In one feasible implementation, step S30 may include steps A31 to A36: Step A31: Extract CPU usage, execution time, data shuffling volume, and memory usage from historical job execution data; Understandably, step A31 extracts CPU usage, execution time, data shuffling volume, and memory usage from historical job execution data to obtain the raw performance data required for job profile scoring, thus establishing a data foundation for quantifying job requirements.

[0063] Step A32: Determine the CPU requirements of the job based on the CPU utilization rate; It should be noted that the CPU requirement value of a job refers to a quantitative indicator that characterizes the degree of CPU dependence of a job, determined based on CPU utilization, i.e., a score that reflects the intensity of the job's computing resource demand.

[0064] Understandably, step A32 determines the CPU demand of the job based on the CPU utilization rate, quantifies the job's dependence on CPU resources, and converts the CPU usage pattern into a standardized demand indicator.

[0065] Step A33: Determine the job's memory requirements based on memory usage; It should be noted that the memory requirement of a job refers to a quantitative indicator that characterizes the degree of memory dependence of a job, determined based on memory usage, and is a score that reflects the intensity of the job's memory resource demand.

[0066] Understandably, step A33 determines the job's memory requirements based on memory usage, quantifies the job's dependence on memory resources, and converts the memory usage pattern into standardized requirement indicators.

[0067] Step A34: Determine the input and output intensity scores based on the data shuffling volume; It should be noted that the input-output intensity score is a quantitative indicator that characterizes the degree of dependence between the input and output of a task, determined based on the amount of data shuffling; that is, a score value that reflects the intensity of the task's data transmission demand.

[0068] Understandably, step A34 determines the input and output strength scores based on the amount of data shuffling, quantifies the job's dependence on input and output bandwidth, and converts the data transmission pattern into standardized demand indicators.

[0069] Step A35: Determine the estimated execution time value based on the execution duration; It should be noted that the estimated execution time value refers to a quantitative indicator that represents the time and resource requirements of the operation, determined based on the execution time, that is, a score value that reflects the expected execution time of the operation.

[0070] Understandably, step A35 determines the estimated execution time value based on the execution duration, quantifies the time resource requirements of the operation, and converts the execution time pattern into standardized requirement indicators.

[0071] Step A36: The job profile score is obtained by weighting and summing the job’s CPU requirements, memory requirements, input / output intensity score and estimated execution time based on the weighting coefficients. The weighting coefficients are determined by a regression model of historical running data.

[0072] It should be noted that the weighting coefficients refer to the importance parameters of each demand indicator in the comprehensive score, determined by a regression model using historical operational data. These values ​​reflect the proportion of contribution of different resource dimensions to the overall characteristics of the operation. Furthermore, the regression model refers to a machine learning model trained using historical data to determine the quantitative relationship between variables; it is an algorithmic tool that uses statistical learning methods to extract the optimal solution for indicator weights from historical execution records. Finally, weighted summation refers to the calculation process of multiplying each demand value by its corresponding weighting coefficient and then summing the results, thus forming the mathematical calculation method for the comprehensive score.

[0073] Understandably, step A36 calculates a weighted sum based on the job's CPU requirements, memory requirements, input / output intensity score, and estimated execution time, using weighted coefficients to obtain a job profile score. The weighted coefficients are determined through a regression model of historical running data, enabling quantitative assessment of job requirements based on machine learning, thus allowing the job profile to be continuously optimized as historical data accumulates.

[0074] In its specific implementation, during the job submission phase, this invention models the resource consumption characteristics of the job based on historical operational data.

[0075] By extracting metrics such as CPU utilization, execution time, Shuffle data volume, and memory usage of a job, a job profile score is generated to describe the job's demand for computing resources.

[0076] Based on this, the present invention determines the target resource pool for a job by calculating the matching relationship between the job profile score and the computing power score of each resource pool, thereby realizing intelligent matching and scheduling between jobs and resource pools.

[0077] To achieve the above matching decision, this invention defines a comprehensive matching degree model between jobs and resource pools, the calculation formula of which is shown below.

[0078]

[0079] in, The CPU requirements of the job; The memory requirements of the job; I / O strength rating; For estimated execution time; weight – Determined through a regression model based on historical data.

[0080] The matching model described above enables dynamic optimization of jobs among multiple resource pools.

[0081] Step S40: Based on the resource pool computing power score and job profile score, determine the target resource pool for the job to achieve intelligent matching and scheduling between the job and the resource pool.

[0082] It should be noted that the target resource pool refers to the logical resource pool with the highest matching degree with the job profile score, that is, the set of computing resources where the job actually runs, and is the output result of the scheduling decision. Furthermore, intelligent matching scheduling refers to the process of automatically selecting the optimal execution environment based on a quantitative comparison of the resource pool's computing power score and the job profile score; that is, dynamically adapting the job to the resource pool by replacing manual experience-based configuration with algorithmic decision-making. Further, the matching degree refers to the similarity or fit between the job profile score and the resource pool's computing power score; that is, a comparative indicator used to quantify the fit between job requirements and resource capabilities.

[0083] Understandably, step S40 determines the target resource pool for the job based on the resource pool computing power score and the job profile score, so as to realize intelligent matching and scheduling between the job and the resource pool. This achieves accurate matching between job requirements and pool-level computing power, avoids the problem of high-performance nodes being occupied by low-load jobs and ordinary nodes being under high load for a long time, and improves the resource utilization efficiency and scheduling rationality in heterogeneous cluster environments.

[0084] In one feasible implementation, step S40 may include steps A41 to A44: Step A41: Obtain the resource pool computing power score and job profile score of each resource pool; Understandably, step A41 obtains the resource pool computing power score and the job profile score of each resource pool and the job to be scheduled, thus obtaining two types of basic data for matching degree calculation, namely the capacity index on the resource pool side and the demand index on the job side.

[0085] Step A42: Calculate the matching degree between the job profile score and the resource pool computing power score of each resource pool; It should be noted that the matching degree refers to the degree of numerical closeness between the job profile score and the resource pool computing power score. It is a comparative indicator used to quantify the degree of fit between job requirements and resource capabilities, and is usually determined by calculating the absolute value or relative proportion of the difference between the two.

[0086] Understandably, step A42 calculates the matching degree between the job profile score and the resource pool computing power score of each resource pool, quantifies the degree of fit between the job and different resource pools, and provides a decision basis for the selection of the optimal resource pool.

[0087] Step A43: Determine the resource pool with the highest matching degree as the target resource pool for the job; Understandably, step A43 determines the resource pool with the highest matching degree as the target resource pool for the job, completing the decision-making process from the set of candidate resource pools to a single optimal choice, and achieving accurate matching between the job and the resource pool.

[0088] Step A44: Send a job scheduling instruction to the target resource pool so that the target resource pool can execute the job.

[0089] It should be noted that job scheduling instructions refer to the instruction information used to trigger the target resource pool to execute jobs, that is, control commands that include the parameters required for job execution and resource configuration requirements.

[0090] Understandably, step A44 sends a job scheduling instruction to the target resource pool so that the target resource pool can execute the job, thus completing the final execution stage of the scheduling decision and transforming the algorithm decision result into actual job execution behavior.

[0091] In its implementation, to avoid the scheduling model from using static parameter configuration for a long time, this invention introduces a runtime feedback calibration mechanism based on SparkListener to drive the dynamic adjustment of job profiles and resource configuration strategies.

[0092] During job execution, the Listener module collects actual operation metrics of the job, including CPU usage, memory consumption, execution time, and Shuffle behavior. After the job is completed, the collected results are summarized and analyzed, and compared with the job profile and resource pool computing power model to identify the deviation between resource requests and actual consumption.

[0093] Based on the above analysis results, this invention modifies the job profile parameters and resource pool capability weights, and generates an adjustment strategy for Spark job submission parameters accordingly, so that subsequent jobs can adopt parameter configurations that better match their resource requirement characteristics during the submission stage, thereby achieving periodic calibration of the profile model and resource configuration strategy.

[0094] Through this feedback mechanism, the scheduling strategy can continuously evolve with changes in job characteristics and cluster operating status, forming a closed-loop optimization process of "collection-analysis-adjustment-re-execution".

[0095] This embodiment provides a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method. By constructing static node profiles, resource pool computing power scores, and job profile scores, it achieves precise matching between job resource requirements and cluster computing power capabilities, avoiding the problems of idle high-performance node computing power and excessive load on low-performance nodes, thus improving resource utilization efficiency in heterogeneous cluster environments. By introducing a runtime feedback calibration mechanism based on a monitoring module, actual job performance indicators are incorporated into the scheduling decision-making closed loop, driving dynamic adjustments to job profile parameters and resource configuration strategies. This shifts scheduling behavior from static configuration to data-driven dynamic optimization, enabling continuous evolution of scheduling strategies. By standardizing input and output bandwidth capabilities according to a preset bandwidth level mapping table, the problem of incomparable node input and output capabilities in heterogeneous network environments is solved. By using a regression model based on historical runtime data to determine weight coefficients, job profile scores can be automatically optimized based on historical execution behavior, demonstrating the application value of machine learning in resource scheduling. By calculating the matching degree between job profile scores and the computing power scores of each resource pool and selecting the resource pool with the highest matching degree, intelligent matching scheduling between jobs and resource pools is achieved, replacing the traditional fixed template configuration method and significantly improving scheduling accuracy and resource utilization rationality.

[0096] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 After step S40, steps S401 to S405 are also included: Step S401: During the job execution process, the actual operation indicators of the job are collected through the monitoring module; It should be noted that the monitoring module refers to the event acquisition component embedded in the computing framework, i.e., a data collection tool used to monitor and record the job execution status in real time. In this embodiment, the monitoring module is a built-in listener of the computing framework. Additionally, actual performance metrics refer to the actual performance data generated during job execution, including dynamic monitoring data such as CPU usage, memory consumption, execution time, and data shuffling behavior.

[0097] Understandably, in step S401, during the job execution process, the actual operation indicators of the job are collected through the monitoring module, establishing a real-time monitoring link from job execution to data collection, providing a real data foundation for subsequent deviation analysis and strategy optimization.

[0098] Step S402: Compare the actual operating indicators with the work profile score to obtain the deviation analysis results; It should be noted that the deviation analysis results refer to the description of the differences obtained by comparing the actual operating indicators with the operational profile scores, that is, the analytical conclusions reflecting the degree of deviation between the resource application forecast and the actual consumption.

[0099] Understandably, step S402 compares the actual operating indicators with the job profile score to obtain the deviation analysis results, which quantifies the difference between the profile prediction and the actual execution, and provides a basis for problem diagnosis for adjusting the weight coefficient.

[0100] Step S403: Adjust the weighting coefficients of the job profile score based on the deviation analysis results; Understandably, step S403 adjusts the weighting coefficients of the job profile score based on the deviation analysis results, so that the job profile score can be dynamically corrected based on actual execution feedback, thereby improving the accuracy of job demand prediction.

[0101] Step S404: Adjust the weight coefficient of the resource pool computing power score according to the adjusted weight coefficient; Understandably, step S404 corrects the weight coefficient of the resource pool computing power score based on the adjusted weight coefficient, and transmits the feedback correction at the job level to the evaluation system at the resource pool level, thereby achieving global optimization of the scheduling strategy.

[0102] Step S405: Generate an adjustment strategy for job submission parameters based on the corrected weight coefficients and apply the adjustment strategy to subsequent job scheduling.

[0103] It should be noted that the job submission parameter adjustment strategy refers to the parameter scheme generated based on the corrected weight coefficients to guide the configuration of subsequent job submissions, which includes resource configuration suggestions including CPU core count configuration, memory size configuration, and input / output optimization parameters.

[0104] Understandably, step S405 generates an adjustment strategy for job submission parameters based on the corrected weight coefficients and applies the adjustment strategy to subsequent job scheduling, forming a closed-loop optimization process of collection, analysis, adjustment, and re-execution, enabling the scheduling strategy to continuously evolve with changes in job characteristics and cluster status.

[0105] In one feasible implementation, step S405 may include steps A51 to A56: Step A51: Summarize and analyze the actual operating indicators after the task is completed, and generate a task execution report; It should be noted that the job execution report is a structured document that summarizes and analyzes the actual operating indicators after the job execution, that is, a summary document that includes statistical values ​​of various performance indicators and conclusions of deviation analysis.

[0106] Understandably, step A51 summarizes and analyzes the actual operating indicators after the operation is completed, generates an operation execution report, and completes the integration and structured presentation of the dispersed monitoring data, providing a systematic data view for the identification of deviation types.

[0107] Step A52: Identify the type of deviation between the resource request amount and the actual consumption amount in the job execution report; It should be noted that resource request quantity refers to the amount of resources requested when the job is submitted, i.e., the predicted or declared resource demand value. Actual consumption refers to the actual amount of resources used during job execution, i.e., the monitored resource usage value. Furthermore, deviation type refers to the nature of the difference between resource request quantity and actual consumption quantity, i.e., the difference category used to guide the direction of weight adjustment.

[0108] Understandably, step A52 identifies the type of deviation between the resource request amount and the actual consumption amount in the job execution report, diagnoses the root cause of the resource allocation problem, and provides a classification basis for targeted weight adjustment.

[0109] Step A53: When the deviation type is CPU request overload, reduce the CPU weight coefficient; Understandably, step A53 reduces the CPU weight coefficient when the deviation type is CPU over-allocation, thereby reducing the influence weight of the CPU dimension in the overall evaluation and suppressing the tendency to over-allocate CPU resources.

[0110] Step A54: When the deviation type is excessive memory allocation, reduce the memory weight coefficient; Understandably, step A54 reduces the memory weight coefficient when the deviation type is excessive memory allocation, thereby reducing the influence weight of the memory dimension in the overall evaluation and suppressing the tendency to over-allocate memory resources.

[0111] Step A55: When the deviation type is insufficient input / output bandwidth, increase the input / output weighting coefficient; Understandably, step A55 increases the input / output weight coefficient when the deviation type is insufficient input / output bandwidth, thereby increasing the influence weight of the input / output dimension in the comprehensive evaluation and strengthening the scheduling preference for input / output capabilities.

[0112] Step A56: Generate an adjustment strategy for job submission parameters based on the corrected CPU weight coefficient, memory weight coefficient, and input / output weight coefficient, so as to apply the job submission parameter adjustment strategy to the resource pool computing power score calculation and job scheduling of subsequent jobs.

[0113] Understandably, step A56 generates an adjustment strategy for job submission parameters based on the corrected central processing unit weight coefficient, the corrected memory weight coefficient, and the corrected input / output weight coefficient. This strategy is then applied to the resource pool computing power score calculation and job scheduling of subsequent jobs, completing the entire closed loop from deviation diagnosis to strategy generation and strategy application, and realizing continuous adaptive optimization of the scheduling strategy.

[0114] This embodiment provides a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method. By collecting actual operational metrics during job execution and comparing them with job profile scores, dynamic feedback calibration of the scheduling strategy is achieved, enabling continuous optimization of resource allocation based on actual execution conditions. By identifying the types of deviations between resource requests and actual consumption and adjusting various weight coefficients accordingly, the problem of significant discrepancies between resource requests and actual usage is solved, avoiding resource waste and under-configuration. By generating adjustment strategies for job submission parameters and applying them to subsequent job scheduling, a closed-loop optimization process of collection, analysis, adjustment, and re-execution is formed, shifting scheduling behavior from static configuration to data-driven dynamic evolution. By reducing the CPU and memory weight coefficients to address deviations of excessive resource requests, the tendency for over-configuration of computing and storage resources is suppressed. By increasing the input / output weight coefficients to address deviations of insufficient input / output bandwidth, the scheduling emphasis on data transmission capabilities is strengthened, improving the execution efficiency of input / output intensive jobs. By applying the corrected weight coefficients to the resource pool computing power score calculation for subsequent jobs, adaptive optimization of the scheduling strategy is achieved, enabling continuous improvement of cluster resource allocation based on changes in job characteristics and the accumulation of historical data.

[0115] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0116] This application also provides a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device, please refer to... Figure 3 The machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device includes: The node profile building module 10 is used to obtain the static hardware indicators of each node in the cluster and to normalize the static hardware indicators to obtain the node static profile. The resource pool scoring module 20 is used to perform aggregation processing based on the static profile of the node to obtain the resource pool computing power score. The job profile construction module 30 is used to extract resource consumption characteristics of jobs based on historical operation data and obtain job profile scores. The pool selection and matching module 40 is used to determine the target resource pool for a job based on the resource pool's computing power score and job profile score, so as to realize intelligent matching and scheduling between jobs and resource pools.

[0117] The machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device provided in this application, employing the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method in the above embodiments, can solve the technical problem of how to accurately match the job resource requirements and node computing power capabilities in a heterogeneous cluster. Compared with the prior art, the beneficial effects of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device provided in this application are the same as those of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method provided in the above embodiments, and other technical features in the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0118] In one embodiment, the node profile construction module 10 is also used to collect the number of CPU cores, memory capacity, and input / output bandwidth of the node. Determine the standardized value of the CPU based on the number of CPU cores and the maximum number of cores in the cluster; Determine the memory standardization value based on memory capacity and maximum cluster memory; The input and output bandwidth capabilities are standardized and mapped based on a preset bandwidth level mapping table to obtain standardized input and output values. A static profile of a node is constructed based on the normalized values ​​of the central processing unit, memory, and input / output.

[0119] In one embodiment, the resource pool scoring module 20 is further used to determine the CPU normalized value, memory normalized value and input / output normalized value of each node in the resource pool; The CPU weighting value is determined based on the CPU's standardized value and CPU weighting coefficient. The memory weighting value is determined based on the memory standardization value and the memory weighting coefficient; The input and output weight values ​​are determined based on the input and output standardized values ​​and input and output weight coefficients; The single-node computing power value is obtained by summing the weighted values ​​of the central processing unit, memory, and input / output. The average of the single-node computing power values ​​of all nodes in the pool is then used to obtain the resource pool computing power score.

[0120] In one embodiment, the job profile construction module 30 is also used to extract the CPU usage, execution time, data shuffling amount and memory usage from historical job execution data; Determine the CPU requirements of the job based on the CPU utilization rate; Determine the memory requirements of the job based on memory usage; The input and output intensity scores are determined based on the amount of data mixed and washed. The estimated execution time value is determined based on the execution duration. The job profile score is obtained by weighting and summing the job's CPU requirements, memory requirements, input / output intensity score, and estimated execution time based on the weighted coefficients. The weighted coefficients are determined by a regression model of historical execution data.

[0121] In one embodiment, the pool selection and matching module 40 is also used to obtain the resource pool computing power score of each resource pool and the job profile score of the job to be scheduled. Calculate the matching degree between the job profile score and the resource pool computing power score of each resource pool; The resource pool with the highest matching degree is determined as the target resource pool for the operation; Send job scheduling instructions to the target resource pool so that the target resource pool can execute the job.

[0122] In one embodiment, the pool matching module 40 is also used to collect the actual operation indicators of the job through the monitoring module during the job execution process; The actual operational indicators are compared with the operational profile scores to obtain the deviation analysis results; Adjust the weighting coefficients of the assignment profile scoring based on the deviation analysis results; The weighting coefficients for the resource pool computing power score are adjusted based on the adjusted weighting coefficients. Based on the revised weighting coefficients, an adjustment strategy for job submission parameters is generated and applied to subsequent job scheduling.

[0123] In one embodiment, the pool matching module 40 is also used to summarize and analyze the actual operating indicators after the job execution and generate a job execution report; Identify the types of discrepancies between resource requests and actual consumption in the job execution report; When the deviation type is CPU request overload, reduce the CPU weight coefficient; When the deviation type is excessive memory allocation, reduce the memory weight coefficient; When the deviation type is insufficient input / output bandwidth, increase the input / output weighting coefficient; An adjustment strategy for job submission parameters is generated based on the revised CPU weight coefficient, memory weight coefficient, and input / output weight coefficient, so that the adjustment strategy for job submission parameters can be applied to the resource pool computing power score calculation and job scheduling of subsequent jobs.

[0124] This application provides a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method in the first embodiment described above.

[0125] The following is for reference. Figure 4 This document illustrates a structural diagram of a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device suitable for implementing embodiments of this application. The machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0126] like Figure 4As shown, the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device with various systems, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0127] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0128] The machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device provided in this application, employing the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method described in the above embodiments, can solve the technical problem of how to accurately match the job resource requirements and node computing power capabilities in a heterogeneous cluster. Compared with the prior art, the beneficial effects of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device provided in this application are the same as those of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device method provided in the above embodiments, and other technical features of this machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0129] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0131] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method in the above embodiments.

[0132] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0133] The aforementioned computer-readable storage medium may be included in a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device; or it may exist independently and not be assembled into a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device.

[0134] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a machine learning-based dynamic adaptive heterogeneous cluster resource scheduling device, the following actions are taken: static hardware indicators of each node in the cluster are acquired and normalized to obtain a static node profile; aggregation processing is performed based on the node static profiles to obtain a resource pool computing power score; resource consumption characteristics of jobs are extracted based on historical running data to obtain a job profile score; and the target resource pool for the job is determined based on the resource pool computing power score and the job profile score to achieve intelligent matching and scheduling between jobs and resource pools.

[0135] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0138] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method. This solves the technical problem of accurately matching job resource requirements with node computing power in a heterogeneous cluster. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method provided in the above embodiments, and will not be repeated here.

[0139] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method described above.

[0140] The computer program product provided in this application can solve the technical problem of how to accurately match the job resource requirements and node computing power in heterogeneous clusters. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method provided in the above embodiments, and will not be repeated here.

[0141] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A dynamic adaptive heterogeneous cluster resource scheduling method based on machine learning, characterized in that, The machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method includes: Obtain the static hardware metrics of each node in the cluster, and normalize the static hardware metrics to obtain a static profile of the node. Based on the static profiles of the nodes, an aggregation process is performed to obtain the resource pool computing power score. Resource consumption characteristics of the operation are extracted based on historical operational data to obtain an operation profile score; Based on the resource pool computing power score and the job profile score, the target resource pool for the job is determined to achieve intelligent matching and scheduling between the job and the resource pool.

2. The method as described in claim 1, characterized in that, The step of obtaining static hardware metrics for each node in the cluster and normalizing these metrics to obtain a static node profile includes: The number of CPU cores, memory capacity, and input / output bandwidth of the acquisition node; Based on the number of CPU cores and the maximum number of cores in the cluster, determine the standardized value of the CPU; Based on the aforementioned memory capacity and the maximum cluster memory, a standardized memory value is determined; The input and output bandwidth capabilities are standardized and mapped based on a preset bandwidth level mapping table to obtain standardized input and output values. A static profile of a node is constructed based on the normalized values ​​of the central processing unit, the normalized values ​​of the memory, and the normalized values ​​of the input and output.

3. The method as described in claim 1, characterized in that, The step of aggregating the static profiles of the nodes to obtain the resource pool computing power score includes: Determine the CPU normalization value, memory normalization value, and I / O normalization value for each node in the resource pool; The weighted value of the central processing unit is determined based on the standardized value of the central processing unit and the weight coefficient of the central processing unit; Based on the standardized memory value and memory weight coefficient, the memory weighting value is determined; Based on the standardized input and output values ​​and the input and output weight coefficients, the input and output weight values ​​are determined. The single-node computing power value is obtained by summing the weighted values ​​of the central processing unit, the memory, and the input / output weighted values. The single-node computing power value of all nodes in the pool is averaged to obtain the resource pool computing power score.

4. The method as described in claim 1, characterized in that, The step of extracting resource consumption characteristics of the operation based on historical operation data to obtain an operation profile score includes: Extract CPU usage, execution time, data shuffling volume, and memory usage from historical job execution data; The CPU requirement of the job is determined based on the CPU utilization rate. Determine the memory requirement of the job based on the memory usage; The input and output intensity scores are determined based on the amount of data mixed and washed. The estimated execution time value is determined based on the execution time. The job profile score is obtained by weighting and summing the job's CPU requirements, memory requirements, input / output intensity score, and estimated execution time based on weighted coefficients. The weighted coefficients are determined by a regression model of historical execution data.

5. The method as described in claim 1, characterized in that, The step of determining the target resource pool for a job based on the resource pool's computing power score and the job profile score, in order to achieve intelligent matching and scheduling between jobs and resource pools, includes: Obtain the resource pool computing power score and job profile score of each resource pool; Calculate the matching degree between the job profile score and the resource pool computing power score of each resource pool; The resource pool with the highest matching degree is determined as the target resource pool for the operation; A job scheduling instruction is sent to the target resource pool so that the target resource pool executes the job.

6. The method as described in claim 1, characterized in that, The method further includes: During job execution, the actual operation metrics of the job are collected through the monitoring module; The actual operating indicators are compared with the job profile scores to obtain the deviation analysis results; The weighting coefficients of the job profile score are adjusted based on the deviation analysis results. The weighting coefficients of the resource pool computing power score are adjusted based on the adjusted weighting coefficients. An adjustment strategy for job submission parameters is generated based on the revised weighting coefficients, and the adjustment strategy is applied to subsequent job scheduling.

7. The method as described in claim 6, characterized in that, The step of generating an adjustment strategy for job submission parameters based on the corrected weighting coefficients and applying the adjustment strategy to subsequent job scheduling includes: Summarize and analyze the actual operational indicators after the task is completed, and generate a task execution report; Identify the type of deviation between the resource request amount and the actual consumption amount in the job execution report; When the deviation type is CPU request overload, reduce the CPU weight coefficient; When the deviation type is memory allocation excess, reduce the memory weight coefficient; When the deviation type is insufficient input / output bandwidth, increase the input / output weighting coefficient; An adjustment strategy for job submission parameters is generated based on the corrected central processing unit weight coefficient, the corrected memory weight coefficient, and the corrected input / output weight coefficient, so as to apply the adjustment strategy for job submission parameters to the resource pool computing power score calculation and job scheduling of subsequent jobs.

8. A dynamic adaptive heterogeneous cluster resource scheduling device based on machine learning, characterized in that, The device includes: The node profile building module is used to obtain the static hardware indicators of each node in the cluster and to normalize the static hardware indicators to obtain the node static profile. The resource pool scoring module is used to perform aggregation processing based on the static profile of the node to obtain the resource pool computing power score. The job profile building module is used to extract resource consumption characteristics of jobs based on historical operation data and obtain job profile scores. The pool selection and matching module is used to determine the target resource pool for a job based on the resource pool's computing power score and the job profile score, so as to achieve intelligent matching and scheduling between jobs and resource pools.

9. A dynamic adaptive heterogeneous cluster resource scheduling device based on machine learning, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the machine learning-based dynamic adaptive heterogeneous cluster resource scheduling method as described in any one of claims 1 to 7.

Images