A heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithm

Abstract

This invention discloses a fusion method for heterogeneous cluster task scheduling based on Q-learning and genetic algorithms, belonging to the field of intelligent scheduling technology. This invention achieves synergistic optimization through the dynamic feedback mechanism of Q-Learning and the global search capability of genetic algorithms: In the initialization phase, the cluster state space and action space are defined, generating multiple Q-value tables and an initial population; during task scheduling, nodes are selected based on a greedy strategy, and reward values ​​are calculated to update the Q-value table; simultaneously, the scheduling scheme is encoded as chromosomes, and a new population is generated through roulette wheel selection, crossover, and mutation to optimize the Q-value table; during iteration, convergence is judged based on changes in Q-values ​​or population fitness, and the strategy is dynamically adjusted. This invention integrates the advantages of both algorithms, avoids local optima, significantly improves task processing efficiency and resource utilization, and supports adaptive strategy updates when the cluster state changes, making it suitable for efficient task scheduling scenarios in large-scale heterogeneous clusters.

Description

Technical Field

[0001] This invention belongs to the field of computer cluster task scheduling technology, and particularly relates to a heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithm. Background Technology

[0002] In heterogeneous cluster environments, task scheduling faces numerous challenges. Traditional scheduling strategies often struggle to simultaneously balance resource utilization efficiency, task execution time, and the dynamic adaptability of the system.

[0003] On the one hand, existing rule-based scheduling algorithms, such as First-Come, First-Served (FCFS) and Shortest Job First (SJF), often cannot adapt to the complexity of heterogeneous clusters because they lack consideration for the heterogeneity of tasks and resources. To address this issue, researchers have proposed various scheduling strategies based on intelligent algorithms, such as genetic algorithms, particle swarm optimization (PSO), and ant colony optimization. For example, the design scheme with patent application number CN202311605127.1 divides the nodes in the cluster into regions and labels them as master nodes. The optimal path to traverse the master nodes is determined by the ant colony optimization algorithm, thereby determining the optimal region and the optimal node. The task to be scheduled is then assigned to the optimal node based on the SDN controller. Such schemes optimize task scheduling through swarm intelligence, but they usually require a large amount of computing resources and may be difficult to adapt quickly in dynamically changing environments.

[0004] On the other hand, the concept of tenants is rarely considered in GPU cluster scheduling. However, cloud vendors often provide GPUs involving multi-tenant resource allocation, making it challenging to ensure tenant fairness while improving cluster efficiency. For example, the design scheme with patent application number CN202310226362.1 involves tenants submitting tasks to a waiting queue, obtaining task information from cluster compute nodes, selecting tasks that meet certain conditions, and scheduling them to appropriate compute nodes based on affinity. Before tenants submit tasks, the GPU is virtualized into fine-grained resources, and global and local schedulers are started. When there are many tenants and complex tasks, the computational cost of determining task priorities and compute node affinity can be high, and the fine-grained partitioning and management of GPU resources increases system complexity, limiting its application effectiveness in large-scale GPU cluster scheduling.

[0005] With the widespread application of heterogeneous clusters in various fields, the demand for efficient and adaptive task scheduling strategies is becoming increasingly urgent. However, existing task scheduling strategies have many shortcomings when facing the complexity and dynamism of heterogeneous clusters. Therefore, a hybrid adaptive task scheduling strategy for heterogeneous clusters based on Q-Learning and genetic algorithms is needed to solve these problems. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a heterogeneous cluster task scheduling fusion method and system based on Q-learning and genetic algorithms, thereby resolving the issues present in the prior art.

[0007] Firstly, to achieve the above objectives, this invention provides a heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithms, comprising the following steps:

[0008] S1. Initialize the Q-Learning algorithm and the genetic algorithm, define the state space and action space of the heterogeneous cluster, wherein the state space includes the resource attribute information of the nodes, and the action space is the set of actions for assigning tasks to different nodes, and generate the initial Q-value table and the genetic algorithm population.

[0009] S2. Receive the scheduling task, select an action from the Q-value table according to the current cluster status, execute the task allocation and calculate the reward value based on the task execution time and resource utilization, and update the Q-value table.

[0010] S3. Encode the task scheduling scheme into chromosomes and add them to the genetic algorithm population. Perform selection, crossover and mutation operations on the population based on the fitness function to generate a new population and update the Q-value table.

[0011] S4. Repeat the iteration process from S2 to S3, judge the convergence of the algorithm based on the changes in the Q-value table or the population fitness, and determine the final task scheduling strategy.

[0012] Optionally, in S1, the state space includes the node's CPU load, memory load, and network bandwidth, and the initial Q-value table is initialized with uniformly distributed random numbers, and multiple Q-value tables are generated according to the maximum number of schedulable tasks.

[0013] Optionally, in S2, the process of selecting an action from the Q-value table based on the current cluster state, performing task assignment, and calculating reward values ​​based on task execution time and resource utilization includes: the selection of actions using... - Greedy strategy, based on probability Randomly select actions, with probability Select the action with the highest current Q value; the reward value is obtained by weighted summation of the reciprocal of the task execution time and the reciprocal of the resource utilization rate.

[0014] Optionally, in S2, updating the Q-value table includes the following operations: updating the Q-value corresponding to the current state and action based on the Q-value of the current state and action, the reward value obtained, the expected maximum Q-value of the next state, combined with the learning rate and discount factor; and encoding the executed task scheduling scheme into a binary chromosome, wherein the gene position in the chromosome indicates whether the task is assigned to the corresponding node, and adding it to the genetic algorithm population for subsequent evolutionary operations.

[0015] Optionally, in S3, the fitness function is determined based on task execution time and resource utilization. Population evolution includes roulette wheel selection of parent individuals, crossover operation to generate offspring individuals, and mutation operation to introduce new genes. The updated Q-value table is used to optimize subsequent task scheduling strategies.

[0016] Optionally, in S4, the convergence determination process includes the following conditions for determining convergence: the change in Q value is less than a preset change threshold or the change in the optimal fitness of the population is less than a preset fitness threshold within a number of consecutive scheduling cycles.

[0017] Secondly, the present invention also provides a heterogeneous cluster task scheduling fusion system based on Q-learning and genetic algorithms, for implementing a heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithms, the system comprising:

[0018] The initialization module is used to initialize the Q-learning algorithm and the genetic algorithm, define the state space and action space of the heterogeneous cluster, and generate the initial Q-value table and the genetic algorithm population.

[0019] The scheduling module is used to receive scheduling tasks, select actions from the Q-value table according to the current cluster status, execute task allocation, calculate reward values ​​based on task execution time and resource utilization, and update the Q-value table at the same time.

[0020] The genetic algorithm module is used to encode the task scheduling scheme into chromosomes and add them to the population, and to perform selection, crossover and mutation operations on the population based on the fitness function to generate a new population, while updating the Q-value table.

[0021] The convergence judgment module is used to detect changes in the Q-value table or the optimal fitness of the population within a continuous scheduling cycle, and to determine the convergence of the algorithm based on this to determine the final task scheduling strategy.

[0022] Optionally, in the initialization module, the defined state space includes the CPU load, memory load, and network bandwidth of each node, and the initial Q-value table is initialized with uniformly distributed random numbers, while multiple Q-value tables are generated according to the maximum number of schedulable tasks.

[0023] Thirdly, the present invention also provides a computer terminal device, comprising:

[0024] One or more processors;

[0025] A memory, coupled to the processor, for storing one or more programs;

[0026] When the one or more programs are executed by the one or more processors, the one or more processors implement a heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithms.

[0027] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements a heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithms.

[0028] Compared with the prior art, the present invention has the following advantages and technical effects:

[0029] This invention provides a heterogeneous cluster task scheduling fusion method and system based on Q-learning and genetic algorithms. By deeply integrating the reinforcement learning feedback mechanism of Q-Learning with the global search capability of genetic algorithms, this invention overcomes the insufficient adaptability of traditional heterogeneous cluster scheduling strategies in dynamic environments. Q-Learning dynamically selects task allocation actions based on real-time cluster status (such as CPU load, memory load, and network bandwidth) and optimizes the scheduling strategy through a reward function (integrating task execution time and resource utilization), significantly shortening task completion time. The genetic algorithm generates diverse scheduling schemes through crossover and mutation operations, expanding the search space and preventing Q-Learning from getting trapped in local optima. The collaborative iterative optimization of both (Q-value table updates and population evolution) improves the global optimality of the scheduling strategy while reducing the computational cost of a single algorithm. Furthermore, the algorithm dynamically judges convergence based on changes in Q-value or population fitness and restarts the learning process when the cluster state changes significantly, ensuring the adaptive capability of the scheduling strategy. This invention effectively solves the problems of high computational cost of intelligent algorithms, complex multi-tenant resource allocation, and slow response in dynamic environments in existing technologies, and is suitable for the efficient task scheduling needs in large-scale heterogeneous cluster scenarios. Attached Figure Description

[0030] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0031] Figure 1 This is a flowchart of an optional implementation method in Embodiment 1 of the present invention;

[0032] Figure 2 This is a schematic diagram of the overall structure of the Q-Learning-based scheduler according to an embodiment of the present invention;

[0033] Figure 3 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0036] This invention proposes a fusion model that combines the global search capability of genetic algorithms with the adaptive learning capability of Q-Learning, preventing the algorithm from getting trapped in local optima and enabling the system to learn the overall optimal decision through interaction with the environment, thereby solving the complex task scheduling problem of heterogeneous clusters.

[0037] The present invention, combined with the proposed method, can dynamically adjust the scheduling strategy according to the real-time status of the cluster and task requirements to adapt to changes in the environment.

[0038] Example 1

[0039] like Figure 3 As shown, this embodiment provides a heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithms, including:

[0040] S1. Initialize the Q-Learning algorithm and the genetic algorithm, define the state space and action space of the heterogeneous cluster, wherein the state space includes the resource attribute information of the nodes, and the action space is the set of actions for assigning tasks to different nodes, and generate the initial Q-value table and the genetic algorithm population.

[0041] S2. Receive the scheduling task, select an action from the Q-value table according to the current cluster status, execute the task allocation and calculate the reward value based on the task execution time and resource utilization, and update the Q-value table.

[0042] S3. Encode the task scheduling scheme into chromosomes and add them to the genetic algorithm population. Perform selection, crossover and mutation operations on the population based on the fitness function to generate a new population and update the Q-value table.

[0043] S4. Repeat the iteration process from S2 to S3, judge the convergence of the algorithm based on the changes in the Q-value table or the population fitness, and determine the final task scheduling strategy.

[0044] like Figure 1 As shown, this is one possible implementation method:

[0045] 1. Initialize the Q-leraing algorithm, establish a Q-value table to record expected returns, and initialize the genetic evolution algorithm.

[0046] 2. Receive the scheduled task, select the corresponding Q-value table, obtain the current state of the cluster, use the Q-Learning algorithm to select the optimal action, and calculate the reward obtained.

[0047] 3. Based on the rewards obtained in step 2, analyze the statistical data of the reward function and update the Q-value table of the Q-learning algorithm.

[0048] 4. Use a genetic evolution algorithm to perform genetic evolution on the scheduling schemes in the Q-value table, generate a better scheduling scheme, and update the Q-value table used in step 2 to prevent the Q-learning algorithm from getting trapped in local optima.

[0049] 5. Repeat the iterative optimization process until the scheduling strategy is stable and meets the predetermined performance indicators.

[0050] As one implementation method in this embodiment, in S1, the state space includes the CPU load, memory load and network bandwidth of the node, the initial Q value table is initialized by uniformly distributed random numbers, and multiple Q value tables are generated according to the maximum number of schedulable tasks.

[0051] Specifically, step S1 includes:

[0052] 1. Strategy initialization:

[0053] Define the state space and policy space, initialize the parameters of the Q-function and genetic algorithm, and generate the initial population. The specific steps are as follows:

[0054] Step 1-1: Define the state space and action space:

[0055] The state of a heterogeneous cluster is represented as a vector, containing various attribute information of the nodes, such as node... CPU load Memory load Network bandwidth Etc. That is, state space. :

[0056] ;

[0057] The action is to distribute tasks to different nodes. Assuming there are nodes in the cluster... Each node represents a specific action space. :

[0058] ;

[0059] in This indicates that the task will be assigned to a node. , .

[0060] Step 1-2: Initialize the Q-value table:

[0061] set up For state Take action below The Q value, initially Set in the interval Initialize the Q-value table with uniformly distributed random numbers. Based on the maximum number of tasks r that can be scheduled simultaneously, initialize r Q-value tables to improve the scheduling algorithm's targeting for specific tasks.

[0062] Steps 1-3: Initialize the genetic algorithm parameters:

[0063] Determine the number of individuals (i.e., the task scheduling scheme) in the initial population of the genetic algorithm, i.e., the population size. Set crossover probability and mutation probability The task scheduling scheme is encoded into chromosomes using binary encoding, with each gene bit indicating whether a task is assigned to the corresponding node.

[0064] As one implementation method in this embodiment, in S2, the process of selecting an action from the Q-value table based on the current cluster state, performing task allocation, and calculating a reward value based on task execution time and resource utilization includes: the selection of the action adopts... - Greedy strategy, based on probability Randomly select actions, with probability Select the action with the highest current Q value; the reward value is obtained by weighted summation of the reciprocal of the task execution time and the reciprocal of the resource utilization rate.

[0065] Specifically, step S2 includes:

[0066] 2. Iterative optimization of scheduling strategies:

[0067] The overall structure of the Q-Learning-based scheduler is as follows: Figure 2 As shown, the specific steps in each task scheduling cycle are as follows:

[0068] Step 2-1: Observe the current state :

[0069] Receive scheduling tasks and select the corresponding Q-value table based on the number of tasks to be scheduled simultaneously. Obtain the current state information of the heterogeneous cluster and determine its current state space. The state in .

[0070] Step 2-2: Select actions based on Q-Learning :

[0071] Calculate each action Q value in the current state ,use - Greedy strategy selects actions based on probability. Randomly select an action, with probability Choose the action with the maximum Q value. That is: if ,but ,in From the action space A random action is selected from the options; otherwise, ,in This indicates that the Q value is... The value of the independent variable when it reaches its maximum value.

[0072] Steps 1-3: Execute task scheduling and obtain rewards :

[0073] Task scheduling is performed according to the selected action, that is, tasks are assigned to the corresponding nodes. Based on task execution time... and resource utilization rate To define the reward function, that is:

[0074] ;

[0075] in and These are weighting coefficients used to balance the importance of task execution time and resource utilization in the reward. The reward is calculated by observing the task execution results. .

[0076] As one implementation method in this embodiment, in S2, updating the Q-value table includes the following operations: based on the Q-value of the current state and action, the obtained reward value, the expected maximum Q-value of the next state, combined with the learning rate and discount factor, updating the Q-value corresponding to the current state and action; and encoding the executed task scheduling scheme into a binary chromosome, wherein the gene position in the chromosome indicates whether the task is assigned to the corresponding node, and adding it to the genetic algorithm population for subsequent evolutionary operations.

[0077] Specifically, the process of updating the Q-value table in step S2 includes:

[0078] 3. Update the Q-value table:

[0079] Step 3-1: Update the Q function and add the task scheduling scheme to the genetic algorithm population;

[0080] Update the Q value according to the Q-Learning update formula, that is:

[0081] ;

[0082] in, It's the learning rate. It is a discount factor. It is to perform an action The next state after that, Indicates the execution of an action The next state reached after Below, action space All possible actions. It is taken in the state The maximum value among all possible Q-values ​​is used to calculate the update of the current state-action pair. The Q value.

[0083] The executed task scheduling scheme is encoded into chromosome form and added as an individual to the population of the genetic algorithm.

[0084] As one implementation method in this embodiment, in S3, the fitness function is determined based on task execution time and resource utilization. Population evolution includes roulette wheel selection of parent individuals, crossover operation to generate offspring individuals, and mutation operation to introduce new genes. The updated Q-value table is used to optimize subsequent task scheduling strategies.

[0085] Specifically, step S3 includes:

[0086] 4. Use genetic algorithms to optimize the scheduling strategy:

[0087] Step 4-1: Calculate the fitness of each individual in the population:

[0088] Based on task execution time and resource utilization rate Define the fitness function, i.e.:

[0089] ;

[0090] in and These are weighting coefficients. The fitness of each individual in the population (i.e., the task scheduling scheme) is calculated based on the fitness function.

[0091] Step 4-2: Selection, crossover, and mutation to generate a new population:

[0092] Based on individual fitness, a roulette wheel selection strategy is used to select a subset of individuals from the population to serve as parents. Individuals with higher fitness have a greater probability of being selected. The probability of an individual being selected is directly proportional to its fitness. That is:

[0093] ;

[0094] in The probability of an individual being selected. For individual fitness, It is the sum of the fitness of all individuals in the population.

[0095] Selected parent individuals are processed according to crossover probability. Crossover operations can be performed to combine the superior genes of parent individuals, producing offspring individuals with new gene combinations and increasing population diversity.

[0096] After crossover, individuals are classified according to their mutation probability. Mutation operations can introduce new genes, preventing the population from converging to a local optimum prematurely and maintaining the population's diversity and search capabilities.

[0097] After selection, crossover, and mutation operations, a new population of task scheduling schemes is generated, the corresponding Q-value table is updated, and the scheduling strategy is optimized.

[0098] As one implementation method in this embodiment, in S4, the conditions for determining convergence during the convergence determination process include: the change in Q value is less than a preset change threshold or the change in the optimal fitness of the population is less than a preset fitness threshold within a consecutive number of scheduling cycles.

[0099] Specifically, step S4 includes:

[0100] 5. Continuous optimization and convergence:

[0101] Step 5-1: Repeat the iteration:

[0102] By continuously repeating the above scheduling cycle, the Q-Learning algorithm learns better task scheduling strategies as the number of iterations increases, and the genetic algorithm also continuously evolves the population, providing more potential excellent task scheduling schemes.

[0103] Step 5-2: Convergence Judgment:

[0104] If continuous The change in Q value within a scheduling period is less than a threshold. Or, the change in fitness of the best individual in the population is less than a threshold. If the algorithm converges, then it is considered to have converged.

[0105] Step 5-3: Final Decision Determination:

[0106] Once the algorithm converges, the final task scheduling strategy can be determined based on the converged Q-value or the optimal task allocation scheme in the population. In practical applications, the cluster state can be continuously monitored, and the learning and evolution process can be restarted when the state changes significantly to maintain the adaptive capability of the task scheduling strategy.

[0107] In summary, existing technologies lack a solution that combines reinforcement learning with other techniques to improve task scheduling in heterogeneous clusters. This invention utilizes a genetic algorithm to provide diverse initial solutions, expanding the search space and preventing reinforcement learning from getting trapped in local optima. Furthermore, the feedback mechanism of reinforcement learning helps the genetic algorithm better adjust its search direction in subsequent iterations.

[0108] This invention can improve scheduling flexibility and task processing efficiency. By intelligently selecting the optimal node and allocating tasks, it is expected to shorten task completion time and improve the overall task processing efficiency of the cluster.

[0109] Furthermore, this invention combines the global search capability of genetic algorithms with the adaptive learning capability of Q-Learning, and dynamically adjusts the key parameters in the genetic algorithm through the Q-Learning algorithm.

[0110] This invention designs a reward mechanism that calculates rewards based on task scheduling results and uses these rewards to update the Q-value table of Q-Learning. This reward mechanism encourages scheduling strategies to evolve towards the optimization goal.

[0111] Based on this, this invention provides a heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithms. By deeply integrating the global search capability of the genetic algorithm with the adaptive learning capability of Q-Learning, it effectively overcomes the limitations of traditional scheduling strategies in heterogeneous cluster environments. The genetic algorithm provides diverse initial scheduling schemes and expands the search space, avoiding reinforcement learning from getting trapped in local optima. Meanwhile, the real-time feedback mechanism of Q-Learning dynamically adjusts the search direction of the genetic algorithm, enhancing the flexibility of the strategy. By designing a reward mechanism based on task execution time and resource utilization, the scheduling strategy is guided to continuously optimize, significantly shortening task completion time and improving the overall efficiency of the cluster. Furthermore, after the algorithm converges, the learning and evolution process can be restarted according to the real-time cluster state, ensuring the adaptability of the scheduling strategy in dynamic environments, thereby achieving a synergistic improvement in resource utilization and task processing efficiency in complex heterogeneous cluster scenarios.

[0112] Example 2

[0113] In this embodiment, a computer terminal device is provided, including:

[0114] One or more processors;

[0115] A memory, coupled to the processor, for storing one or more programs;

[0116] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods in the above embodiments.

[0117] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in the above embodiments.

[0118] In this embodiment, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the methods described in the above embodiments.

[0119] The aforementioned program can run on a processor or be stored in memory (or a computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0120] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented using different modules, and different steps can be implemented using different modules.

[0121] This embodiment provides such an apparatus or system. The system, referred to as a heterogeneous cluster task scheduling fusion system based on Q-learning and genetic algorithms, includes:

[0122] The initialization module is used to initialize the Q-learning algorithm and the genetic algorithm, define the state space and action space of the heterogeneous cluster, and generate the initial Q-value table and the genetic algorithm population.

[0123] The scheduling module is used to receive scheduling tasks, select actions from the Q-value table according to the current cluster status, execute task allocation, calculate reward values ​​based on task execution time and resource utilization, and update the Q-value table at the same time.

[0124] The genetic algorithm module is used to encode the task scheduling scheme into chromosomes and add them to the population, and to perform selection, crossover and mutation operations on the population based on the fitness function to generate a new population, while updating the Q-value table.

[0125] The convergence judgment module is used to detect changes in the Q-value table or the optimal fitness of the population within a continuous scheduling cycle, and to determine the convergence of the algorithm based on this to determine the final task scheduling strategy.

[0126] As one implementation method in this embodiment, the initialization module defines a state space including the CPU load, memory load, and network bandwidth of each node, and the initial Q-value table is initialized with uniformly distributed random numbers, while multiple Q-value tables are generated according to the maximum number of schedulable tasks.

[0127] As one implementation method in this embodiment, the scheduling module selects an action from the Q-value table based on the current cluster state to perform task allocation, and calculates a reward value based on task execution time and resource utilization. The action selection adopts an ε-greedy strategy, that is, randomly selects an action with probability ε, and selects the action with the largest current Q value with probability 1-ε. The reward value is obtained by weighted summation of the reciprocal of the task execution time and the reciprocal of the resource utilization.

[0128] As one implementation method in this embodiment, the process of updating the Q-value table in the scheduling module includes: updating the corresponding Q-value based on the Q-value of the current state and the selected action, the obtained reward value and the expected maximum Q-value of the next state, and combining the learning rate and discount factor; at the same time, encoding the executed task scheduling scheme into a binary chromosome, wherein each gene position in the chromosome indicates whether the task is assigned to the corresponding node, and adding the chromosome to the population of the genetic algorithm module for subsequent evolutionary operations.

[0129] As one implementation method in this embodiment, in the genetic algorithm module, the fitness function is determined based on task execution time and resource utilization. The population evolution operation includes: selecting parent individuals using roulette wheel selection, generating offspring individuals through crossover, and introducing new genes through mutation. The updated Q-value table is used to further optimize the task scheduling strategy.

[0130] As one implementation method in this embodiment, the convergence judgment module includes the following conditions for system convergence: within a number of consecutive scheduling cycles, the change of each Q value in the Q value table is less than a preset change threshold, or the change of the optimal fitness in the population is less than a preset fitness threshold.

[0131] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.

[0132] The above implementation addresses the problem of integrating Q-learning and genetic algorithms in heterogeneous cluster task scheduling in related technologies. This invention effectively overcomes the limitations of traditional scheduling strategies in heterogeneous cluster environments by deeply integrating the global search capability of the genetic algorithm with the adaptive learning capability of Q-Learning. The genetic algorithm provides diverse initial scheduling schemes and expands the search space, preventing reinforcement learning from getting trapped in local optima. Meanwhile, the real-time feedback mechanism of Q-Learning dynamically adjusts the search direction of the genetic algorithm, enhancing the flexibility of the strategy. By designing a reward mechanism based on task execution time and resource utilization, the scheduling strategy is continuously optimized, significantly shortening task completion time and improving overall cluster efficiency. Furthermore, after convergence, the learning and evolution process can be restarted based on the real-time cluster state, ensuring the adaptability of the scheduling strategy in dynamic environments. This achieves a synergistic improvement in resource utilization and task processing efficiency in complex heterogeneous cluster scenarios.

[0133] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithm, characterized in that, Includes the following steps: S1. Initialize the Q-Learning algorithm and the genetic algorithm, define the state space and action space of the heterogeneous cluster, wherein the state space includes the resource attribute information of the nodes, and the action space is the set of actions for assigning tasks to different nodes, and generate the initial Q-value table and the genetic algorithm population. In S1, the state space includes the node's CPU load, memory load, and network bandwidth. The initial Q-value table is initialized with uniformly distributed random numbers, and multiple Q-value tables are generated based on the maximum number of schedulable tasks. S2. Receive the scheduling task, select an action from the Q-value table according to the current cluster status, perform task allocation and calculate the reward value based on task execution time and resource utilization, and update the Q-value table. In S2, the process of selecting an action from the Q-value table based on the current cluster state, performing task assignment, and calculating reward values ​​based on task execution time and resource utilization includes: action selection using... - Greedy strategy, based on probability Randomly select actions, with probability Select the action with the highest current Q value; the reward value is obtained by weighted summation of the reciprocal of the task execution time and the reciprocal of the resource utilization rate; In S2, the Q value is updated according to the Q-Learning update formula, that is: ； in, It's a reward value. It's the learning rate. It is a discount factor. It is to perform an action The next state after that, Indicates the execution of an action The next state reached after Below, action space All possible actions; It is taken in the state The maximum value among all possible Q-values ​​is used to calculate the update of the current state-action pair. Q value; S3. Encode the task scheduling scheme into chromosomes and add them to the genetic algorithm population. Perform selection, crossover and mutation operations on the population based on the fitness function to generate a new population and update the Q-value table. In S3, the fitness function is obtained by weighted summation of the reciprocal of task execution time and the reciprocal of resource utilization. S4. Repeat the iteration process from S2 to S3, judge the convergence of the algorithm based on the changes in the Q-value table or the population fitness, and determine the final task scheduling strategy.

2. The method according to claim 1, characterized in that, In S2, updating the Q-value table includes the following operations: based on the Q-value of the current state and action, the reward value obtained, the expected maximum Q-value of the next state, and in combination with the learning rate and discount factor, update the Q-value corresponding to the current state and action. The executed task scheduling scheme is encoded into a binary chromosome, where gene positions in the chromosome indicate whether a task is assigned to a corresponding node. This chromosome is then added to the genetic algorithm population for subsequent evolutionary operations.

3. The method according to claim 1, characterized in that, In S3, the fitness function is determined based on task execution time and resource utilization. Population evolution includes roulette wheel selection of parent individuals, crossover operation to generate offspring individuals, and mutation operation to introduce new genes. The updated Q-value table is used to optimize subsequent task scheduling strategies.

4. The method according to claim 1, characterized in that, In S4, the convergence determination process includes the following conditions for convergence: the change in Q value is less than a preset change threshold or the change in the optimal fitness of the population is less than a preset fitness threshold within a number of consecutive scheduling cycles.

5. A heterogeneous cluster task scheduling fusion system based on Q-learning and genetic algorithm, characterized in that, The system for implementing the method as described in any one of claims 1-4 comprises: The initialization module is used to initialize the Q-learning algorithm and the genetic algorithm, define the state space and action space of the heterogeneous cluster, and generate the initial Q-value table and the genetic algorithm population. The scheduling module is used to receive scheduling tasks, select actions from the Q-value table according to the current cluster status, execute task allocation, calculate reward values ​​based on task execution time and resource utilization, and update the Q-value table at the same time. The genetic algorithm module is used to encode the task scheduling scheme into chromosomes and add them to the population, and to perform selection, crossover and mutation operations on the population based on the fitness function to generate a new population, while updating the Q-value table. The convergence judgment module is used to detect changes in the Q-value table or the optimal fitness of the population within a continuous scheduling cycle, and to determine the convergence of the algorithm based on this to determine the final task scheduling strategy.

6. The system according to claim 5, characterized in that, In the initialization module, the defined state space includes the CPU load, memory load and network bandwidth of each node, and the initial Q-value table is initialized with uniformly distributed random numbers, while multiple Q-value tables are generated according to the maximum number of schedulable tasks.

7. A computer terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithm as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the heterogeneous cluster task scheduling fusion method based on Q-learning and genetic algorithm as described in any one of claims 1-4.

Images