Resource scheduling method and device, storage medium and electronic equipment

Abstract

The invention provides a resource scheduling method and device, a storage medium and electronic equipment, and relates to the technical field of cloud computing. The method comprises: generating an initial population comprising one or more candidate solutions, each candidate solution representing a resource allocation scheme; performing local search and global search based on the candidate solution of the initial population to determine a candidate solution of the population after iteration; taking the candidate solution of the population after iteration as an initial state of reinforcement learning optimization, and executing a reinforcement learning strategy to obtain a candidate solution after reinforcement learning optimization; and determining a final resource allocation scheme from the candidate solutions of the population after iteration and the candidate solutions after reinforcement learning optimization based on the fitness of the candidate solutions. A wider solution space is explored by utilizing global search capability and a diversity preserving mechanism, a resource allocation strategy is optimized by utilizing environment feedback of a reinforcement learning algorithm and a strategy adjusting mechanism, and a local optimal trap is effectively avoided and the efficiency and effect of resource scheduling are improved through the synergistic effect of the global search capability and the diversity preserving mechanism.

Description

Background Art

[0002] Cloud (or cloud platform) resource scheduling and management have a great impact on system performance. In actual system environments, the demand and availability of computing resources change dynamically, and traditional static resource scheduling methods are difficult to adapt to such changes, resulting in low resource utilization.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a resource scheduling method and device, a storage medium and an electronic device, which can at least to some extent overcome the problem of low resource utilization rate in resource scheduling in related technologies.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a resource scheduling method is provided, comprising: generating an initial population including one or more candidate solutions, each candidate solution representing a resource allocation scheme; performing local search and global search based on the candidate solutions of the initial population to determine the candidate solutions of the iterative population; using the candidate solutions of the iterative population as the initial state of reinforcement learning optimization, executing the reinforcement learning strategy to obtain the candidate solutions after reinforcement learning optimization; and determining a final solution from the candidate solutions of the iterative population and the candidate solutions after reinforcement learning optimization based on the fitness of the candidate solutions as the resource allocation scheme.

[0007] In one embodiment, the method further includes: adding the final solution to the initial population; removing the candidate solution with the lowest fitness from the initial population; and performing iterative processing using the updated initial population as the starting point of the next iteration.

[0008] In one embodiment, the method further comprises: after each iteration is completed, if it is determined that the convergence condition is met, outputting the obtained final solution as the final resource allocation solution.

[0009] In one embodiment, the convergence condition includes: reaching a maximum number of iterations, or an increase in the fitness value being less than a first threshold.

[0010] In one embodiment, the local search and global search based on the candidate solutions of the initial population to determine the candidate solutions of the iterative population includes: determining the fitness of each candidate solution in the initial population; determining the local optimal solution and the global optimal solution based on the candidate solutions of the initial population; and updating the local leader and the global leader based on the local optimal solution and the global optimal solution.

[0011] In one embodiment, the candidate solutions of the population after iteration are used as the initial state of reinforcement learning optimization, and the reinforcement learning strategy is executed to obtain the candidate solutions after reinforcement learning optimization, including: selecting one or more candidate solutions from the candidate solutions of the population after iteration as the initial state of reinforcement learning optimization, denoted as State RL,0 ; For each selected initial state, the reinforcement learning strategy is executed to perform a series of action selection and resource allocation adjustments; in which, at each step, according to the current state State RL,K , select Action RL,K , after execution, the new state State is obtained RL,K+1 and Reward RL,K ; Wherein, K is a natural number; Update the parameter Q value of the RL model according to the reward and state transition.

[0012] In one embodiment, the resource allocation scheme includes a multi-path routing resource allocation scheme.

[0013] According to another aspect of the present disclosure, a resource scheduling device is provided, including: an initial population generation module, used to generate an initial population including one or more candidate solutions, each candidate solution represents a resource allocation scheme for multi-path routing; an iterative population generation module, used to perform local search and global search based on the candidate solutions of the initial population to determine the candidate solutions of the iterative population; a reinforcement learning execution module, used to use the candidate solutions of the iterative population as the initial state of reinforcement learning optimization, and execute the reinforcement learning strategy to obtain the candidate solutions after reinforcement learning optimization; a resource allocation determination module, used to determine the final solution from the candidate solutions of the iterative population and the candidate solutions after reinforcement learning optimization based on the fitness of the candidate solutions, as the resource allocation scheme for multi-path routing.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned resource scheduling method by executing the executable instructions.

[0015] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned resource scheduling method is implemented.

[0016] The resource scheduling method and device, storage medium and electronic device provided by the embodiments of the present disclosure explore a broader solution space by utilizing global search capabilities and diversity retention mechanisms, and optimize resource allocation strategies by utilizing environmental feedback and strategy adjustment mechanisms of reinforcement learning algorithms. Through the synergy of the two, it is possible to effectively avoid falling into the local optimal trap and improve the efficiency and effect of resource scheduling.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 A flow chart showing a resource scheduling method in one embodiment of the present disclosure is shown;

[0020] Figure 2 A flow chart of an SMO algorithm in one embodiment of the present disclosure is shown;

[0021] Figure 3 A schematic diagram of a resource scheduling system in one embodiment of the present disclosure is shown;

[0022] Figure 4 A schematic diagram showing a flow chart of a resource scheduling method in one embodiment of the present disclosure is shown;

[0023] Figure 5 A resource scheduling system framework diagram in one embodiment of the present disclosure is shown;

[0024] Figure 6 A flow chart of an SMO algorithm in one embodiment of the present disclosure is shown;

[0025] Figure 7 A schematic diagram showing a resource scheduling device in one embodiment of the present disclosure; and

[0026] Figure 8 A structural block diagram of a computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0029] The solution provided by this application explores a broader solution space by maintaining local search capabilities and global search capabilities and diversity, and at the same time uses the environmental feedback and strategy adjustment mechanism of the reinforcement learning algorithm to optimize the task scheduling routing strategy. Through the synergy of the two, it effectively avoids falling into the local optimal trap and improves the efficiency and effect of resource scheduling.

[0030] To facilitate understanding, several terms involved in this application are first explained below.

[0031] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0032] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0033] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0034] Below, each step of the resource scheduling method in this example implementation will be described in more detail with reference to the accompanying drawings and embodiments.

[0035] Figure 1 A flowchart of a resource scheduling method in one embodiment of the present disclosure is shown.

[0036] like Figure 1 As shown, S102, an initial population including one or more candidate solutions is generated, and each candidate solution represents a resource allocation scheme.

[0037] S104, local search and global search are performed based on the candidate solutions of the initial population to determine the candidate solutions of the population after iteration. The fitness of the candidate solutions of the population after iteration can be determined.

[0038] S106, taking the candidate solution of the population after iteration as the initial state of reinforcement learning optimization, executing the reinforcement learning strategy to obtain the candidate solution after reinforcement learning optimization. The fitness of the candidate solution after reinforcement learning optimization can be determined. In one embodiment, at each step of executing the reinforcement learning strategy, according to the current state State RL,K , select Action RL,K , after execution, the new state State is obtained RL,K+1 and Reward RL,K , and update the parameter Q value of the reinforcement learning model based on the reward and state transfer.

[0039] S108, based on the fitness of the candidate solutions, a final solution is determined from the candidate solutions of the iterative population and the candidate solutions optimized by reinforcement learning as a final resource allocation solution.

[0040] In the above embodiment, a broader solution space is explored by utilizing global search capability and diversity preservation mechanism, while the environment feedback and strategy adjustment mechanism of the reinforcement learning algorithm are utilized to optimize the resource allocation strategy. Through the synergy of the two, it is effective to avoid falling into the local optimal trap, and improve the efficiency and effect of resource scheduling such as large model training and routing selection. Furthermore, reinforcement learning technology can interact with the environment, learn and adapt to such dynamic changes, and improve the flexibility and efficiency of resource scheduling.

[0041] Figure 2 The flowchart of the SMO (Spider Monkey Optimization) algorithm in one embodiment of the present disclosure is shown. In this embodiment, SMO is a meta-heuristic technology. In the SMO algorithm, each spider monkey (SMO) represents an optional solution. In addition to the initialization phase, the SMO algorithm also includes six phases: a local leader phase, a global leader phase, a local leader learning phase, a global leader learning phase, a local leader decision phase, and a global leader decision phase. Next, combined with Figure 2 Describe each stage.

[0042] like Figure 2 As shown, S201, initializing the population. In the initialization stage, SMO generates an initial population including N spider monkeys.

[0043] S202, evaluating the initialization population.

[0044] In one embodiment, the fitness of each candidate solution in the initial population is calculated.

[0045] S203, identifying a global leader and a local leader.

[0046] The global leader and local leaders are determined based on the fitness of each candidate solution.

[0047] S204, local leader phase (LLP) position update.

[0048] In this phase, spider monkeys have a chance to update themselves. Spider monkeys update their positions based on the experience of their local leader and group members. The fitness value of each spider monkey is calculated at the new position, and if the fitness is higher than its old position, it is updated, otherwise it is not updated.

[0049] In one embodiment, the position update equation is:

[0050] SMnew ij =SM ij +U(0,1)*(LL kj -SM ij )+U(-1,1)*(SM rj-SM ij ) (1)

[0051] Among them, SM ij is the jth dimension of the i-th spider monkey, LL kj represents the jth dimension of the kth group of local leaders, SM rj is the jth dimension of a spider monkey randomly selected from the rth group, with r≠i, U(0,1) is a random number uniformly distributed in the range (0,1), and U(-1,1) is a random number uniformly distributed in the range (-1,1).

[0052] From formula (1), it can be seen that the spider monkeys waiting for position update will be attracted by the local leader while maintaining their persistence. The last part helps to introduce disturbances in the search process and maintain the randomness of the algorithm, thus avoiding premature stagnation.

[0053] S205, global leader phase (GLP) position update.

[0054] After completing the local leader stage, we enter the global leader stage, where candidate solutions are updated based on a certain selection probability.

[0055] In one embodiment, the selection probability is a function of fitness. i The fitness can be calculated according to the following formula:

[0056]

[0057] Selection probability prob i Determined by roulette selection, if fitness i is the fitness of the ith spider monkey, and its probability of being selected in the global leader stage can be calculated by the following formula:

[0058]

[0059] Spider monkeys use the knowledge of the global leader, the experience of neighboring SMs, and their own persistence to update their positions. The position update equation at this stage is:

[0060] SMnew ij =SM ij +U(0,1)*(GL j -SM ij )+U(-1,1)*(SM rj -SM ij ) (4)

[0061] Among them, GL j is the position of the global leader in the jth dimension.

[0062] The above position update equation contains three parts: the first part shows the persistence of the parent (current) spider monkey, the second part attracts the parent spider monkey towards the global leader, and the last one is used to maintain the randomness of the algorithm. In this equation, the second term is used to strengthen the utilization of the determined search space, and the third term is used to avoid premature convergence of the search process and reduce the chance of falling into the local optimum.

[0063] S206, learning through the global leader stage.

[0064] At this stage, the algorithm finds the optimal solution for the entire group and the identified spider monkey is considered the global leader of the group. In addition, the position of the global leader is checked and if it is not updated then the counter associated with the global leader (called global limit count (GLC)) is increased by 1, otherwise it is set to 0. The global limit count of the global leader is checked and compared with the global leader limit.

[0065] S207, learning through the local leader learning phase.

[0066] In this phase of the algorithm, the position of the local leader is updated by greedily selecting members within the group. If the local leader does not update its position, then the counter Local Limit Count(LLC) associated with the local leader is increased by 1; otherwise the counter is set to 0. This process is applied to each group to find its respective local leader. The Local Limit Count is a counter that increments until it reaches a fixed threshold called the Local Leader Limit(LLL).

[0067] S208, local leader decision-making stage.

[0068] Prior to this phase, local and global leaders have been determined. If any local leader is not reorganized to a specific edge, known as local leader restriction, then all members of the group update their positions either by random initialization or using the experience of the global leader through Equation 5. Equation 5 is applied with a probability pr called the perturbation rate.

[0069] SMnew ij =SM ij +U(0,1)*(GL j -SM ij )+U(0,1)*(SM rj -LL kj ) (5)

[0070] From this equation, it can be seen that when the existing local leader is exhausted (no updates for LLL iterations), the solutions of the group are repelled, and the solutions are attracted to the global leader to change the existing search direction and position. Based on pr, some dimensions of the solution are randomly initialized to introduce perturbations in the existing positions of the solution. Here the local leader limit is a parameter to check whether the local leader is trapped in a local minimum, usually calculated as DxN, where D is the dimension and N is the total number of SMs. If LLC is greater than LLL, LLC is set to zero and the SM is initialized as described above to improve the exploration of the search space.

[0071] S209, the global leader decision stage decides whether to split or merge.

[0072] Similar to the local leader decision phase, if the global leader is not reorganized to a specific edge, that is, the global leader limit, then the global leader will divide the group into smaller groups or merge the groups into a single unit group. Here, the global leader limit GLL is a parameter to check whether there is premature, and the value range is generally [n / 2, 2N]. If the global leader count exceeds the global leader limit, then set the count to 0 and compare the number of groups with the maximum number of groups. If the existing number of groups is less than the predefined maximum number of groups, then the global leader will further group, otherwise form a single parent group by combining.

[0073] S210, determining whether a termination condition is satisfied.

[0074] If the termination condition is met, the global leader position is output as the optimal solution, otherwise, jump to S204 to continue iteration.

[0075] SMO provides a better balance between exploitation and exploration while searching for the optimal solution. The local leader phase is used to explore the search area, in this phase all the group members update their positions with high perturbations in the dimension. Whereas the global leader phase promotes exploitation, in this phase better candidates have more chances to update their positions. This feature makes SMO a good candidate among search-based optimization algorithms. SMO also has a built-in stagnation check mechanism. The local leader learning phase and the global leader learning phase are used to check if the search process is stagnant. In case of stagnation (at the local or global level), the decision phases of the local leader and the global leader will work. The local leader decision phase creates an additional exploration, while in the global leader decision phase, a decision is made about fission or fusion. Therefore, SMO provides a better balance between exploration and exploitation while maintaining the convergence speed.

[0076] In one embodiment, SMO mainly has four control parameters: local leader limit (LLL), global leader limit (GLL), maximum number of groups (MG) and perturbation rate pr. In one example, the above parameters are set as follows: MG = N / 10, that is, to ensure that there are at least 10 spider monkeys in the group; GLL∈[N / 2,2N]; LLL=DxN; pr∈[0.1,0.8].

[0077] Figure 3 FIG. 2 shows a schematic diagram of a resource scheduling system in one embodiment of the present disclosure. Figure 3 As shown, in this embodiment, resource scheduling is collaboratively optimized through the SMO algorithm and reinforcement learning.

[0078] SMO intelligent behavior: simulates the decision-making process of spider monkey groups in the natural environment, and solves problems by simulating their foraging, exploration, cooperation and other behaviors. In the field of resource scheduling, by imitating spider monkeys foraging, each "spider monkey" represents a potential resource allocation plan, and by simulating their "behavior", the optimal solution for resource allocation is continuously searched and optimized.

[0079] SMO+RL collaboration: The SMO algorithm is used to generate a series of initial resource allocation plans as input to the RL algorithm. Based on these initial plans, the RL algorithm continuously optimizes the resource allocation strategy through interactive learning with the environment. The two are performed alternately to form a closed-loop optimization process until the predetermined optimization goal is achieved.

[0080] The request processor receives resource service requests from users iaas-1, iaas-2, ... iaas-n, and generates a virtual machine resource request queue based on the resource service requests. The SDN (Software-Defined Networking) module allocates and schedules resources through the SMO intelligent behavior model and reinforcement learning model, obtains an optimized resource allocation plan, and allocates resources in a distributed network.

[0081] In the above embodiment, by collaboratively optimizing resource scheduling using the spider monkey SMO intelligent behavior algorithm and reinforcement learning, the system can respond to environmental changes more quickly, improve the flexibility and accuracy of resource scheduling, and thereby reduce energy consumption and costs while ensuring service quality.

[0082] Figure 4 A schematic diagram of a resource scheduling method in one embodiment of the present disclosure is shown.

[0083] like Figure 4 As shown in the figure, resource scheduling includes the following processing steps:

[0084] 1. Data collection.

[0085] Collect historical resource usage of N virtual machines. Resource usage includes, for example, CPU, memory, storage, network load, etc., as well as current system load and resource demand forecast.

[0086] 2. Data processing.

[0087] The historical data is represented in matrix form, where the rows represent timestamps and the columns represent the resource usage of each virtual machine.

[0088]

[0089] Among them, U ij Indicates the resource usage of the jth virtual machine at the ith timestamp.

[0090] Environment configuration: Set the resource limit to 80%, that is, for each resource k:

[0091] Resource_Limit k =0.8*Total_Resource k (7)

[0092] Some initialization processing is introduced below.

[0093] 3. SMO algorithm.

[0094] 1) Generate initial solution: randomly generate a set of virtual machine resource allocation schemes, which can be represented as an N-dimensional vector, where each element of the N-dimensional vector represents the resource allocation ratio of the corresponding virtual machine.

[0095]

[0096] Among them, a i Indicates the resource allocation ratio of the i-th virtual machine.

[0097] 2) Define the fitness function.

[0098] In one embodiment, the fitness function is defined as a weighted sum of the inverse of resource usage and system performance.

[0099]

[0100] Among them, Resource_Usage is the resource usage under the current resource allocation, λ is the performance weight coefficient, and System_Performance is the system performance indicator (such as throughput, latency, etc.).

[0101] 4. RL model design:

[0102] 1) State space: represents the combination of the current virtual machine resource allocation and system load.

[0103] State=[a1,a2 …a N ,l1,l2 …l M ,] (10)

[0104] 2) Action space Action: An action is to increase / decrease the resource allocation of a virtual machine, which can be represented as an N-dimensional vector, where each element represents the increase or decrease in the resource allocation of the corresponding virtual machine.

[0105]

[0106] 3) Reward function: designed to encourage reducing the risk of resource overload and improving system performance.

[0107]

[0108] Among them, rt_reward, er_reward, tps_reward, and cpu_usage_reward respectively represent the rewards of rt (response time), er (error rate), tps (transactions per second), and cpu_usage (CPU usage). WEIGHT_RT, WEIGHT_ER, WEIGHT_TPS, and WEIGHT_CPU_USAGE respectively represent the corresponding weights, which can be assigned weights according to the importance of KPI.

[0109] If system_Performance t >system_Performance t-1 ,but

[0110] system_Performance improves (improved), otherwise system_Performance degrades (degraded)

[0111] Combining SMO and RL for iterative optimization

[0112] A. Initialization phase

[0113] 1) Define the problem parameters:

[0114] N represents the number of virtual machines, and M represents the number of resource types (such as CPU, memory, etc.);

[0115] Resource_Limit i,m represents the upper limit of the mth resource of the ith virtual machine (e.g., 80% of the total resources);

[0116] Initial_Allocation: Initial resource allocation plan, an N×M matrix.

[0117] 2) Initialize SMO parameters: set population size, number of iterations, number of leader monkeys, etc.

[0118] 3) Initialize RL parameters:

[0119] Define state space, action space Action, reward function Reward, discount factor, learning rate, etc.;

[0120] Initialize the RL model.

[0121] B. Iterative Optimization Phase

[0122] For each iteration t:

[0123] 1.SMO global search:

[0124] 1) Generate candidate solutions: Generate a set of candidate resource allocation solutions based on the current population and SMO's search mechanism (leader monkey learning, etc.) t .

[0125] 2) Evaluate candidate solutions: For each candidate solution Candidates t,j ∈Candidates t , calculate its fitness value Fitness t,j ,The fitness function considers factors such as resource utilization and system ,performance.

[0126]

[0127] Among them, Usage i,m,j is the usage of the Mth resource of the i-th virtual machine of candidate solution j, and λ is the performance weight.

[0128] Specifically include: generating an initialization group;

[0129] Calculate the fitness of each candidate solution;

[0130] Calculate local optimal solutions and global optimal solutions;

[0131] Update local and global optimal solutions;

[0132] Determine whether the maximum number of iterations has been reached. If so, output the determined optimal solution; otherwise, continue iterating.

[0133] 2.RL Local Optimization:

[0134] 1) Select candidate solutions as the initial state of RL: From Candidates tSelect one or more candidate solutions as the starting point of RL optimization, denoted as State RL,0

[0135] 2) Execute RL strategy: For each selected initial state, execute RL strategy (Q-learning) to perform a series of action selection and resource allocation adjustments. In each step, according to the current state State RL,K , select Action RL,K , after execution, the new state State is obtained RL,K+1 and Reward RL,K .

[0136] 3) Update the RL model: Update the Q-values ​​of the RL model’s parameters based on the observed rewards and state transitions.

[0137] 3. Evaluation and selection: Evaluate the solution after RL optimization, that is, for each solution after RL optimization, calculate its final fitness value. Select the solution with the highest fitness value from all evaluated solutions (including the original candidate solution and the solution after RL optimization) as the starting point of the next iteration, recorded as Best_Solution t .

[0138] 4. Update population: According to the update mechanism of SMO, Best_Solution t Join the population and remove the solution with the lowest fitness.

[0139] 5. Check convergence conditions: If the maximum number of iterations is reached, or the improvement in fitness value is less than a certain threshold, stop the iteration.

[0140] C. Output the final converged resource allocation plan.

[0141] Output the final converged resource allocation solution Final_Allocation.

[0142] Figure 5 FIG. 2 shows a resource scheduling system framework diagram in one embodiment of the present disclosure. Figure 5 As shown, the system includes:

[0143] Data collection module 51: The centralized control of SDN provides global network information in the data collection module. The flow information and the acquired network information are sent to the data processing module.

[0144] Data Processing Module 52: The input of the entire Reinforcement Learning (RL) module is the result of the Data Processing Module. This module processes the information from the Data Collection Module, sends it to the Reinforcement Learning Module, and extracts the available features. The flow feature matrix and the network topology matrix are the results of this module. The network topology matrix contains the total bandwidth information and the available bandwidth information. The flow feature matrix contains the flow bandwidth requirements, QoS level, destination address, and flow source address.

[0145] State perception module 53: performs two functions. The environment state is mapped to the agent's internal perception, and the mapping result is updated to the current environment state change after the agent selects an action. The input of the data processing module is regarded as the initial state of the environment. Two matrices are used to determine whether the current flow meets the selected link bandwidth required for the flow at the end of the RL module training. If the generated path does not meet the requirements, the state perception module continues to plan the path: if the path meets the requirements, the results of the Q table processing module are updated.

[0146] Learning module 55: The internal perception from the state perception module and the reward value of the current environment state are used to update the strategy and select actions that are conducive to the accumulation of total rewards. The module that selects actions is called the action module.

[0147] Action module 56: The received action is executed by the action module and the environment state is updated.

[0148] Q-Table flow scheduling module 57: OpenFlow switch helps the module to generate the post-flow table and flow table, and uses the Q-table information. According to the policy, after updating the flow table in the forwarding layer, the flow is forwarded.

[0149] The optimal solution or a set of high-quality solutions output by the SMO module 54 is used as the initial state or input of the RL environment. The RL algorithm selects actions based on these inputs and the current state, and observes environmental feedback to update the strategy. Ultimately, by continuously iterating the optimization process of SMO and RL, continuous optimization of resource scheduling and multi-path routing is achieved.

[0150] Figure 6 The flowchart of the SMO algorithm in one embodiment of the present disclosure is shown. This embodiment introduces a specific implementation process of the SMO algorithm.

[0151] S602, initialization generation of an initial population: generating an initial population including a plurality of spider monkeys (candidate solutions), each candidate solution SM_ij representing a configuration scheme of resource scheduling and multi-path routing.

[0152] SM_ij=SM_min,j+U(0,1)*(SM_max,j-SM_min,j) (14)

[0153] Among them, U(0,1) is a uniformly distributed random number in the interval (0,1), which is used to adjust the direction and size of random_step. SM_min and SM_max represent the search range of routing selection in each dimension.

[0154] S604, local leader (LL) stage position update.

[0155] SM_new,ij=SM_ij+beta*(LL_kj-SM_min,j)+U(0,1)*random_step (15)

[0156] Among them, LL_kj is the j-th dimension position of the k-th group local leader, beta is the local leader position influence coefficient, and random_step is the random step size determined according to the size of the search space.

[0157] S606, global leader (GL) stage position update.

[0158] SM_new,ij=SM_ij+alpha(GL_j-SM_ij)+alpha*random_perturbation(16)

[0159] Among them, GL_j is the j-th dimension position of the global leader, alpha is the influence coefficient of the global leader position, and random_perturbation is the random perturbation term.

[0160] S608, local leader decision (LLD) phase.

[0161] If the local limit counter of the local leader LL reaches the threshold, all group members are reinitialized.

[0162] S610, global leader decision (GLD) phase.

[0163] If the position of the local leader LL is updated for a predefined number of iterations, then GL uses small subgroups. Each local leader of GL is elected by the LL procedure. If the position is not updated with respect to a predefined threshold, then the global leader merges all subgroups into one group.

[0164] 2. Reinforcement Learning (RL):

[0165] State: Real-time data of the network environment, including node status, link status, and business requirements.

[0166] Action: The action space contains possible resource scheduling and routing adjustment operations, which are the output of SMO.

[0167] Reward function: Design a reward function based on the service QoS satisfaction and resource utilization indicators to evaluate the quality of the actions performed in the current state.

[0168] Reward=QoS_Satisfaction+λ*Resource_Utilization (17)

[0169] Among them, QoS_Satisfaction is the quantitative value of QoS satisfaction, Resource_Utilization is the quantitative value of resource utilization, and λ is the weight coefficient

[0170] Strategy Update: Update the execution strategy based on feedback.

[0171]

[0172] Among them, State_t is the current system state at time t, Action_t is the action at time t, which is the routing solution SM_new,ij output by SMO, Reward_t is the reward obtained after executing the action, State_t+1 is the next state, α is the learning rate, γ is the discount factor, and Action_t+1 is the optional action in the next state.

[0173] Figure 7 FIG. 1 is a schematic diagram showing a resource scheduling device in one embodiment of the present disclosure. Figure 7 As shown, the resource scheduling device includes: an initial population generation module 71, which is used to generate an initial population including one or more candidate solutions, each candidate solution represents a resource allocation scheme for multi-path routing; an iterative population generation module 72, which is used to perform local search and global search based on the candidate solutions of the initial population to determine the candidate solutions of the iterative population; a reinforcement learning execution module 73, which is used to use the candidate solutions of the iterative population as the initial state of reinforcement learning optimization, and execute the reinforcement learning strategy to obtain the candidate solutions after reinforcement learning optimization; a resource allocation determination module 74, which is used to determine the final solution from the candidate solutions of the iterative population and the candidate solutions after reinforcement learning optimization based on the fitness of the candidate solutions, as the resource allocation scheme for multi-path routing.

[0174] The technical solution of this application can be used in the following application scenarios.

[0175] Cloud computing and big data centers: In cloud computing and big data centers, large model training is one of the common tasks. This method can be applied to these scenarios to help data centers achieve efficient resource scheduling and improve training efficiency and resource utilization.

[0176] Artificial intelligence research and development: In the field of artificial intelligence research and development, large model training is an indispensable part. This method can be applied to the training process of various AI models, providing powerful resource scheduling support for researchers and developers.

[0177] Autonomous driving and intelligent transportation: Autonomous driving and intelligent transportation systems need to process large amounts of data and complex computing tasks. This method can be applied to the development and optimization of these systems to help achieve efficient resource scheduling and computing optimization.

[0178] Healthcare and bioinformatics: In the healthcare and bioinformatics fields, large model training is also widely used in disease prediction, gene analysis, etc. This method can be applied to the large model training process in these fields to improve training efficiency and accuracy.

[0179] Smart manufacturing and industrial Internet of Things: In the field of smart manufacturing and industrial Internet of Things, large model training can be used to optimize production processes, improve equipment efficiency, etc. This method can be applied to resource scheduling and computing optimization in these scenarios to promote the development of smart manufacturing and industrial Internet of Things.

[0180] Some embodiments of the present disclosure have the following advantages:

[0181] 1) Powerful global search and local development capabilities: The SMO algorithm balances the algorithm's global search and local exploration capabilities through the setting of local leaders and global leaders. In the global leader stage, the algorithm uses the information of the global optimal solution to guide the search direction of the entire population; in the local leader stage, the quality of the solution is refined through local search of small groups. This mechanism enables the algorithm to both jump out of the local optimal solution and perform fine search near the global optimal solution, thereby improving the accuracy and efficiency of resource scheduling.

[0182] 2) Adaptive optimization capability: Combining the SMO algorithm with reinforcement learning can fully utilize the global search capability of SMO and the adaptive optimization capability of reinforcement learning to jointly solve the problem of scheduling large model training resources. This combination can more accurately control resource allocation and improve training efficiency and model performance.

[0183] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0184] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0185] Refer to the following Figure 8 An electronic device 800 according to this embodiment of the present invention will be described. Figure 8 The electronic device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0186] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0187] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 1 As shown in S102, an initial population including one or more candidate solutions is generated, each candidate solution represents a resource allocation scheme; S104, local search and global search are performed based on the candidate solutions of the initial population to determine the candidate solutions of the population after iteration; S106, the candidate solutions of the population after iteration are used as the initial state of reinforcement learning optimization, and the reinforcement learning strategy is executed to obtain the candidate solutions after reinforcement learning optimization; S108, based on the fitness of the candidate solutions, a final solution is determined from the candidate solutions of the population after iteration and the candidate solutions after reinforcement learning optimization as the resource allocation scheme.

[0188] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0189] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0190] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0191] The electronic device 800 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0192] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0193] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0194] A program product for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0195] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0196] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0197] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0198] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0199] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0200] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0201] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0202] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A resource scheduling method, characterized in that: include: generating an initial population including one or more candidate solutions, each candidate solution representing a resource allocation scheme; Performing local search and global search based on the candidate solutions of the initial population to determine the candidate solutions of the population after iteration; Taking the candidate solutions of the population after the iteration as the initial state of the reinforcement learning optimization, executing the reinforcement learning strategy to obtain the candidate solutions after the reinforcement learning optimization; Based on the fitness of the candidate solutions, a final solution is determined from the candidate solutions of the iterative population and the candidate solutions optimized by reinforcement learning as a resource allocation solution.

2. The method according to claim 1, characterized in that Also includes: adding the final solution to the initial population; Eliminate the candidate solution with the lowest fitness from the initial population; The updated initial population is used as the starting point for the next iteration to perform iterative processing.

3. The method according to claim 2, characterized in that Also includes: After each iteration, if it is determined that the convergence condition is met, the final solution obtained is output as the final resource allocation solution.

4. The method according to claim 3, characterized in that The convergence condition includes: reaching a maximum number of iterations, or an increase in the fitness value being less than a first threshold.

5. The method according to any one of claims 1 to 4, characterized in that: The method of performing local search and global search based on the candidate solutions of the initial population to determine the candidate solutions of the iterative population includes: Determining the fitness of each candidate solution in the initial population; Determine a local optimal solution and a global optimal solution based on the candidate solutions of the initial population; A local leader and a global leader are updated based on the local optimal solution and the global optimal solution.

6. The method according to claim 1, characterized in that The step of using the candidate solution of the iterated population as the initial state of the reinforcement learning optimization and executing the reinforcement learning strategy to obtain the candidate solution after the reinforcement learning optimization includes: Select one or more candidate solutions from the candidate solutions of the iterative population as the initial state of reinforcement learning optimization, denoted as State RL,0 ; For each selected initial state, the reinforcement learning strategy is executed to perform a series of action selection and resource allocation adjustments; in each step, according to the current state State RL,K , select Action RL,K , after execution, the new state State is obtained RL,K+1 and Reward RL,K ; Wherein, K is a natural number; Based on the reward and state transition, the parameter Q value of the RL model is updated.

7. The method according to claim 1, characterized in that The resource allocation scheme includes a resource allocation scheme for multi-path routing.

8. A resource scheduling device, characterized in that: include: An initial population generation module, used to generate an initial population including one or more candidate solutions, each candidate solution representing a resource allocation scheme for multi-path routing; An iterative population generation module, used for performing local search and global search based on the candidate solutions of the initial population to determine the candidate solutions of the population after iteration; A reinforcement learning execution module, used to use the candidate solution of the population after the iteration as the initial state of the reinforcement learning optimization, and execute the reinforcement learning strategy to obtain the candidate solution after the reinforcement learning optimization; The resource allocation determination module is used to determine a final solution from the candidate solutions of the iterative population and the candidate solutions optimized by reinforcement learning based on the fitness of the candidate solutions as a resource allocation solution for multipath routing.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the resource scheduling method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the resource scheduling method according to any one of claims 1 to 7 is implemented.

Images