Method, device and computer equipment for determining operation scheduling scheme of micro-service system
By using a duel-style dual-deep Q-network model and reinforcement learning techniques, the problem of insufficient global optimization capability of microservice system scheduling schemes in dynamic environments is solved, achieving efficient and real-time generation of scheduling strategies and improving the system's operating efficiency and adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA ELECTRONICS RELIABILITY AND ENVIRONMENTAL TESTING INSTITUTE ((THE FIFTH INSTITUTE OF ELECTRONICS MINISTRY OF INDUSTRY AND INFORMATION TECHNOLOGY) (CHINA SAIBAO LABORATORY)
- Filing Date
- 2026-01-13
- Publication Date
- 2026-06-05
AI Technical Summary
Existing microservice system scheduling optimization methods are ill-suited to adapting to dynamically changing operating environments when dealing with high-dimensional state spaces, multi-objective optimization needs, and real-time requirements, resulting in insufficient global optimization capabilities of scheduling schemes.
A dual-depth Q-network model is adopted. The model is generated by initializing the scheduling scheme, setting the reward function and scheduling action space, generating a sample data pool using the historical operation information of the microservice system, and obtaining the target operation scheduling scheme generation model through iterative training. The online Q-network model and the target Q-network model are combined for reinforcement learning to optimize the scheduling strategy.
It improves the efficiency of scheduling scheme generation and global optimization capabilities, enhances adaptability and real-time performance in complex dynamic environments, and meets the real-time requirements of microservice systems.
Smart Images
Figure CN121501462B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for determining the operation scheduling scheme of a microservice system. Background Technology
[0002] Microservices is a software architecture style that breaks down a monolithic application into small, independently deployed service units, each of which can be developed, deployed, and scaled independently.
[0003] However, the operating environment of microservice systems is complex, with significant dynamic changes and nonlinear characteristics. Existing scheduling optimization methods, such as rule-based scheduling or mathematical optimization methods, typically rely on a precise description of the system model. In practical applications, system models struggle to fully characterize the complex environmental characteristics, while high-dimensional state spaces, multi-objective optimization requirements, and real-time demands further increase the complexity of the problem. Traditional methods often exhibit poor accuracy in handling these issues, failing to adapt to dynamically changing operating environments, resulting in insufficient global optimization capabilities for scheduling schemes. Summary of the Invention
[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for determining the operation scheduling scheme of a microservice system in response to the above-mentioned technical problems.
[0005] Firstly, this application provides a method for determining the execution scheduling scheme of a microservice system, including:
[0006] Initialize the duel dual-depth Q-network model to obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system;
[0007] Obtain historical operation information of the microservice system, and generate a sample data pool based on the historical operation information and the reward function;
[0008] Based on the sample data pool, the reward function, and the scheduling action space, the initial running scheduling scheme generation model is iteratively trained to obtain the target running scheduling scheme generation model;
[0009] Based on the current running status of the microservice system, and based on the target running schedule generation model and the scheduling action space, the target running schedule of the microservice system is obtained.
[0010] In one embodiment, generating a sample data pool based on the historical running information and the reward function includes:
[0011] Based on the historical operation information and the reward function, the current running status, the scheduling action adopted, the running status after scheduling, and the scheduling reward value of the microservice system at each scheduling node are determined.
[0012] Each scheduling node is treated as a sample to obtain a sample data pool.
[0013] In one embodiment, the runtime scheduling scheme generation model includes an online Q-network model and a target Q-network model;
[0014] The step of iteratively training the initial runtime scheduling scheme generation model based on the sample data pool, the reward function, and the scheduling action space to obtain the target runtime scheduling scheme generation model includes:
[0015] The initial runtime scheduling scheme generation model is used as the runtime scheduling scheme generation model during training.
[0016] Obtain the training samples from the sample data pool;
[0017] For the training samples, the Q-values of the online Q-network model and the target Q-network model in the training are obtained respectively; the online Q-network model and the target Q-network model in the training are respectively the online Q-network model and the target Q-network model of the running scheduling scheme generation model in the training.
[0018] Based on the Q-value of the online Q-network model, the Q-value of the target Q-network model, and the scheduling reward value in the current training sample, a new training scheduling scheme generation model is obtained by optimization. Then, the step of obtaining the current training sample from the sample data pool is returned to execute until a target running scheduling scheme generation model that meets the preset training objective is obtained.
[0019] In one embodiment, after obtaining a sample data pool by treating each scheduling node as a sample, the method further includes:
[0020] Based on the historical operation information, the scheduling period corresponding to each sample in the sample data pool is determined;
[0021] Determine the scheduling evaluation information corresponding to each scheduling cycle;
[0022] Based on the scheduling evaluation information, determine the priority weight of each sample in the sample data pool;
[0023] Obtaining the training samples from the sample data pool includes:
[0024] The training samples are determined from the sample data pool based on the priority weight of each sample in the sample data pool.
[0025] In one embodiment, the step of obtaining the Q-value of the online Q-network model and the Q-value of the target Q-network model during training for the current training sample includes:
[0026] The current running state and scheduling action used in the current training sample are input into the online Q-network model during training to obtain the Q value of the online Q-network model.
[0027] Based on the post-scheduling running state in the current training samples, the scheduling action space, and the online Q-network model in the training, a predicted scheduling action is obtained for the post-scheduling running state.
[0028] The scheduled running state and the predicted scheduling action from the current training samples are input into the target Q-network model during training to obtain the Q-value of the target Q-network model.
[0029] In one embodiment, determining the reward function and scheduling action space for the execution scheduling of the microservice system includes:
[0030] Based on the preset scheduling objectives, determine the reward function for the execution scheduling of the microservice system;
[0031] Based on preset scheduling constraints, the scheduling action space for the operation scheduling of the microservice system is determined.
[0032] Secondly, this application also provides a device for determining the operation scheduling scheme of a microservice system, comprising:
[0033] The initial acquisition module is used to initialize the duel dual-depth Q network model, obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system.
[0034] The sample generation module is used to obtain the historical operation information of the microservice system and generate a sample data pool based on the historical operation information and the reward function.
[0035] The model training module is used to iteratively train the initial running scheduling scheme generation model based on the sample data pool, the reward function, and the scheduling action space to obtain the target running scheduling scheme generation model.
[0036] The scheduling determination module is used to obtain the target operation scheduling scheme of the microservice system based on the current running state of the microservice system, the target operation scheduling scheme generation model, and the scheduling action space.
[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0038] Initialize the duel dual-depth Q-network model to obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system;
[0039] Obtain historical operation information of the microservice system, and generate a sample data pool based on the historical operation information and the reward function;
[0040] Based on the sample data pool, the reward function, and the scheduling action space, the initial running scheduling scheme generation model is iteratively trained to obtain the target running scheduling scheme generation model;
[0041] Based on the current running status of the microservice system, and based on the target running schedule generation model and the scheduling action space, the target running schedule of the microservice system is obtained.
[0042] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0043] Initialize the duel dual-depth Q-network model to obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system;
[0044] Obtain historical operation information of the microservice system, and generate a sample data pool based on the historical operation information and the reward function;
[0045] Based on the sample data pool, the reward function, and the scheduling action space, the initial running scheduling scheme generation model is iteratively trained to obtain the target running scheduling scheme generation model;
[0046] Based on the current running status of the microservice system, and based on the target running schedule generation model and the scheduling action space, the target running schedule of the microservice system is obtained.
[0047] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0048] Initialize the duel dual-depth Q-network model to obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system;
[0049] Obtain historical operation information of the microservice system, and generate a sample data pool based on the historical operation information and the reward function;
[0050] Based on the sample data pool, the reward function, and the scheduling action space, the initial running scheduling scheme generation model is iteratively trained to obtain the target running scheduling scheme generation model;
[0051] Based on the current running status of the microservice system, and based on the target running schedule generation model and the scheduling action space, the target running schedule of the microservice system is obtained.
[0052] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for determining the operation scheduling scheme of a microservice system first initialize a dual-depth Q-network model to obtain an initial operation scheduling scheme generation model. The reward function and scheduling action space for the operation scheduling of the microservice system are then determined. By initializing the dual-depth Q-network model, a reinforcement learning framework capable of handling high-dimensional state spaces and complex nonlinear problems is established. Furthermore, by setting the reward function and scheduling action space, the model is provided with optimization directions and search ranges, enabling it to optimize for specific objectives of the microservice system and improving the adaptability and scalability of the scheduling scheme generation model. Next, historical operation information of the microservice system is acquired, and a sample data pool is generated based on this historical information and the reward function. By utilizing the historical operation information of the microservice system, a representative and diverse sample data pool is generated, fully utilizing existing data resources and reducing the cost and risk of online exploration. Simultaneously, by processing historical data using a reward function, the sample data pool better reflects the needs of the global optimization objective, significantly improving the efficiency of model training and sample utilization. Then, based on the sample data pool, reward function, and scheduling action space, the initial runtime scheduling scheme generation model is iteratively trained to obtain the target runtime scheduling scheme generation model. Through iterative training using historical data, the model can gradually learn the optimal scheduling strategy, making it more adaptable to complex dynamic environments and effectively balancing immediate rewards and long-term benefits, further enhancing the global optimization capability of the scheduling scheme and ensuring the effectiveness and reliability of the scheduling strategy. Finally, based on the current running state of the microservice system, and according to the target runtime scheduling scheme generation model and scheduling action space, the target runtime scheduling scheme of the microservice system is obtained. By utilizing the rapid reasoning capability of the trained model in the high-dimensional state space, a real-time response scheduling strategy is provided, meeting the real-time requirements of the microservice system. In the above method, by combining historical data with a duel-style dual-depth Q-network model, efficient runtime scheduling of the microservice system is achieved, not only improving the generation efficiency of the scheduling scheme but also significantly enhancing the global optimization capability and real-time adaptability of the scheduling scheme. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 This is a flowchart illustrating a method for determining the runtime scheduling scheme of a microservice system in one embodiment.
[0055] Figure 2 This is a flowchart illustrating the model training steps for generating a scheduling scheme in one embodiment.
[0056] Figure 3 A flowchart illustrating a method for determining the runtime scheduling scheme of a microservice system in a specific embodiment;
[0057] Figure 4 A structural block diagram of a device for determining the execution scheduling scheme of a microservice system in one embodiment;
[0058] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0060] In one embodiment, such as Figure 1 As shown, a method for determining the execution scheduling scheme of a microservice system is provided. This embodiment illustrates the method by applying it to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0061] Step S101: Initialize the duel dual-depth Q network model to obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system.
[0062] Among them, the reward function is a quantitative standard for the quality of the scheduling strategy; the scheduling action space is a set definition of all executable actions of the microservice system, ensuring that the scheduling output of the model meets the system's operational constraints.
[0063] For example, the terminal first sets a reward function based on the scheduling objective, transforming the global optimization objective into a trainable policy guide; defines the scheduling action space to ensure that the scheduling scheme conforms to the system's physical constraints and operating conditions; and finally generates an initial running scheduling scheme model by randomly initializing model parameters, providing an initial state for subsequent training and optimization.
[0064] Step S102: Obtain historical operation information of the microservice system, and generate a sample data pool based on the historical operation information and the reward function.
[0065] Historical operation information refers to time-series data collected during the actual operation of the microservice system; the sample data pool is a set of training samples that have been processed from historical operation information into a set of training samples that can be used by reinforcement learning models. Each sample records information such as the current state of the system, the actions performed, the immediate reward, and the next state.
[0066] For example, first, the terminal reads historical runtime data from the runtime database of the microservice system; then, it evaluates the states and actions in the historical data based on a reward function, such as calculating the immediate reward for each scheduled action; next, the terminal processes the data and constructs it into a sample format that conforms to a Markov decision process, including features such as state, action, immediate reward, and next state; finally, these samples are stored in a sample data pool to support subsequent training of reinforcement learning models.
[0067] Step S103: Based on the sample data pool, reward function, and scheduling action space, iteratively train the initial running scheduling scheme generation model to obtain the target running scheduling scheme generation model.
[0068] For example, firstly, the terminal randomly selects a batch of samples from the sample data pool; then, using the states and actions in the samples, it calculates the current Q-value through an online network and the target Q-value through a target network, combining the target value with a reward function to generate the target value; next, the terminal calculates the temporal difference error (TD error) between the current Q-value and the target Q-value, constructs a loss function based on this error, and optimizes the parameters of the online network; the terminal periodically updates the parameters of the target network during training to ensure the stability of the training process; through multiple iterations of training, the model parameters are gradually optimized, enabling the online network to generate high-quality scheduling schemes. Finally, the terminal obtains a target runtime scheduling scheme generation model that can quickly respond to dynamic changes in the system and achieve global optimization, providing real-time scheduling support for microservice systems.
[0069] Step S104: Based on the current running status of the microservice system and the model and scheduling action space generated according to the target running scheduling scheme, the target running scheduling scheme of the microservice system is obtained.
[0070] The current running status can be key status information collected by the microservice system during real-time operation.
[0071] For example, firstly, the terminal collects the current running status data of the microservice system in real time; then, the terminal uses this status data as input to the target running schedule scheme generation model for reasoning, and the model outputs a set of feasible scheduling actions based on the current status and the learned scheduling strategy; subsequently, the terminal verifies and corrects the output results based on the scheduling action space; finally, the terminal generates the target running schedule scheme of the microservice system and passes the scheme to the system control unit for execution.
[0072] In the aforementioned method for determining the operation scheduling scheme of a microservice system, firstly, a dual-depth Q-network model is initialized to obtain an initial operation scheduling scheme generation model. The reward function and scheduling action space for the microservice system's operation scheduling are then determined. By initializing the dual-depth Q-network model, a reinforcement learning framework capable of handling high-dimensional state spaces and complex nonlinear problems is established. Furthermore, by setting the reward function and scheduling action space, the model is provided with optimization directions and search ranges, enabling it to optimize for specific objectives of the microservice system and improving the adaptability and scalability of the scheduling scheme generation model. Next, historical operation information of the microservice system is acquired, and a sample data pool is generated based on this historical information and the reward function. By utilizing the historical operation information of the microservice system, a representative and diverse sample data pool is generated, fully utilizing existing data resources and reducing the cost and risk of online exploration. Simultaneously, the reward function is combined with historical data... Data processing enhances the sample data pool's ability to better reflect the global optimization objective, significantly improving model training efficiency and sample utilization. Then, based on the sample data pool, reward function, and scheduling action space, the initial runtime scheduling scheme generation model is iteratively trained to obtain the target runtime scheduling scheme generation model. Through iterative training using historical data, the model gradually learns the optimal scheduling strategy, making it more adaptable to complex dynamic environments and effectively balancing immediate rewards and long-term benefits. This further improves the global optimization capability of the scheduling scheme, ensuring the effectiveness and reliability of the scheduling strategy. Finally, based on the current running state of the microservice system, the target runtime scheduling scheme generation model, and the scheduling action space, the target runtime scheduling scheme for the microservice system is obtained. By leveraging the trained model's rapid inference capability in a high-dimensional state space, a real-time response scheduling strategy is provided, meeting the real-time requirements of the microservice system. In this method, combining historical data with a dual-depth Q-network model achieves efficient runtime scheduling of the microservice system, not only improving the generation efficiency of the scheduling scheme but also significantly enhancing its global optimization capability and real-time adaptability.
[0073] In an exemplary embodiment, step S102 above, which generates a sample data pool based on historical running information and a reward function, further includes: determining the current running state of the microservice system at each scheduling node, the scheduling action adopted, the running state after scheduling, and the scheduling reward value based on historical running information and a reward function; and treating each scheduling node as a sample to obtain the sample data pool.
[0074] For example, firstly, the terminal reads complete historical operation information from the microservice system's runtime database. This information covers the operation status and scheduling behavior within a specific time period. Then, the terminal analyzes the historical operation information node by node based on a reward function, which evaluates the effect of each scheduling action according to the scheduling objective. Next, the terminal extracts the current operation status, scheduling action, post-scheduling status, and reward value for each scheduling node, integrating this information into a sample data format usable by the reinforcement learning model. Finally, the terminal stores each processed scheduling node as an independent sample in the sample data pool, thereby constructing a comprehensive and structurally complete dataset to support subsequent reinforcement learning model training.
[0075] In this embodiment, historical operational data of the microservice system is comprehensively transformed into training samples usable by the reinforcement learning model, providing high-quality support for model training. Each sample contains complete information on the current running state, scheduling action, post-scheduling state, and reward value, comprehensively reflecting the dynamic characteristics of the system and the effectiveness of scheduling behavior. Based on this sample data pool, the model can significantly improve learning efficiency and generalization ability, effectively reducing the risks and resource consumption brought about by online exploration. In addition, a reward function is used to quantitatively evaluate the scheduling effect of historical operational data, ensuring that the sample generation process closely revolves around the global optimization objective, improving the applicability of the sample data pool and the actual effect of system optimization. Ultimately, this lays a data foundation for the optimization of microservice system operation scheduling in complex dynamic environments and provides strong support for the efficient training of reinforcement learning models.
[0076] In one exemplary embodiment, the runtime scheduling scheme generation model includes an online Q-network model and a target Q-network model; such as Figure 2 As shown, step S103 above iteratively trains the initial runtime scheduling scheme generation model based on the sample data pool, reward function, and scheduling action space to obtain the target runtime scheduling scheme generation model. This can also be achieved through the following steps:
[0077] Step S201: Use the initial runtime scheduling scheme generation model as the runtime scheduling scheme generation model during training;
[0078] Step S202: Obtain the training samples for this training from the sample data pool;
[0079] Step S203: For this training sample, obtain the Q value of the online Q network model in training and the Q value of the target Q network model in training.
[0080] Step S204: Based on the Q-value of the online Q-network model, the Q-value of the target Q-network model, and the scheduling reward value in the current training sample, optimize to obtain a new training scheduling scheme generation model, and return to execute the step of obtaining the current training sample from the sample data pool until a target running scheduling scheme generation model that meets the preset training objective is obtained.
[0081] Among them, the online Q-network model and the target Q-network model during training are the online Q-network model and the target Q-network model of the running scheduling scheme generation model during training, respectively.
[0082] The scheduling scheme generation model includes an online Q-network model and a target Q-network model. The online Q-network model is a value evaluation model for real-time prediction of scheduling actions, with its parameters updated in real-time during training. The target Q-network model is an evaluation model for calculating the target Q-value, with its parameters updated using a delayed update method to ensure the stability of the training process.
[0083] For example, firstly, the terminal sets the initialized runtime scheduling scheme generation model as the runtime scheduling scheme generation model in training, with the online Q-network model and the target Q-network model as its core components. Next, the terminal randomly selects training samples from the sample data pool. Each sample contains the current state of the microservice system, the scheduling action, the immediate reward value, and the post-scheduling state. Then, the terminal evaluates the current state and action in the sample using the online Q-network model, calculating the Q-value of the online Q-network model, representing the long-term estimated benefit of performing a specific action in the current state. Simultaneously, the terminal evaluates the post-scheduling state in the sample using the target Q-network model, calculating the target Q-value by selecting the value of the optimal action in the target Q-network, representing the long-term estimated benefit of performing a specific action in the post-scheduling state. The terminal combines the Q-value of the online Q-network model with the Q-value of the target Q-network model and the immediate reward value in the sample to calculate the TD error (temporal difference error), constructing a loss function to optimize the parameters of the online Q-network model. Simultaneously, the terminal periodically updates the parameters of the online Q-network model to the target Q-network model to ensure the stability and convergence of the training. Through multiple rounds of iterative training, the terminal continuously optimizes the parameters of the runtime scheduling scheme generation model, enabling it to more accurately predict the value of scheduling actions. In each training iteration, the terminal repeatedly retrieves training samples from the sample data pool, updates the parameters of the online Q-network model, and verifies whether the model meets the preset training objectives, such as the convergence of the Q-value / TD error value, the stable improvement of the target reward value, or reaching a set number of training iterations. When the training objectives are met, the terminal marks the runtime scheduling scheme generation model in training as the target runtime scheduling scheme generation model.
[0084] In this embodiment, efficient reinforcement learning training is achieved through the collaborative work of an online Q-network model and a target Q-network model, utilizing historical operational data. The online Q-network model adjusts its prediction capabilities in real time, while the target Q-network model stabilizes the training process through delayed updates, effectively avoiding the problem of Q-value overestimation. Simultaneously, by leveraging the diverse samples in the sample data pool and the optimization guidance of immediate reward values, the model gradually learns the global optimization strategy for microservice system scheduling. After achieving the training objective, the target runtime scheduling scheme generation model possesses the ability to predict the optimal scheduling action in real time, providing efficient and stable operational support for microservice systems in complex and dynamic environments. This significantly improves the convergence efficiency and optimization capability of the scheduling model, providing technical assurance for the overall performance improvement of the microservice system.
[0085] In an exemplary embodiment, after obtaining the sample data pool by treating each scheduling node as a sample, the method further includes: determining the scheduling period corresponding to each sample in the sample data pool based on historical operation information; determining the scheduling evaluation information corresponding to each scheduling period; and determining the priority weight of each sample in the sample data pool based on the scheduling evaluation information.
[0086] Furthermore, in an exemplary embodiment, step S202 of obtaining the current training sample from the sample data pool further includes: determining the current training sample from the sample data pool according to the priority weight of each sample in the sample data pool.
[0087] For example, firstly, the terminal analyzes the scheduling cycle corresponding to each sample in the sample data pool based on historical operation information. The scheduling cycle refers to the time range within which a microservice system transitions from one state to a new state after several scheduling operations, such as an hour, a day, a time range where the interval between consecutive scheduling actions does not exceed a preset interval, or other business-significant time periods. By dividing the scheduling cycle, the terminal enables samples to be mapped more accurately to different operation time periods, ensuring comprehensive coverage of the system's dynamic characteristics. Then, the terminal calculates scheduling evaluation information based on historical operation information within each scheduling cycle. This information reflects the global optimization contribution of scheduling actions and provides a basis for setting priorities. Next, based on the aforementioned scheduling evaluation information, the terminal determines the priority weight for each sample in the sample data pool using weight calculation rules. Through this process, the samples in the sample data pool not only contain state, action, immediate reward, and state transition information, but also include priority weights representing the importance of the sample.
[0088] Furthermore, the terminal's acquisition of training samples from the sample data pool includes the following process: Based on the priority weight of each sample in the sample data pool, the terminal selects training samples from the sample data pool using a priority-based sampling method. Specifically, the terminal first normalizes the sample priority weights, transforming them into a probability distribution for sample selection. Samples with higher weights have a greater probability of being selected, while samples with lower priority are given some selection opportunities to prevent the model from getting trapped in local optima during training. Simultaneously, the terminal selects a batch of training samples from the sample data pool through random sampling. These samples include the current state, scheduling actions, immediate rewards, and post-scheduling states, used to optimize the parameters of the reinforcement learning model.
[0089] In this embodiment, by introducing scheduling cycle division and scheduling evaluation information processing methods into the sample data pool, the operational characteristics and scheduling effects of the microservice system can be described more accurately. By calculating sample priority weights based on scheduling evaluation information and adopting a priority-weighted sampling strategy in sample selection, the utilization rate of key samples in model training can be significantly improved, enhancing the model's ability to learn the global optimization objective. Furthermore, by combining a random sampling mechanism, sample diversity can be maintained during model training, thereby avoiding the risk of training getting trapped in local optima. This not only accelerates the convergence speed of the reinforcement learning model but also improves the global performance and real-time adaptability of the microservice system scheduling optimization model, providing efficient and reliable technical support for system operation scheduling in complex dynamic environments.
[0090] In an exemplary embodiment, step S203, for the current training sample, obtains the Q-value of the online Q-network model and the Q-value of the target Q-network model during training, respectively. This further includes: inputting the current running state and the adopted scheduling action from the current training sample into the online Q-network model during training to obtain the Q-value of the online Q-network model; obtaining a predicted scheduling action for the post-scheduling running state based on the post-scheduling running state, scheduling action space, and the online Q-network model during training; and inputting the post-scheduling running state and the predicted scheduling action from the current training sample into the target Q-network model during training to obtain the Q-value of the target Q-network model.
[0091] First, the terminal inputs the current running state *s* and the scheduling action *a* from the current training sample into the online Q-network model during training. The online Q-network model takes the current state *s* and action *a* as input and calculates the Q-value Q(s, a) through forward propagation. This value is a comprehensive value estimate of executing scheduling action *a* in the current state *s*, including immediate rewards and expected values for subsequent states, and is used to measure the effectiveness of the current scheduling strategy.
[0092] Next, based on the post-scheduled running state s', the scheduling action space, and the online Q-network model in the training samples, the terminal generates a predicted scheduling action a' for the post-scheduled running state s'. Specifically, the terminal first obtains a set of all possible scheduling actions A(s') according to the constraints defined in the scheduling action space, which is defined based on the actual operational constraints of the microservice system. For each possible action a' in the post-scheduled running state s', the terminal inputs s' and a' together into the online Q-network model for forward propagation, calculating the corresponding Q-value Q(s', a'). The terminal compares the Q-values of all possible actions and selects the action a' with the largest Q-value as the predicted scheduling action, representing the optimal scheduling strategy under the post-scheduled running state s'.
[0093] Then, the terminal inputs the post-scheduled running state s' and the predicted scheduling action a' into the target Q-network model under training. The target Q-network model receives s' and a' as input and calculates its Q-value Q'(s', a') through forward propagation. Since the parameters of the target Q-network model are updated synchronously with a delay, it has higher stability, and its Q-value provides a more reliable optimization objective for training. The target Q-value Q'(s', a') represents the estimated future reward generated by executing the predicted scheduling action a' in the post-scheduled running state s'.
[0094] In this embodiment, by inputting the current running state and scheduling actions from the current training samples into the online Q-network model, the Q-value of the online Q-network model in the current state can be efficiently calculated, providing real-time value assessment support for the training of the reinforcement learning model. Utilizing the post-scheduling running state and scheduling action space, the value of all possible actions can be efficiently evaluated through the online Q-network model, and the optimal scheduling action can be selected, ensuring the rationality and effectiveness of subsequent scheduling strategies. By inputting the post-scheduling running state and predicted scheduling actions into the target Q-network model, the terminal reliably obtains the Q-value of the target Q-network model, providing a stable target reference value for the training process. Combining the real-time prediction capability of the online Q-network model and the stability optimization capability of the target Q-network model significantly improves the training efficiency and accuracy of the model. Simultaneously, the constraints of the scheduling action space ensure that the generated scheduling strategy is feasible and optimal, providing efficient and stable technical support for the operational optimization of microservice systems.
[0095] In an exemplary embodiment, step S101, which determines the reward function and scheduling action space for the operation scheduling of the microservice system, further includes: determining the reward function for the operation scheduling of the microservice system according to a preset scheduling target; and determining the scheduling action space for the operation scheduling of the microservice system according to preset scheduling constraints.
[0096] The preset scheduling objective refers to the specific performance requirements of the microservice system, which includes multiple optimization objectives. To balance these objectives, the reward function comprehensively considers them and evaluates the effectiveness of scheduling actions in a positive or negative manner, enabling the model to find a balance among optimizing multiple objectives.
[0097] For example, the terminal first determines the reward function for the scheduling based on a preset scheduling objective. This reward function is designed to quantitatively evaluate the effectiveness of the scheduling actions. In this way, the reward function transforms the multi-objective optimization problem into a single learnable objective for the reinforcement learning model, providing a clear optimization direction for model training. Subsequently, the terminal generates a scheduling action space based on preset scheduling constraints. The scheduling action space consists of all scheduling actions that meet the constraints, ultimately forming a finite set of actions that conforms to actual operational requirements.
[0098] In this embodiment, by determining the reward function based on a preset scheduling objective, the multi-dimensional optimization objective can be effectively transformed into a reinforcement learning training objective, thereby guiding the model to gradually learn a scheduling strategy that can achieve global optimization. By determining the scheduling action space based on preset scheduling constraints, it is ensured that all generated scheduling actions comply with the operational limitations of the microservice system, avoiding the generation of infeasible scheduling schemes. The combination of the reward function and the scheduling action space not only improves the efficiency of model training but also significantly enhances the usability of the scheduling strategy in actual operation, providing reliable technical support for the optimized scheduling of microservice systems.
[0099] In one exemplary embodiment, such as Figure 3 As shown, this application provides a method for determining the execution scheduling scheme of a microservice system, the method comprising:
[0100] Step S301: Initialize the duel dual-depth Q-network model to obtain the initial operation scheduling scheme generation model; determine the reward function for the operation scheduling of the microservice system according to the preset scheduling goal; determine the scheduling action space for the operation scheduling of the microservice system according to the preset scheduling constraints.
[0101] Step S302: Obtain historical operation information of the microservice system; based on the historical operation information and reward function, determine the current operation status, scheduling action, post-scheduling operation status, and scheduling reward value of the microservice system at each scheduling node; treat each scheduling node as a sample to obtain a sample data pool.
[0102] Step S303: Use the initial running schedule scheme generation model as the running schedule scheme generation model in training; obtain the training samples from the sample data pool; for the training samples, obtain the Q value of the online Q network model in training and the Q value of the target Q network model in training.
[0103] Step S304: Based on the Q-value of the online Q-network model, the Q-value of the target Q-network model, and the scheduling reward value in the current training sample, optimize to obtain a new training scheduling scheme generation model, and return to execute the step of obtaining the current training sample from the sample data pool until a target running scheduling scheme generation model that meets the preset training objective is obtained.
[0104] Step S305: Based on the current running status of the microservice system and the generation model and scheduling action space based on the target running scheduling scheme, the target running scheduling scheme of the microservice system is obtained.
[0105] In this embodiment, the microservice system is optimized for operation scheduling by using a duel-based dual-depth Q-network. This avoids the overestimation problem in the optimization process of traditional algorithms, resulting in more accurate results. At the same time, it improves operating efficiency, speeds up the operation response, optimizes the scheduling and operation strategy of the microservice system, reduces operating costs, and improves economic benefits.
[0106] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0107] Based on the same inventive concept, this application also provides a microservice system operation scheduling scheme determination apparatus for implementing the above-described method for determining the operation scheduling scheme of a microservice system. The solution provided by this apparatus is similar to the implementation scheme described in the above-described method. Therefore, the specific limitations in one or more embodiments of the microservice system operation scheduling scheme determination apparatus provided below can be found in the limitations of the microservice system operation scheduling scheme determination method described above, and will not be repeated here.
[0108] In one exemplary embodiment, such as Figure 4 As shown, a device for determining the execution scheduling scheme of a microservice system is provided, comprising: an initial acquisition module 401, a sample generation module 402, a model training module 403, and a scheduling determination module 404, wherein:
[0109] The initial acquisition module 401 is used to initialize the duel dual-depth Q network model, obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system.
[0110] The sample generation module 402 is used to obtain historical operation information of the microservice system and generate a sample data pool based on the historical operation information and the reward function.
[0111] The model training module 403 is used to iteratively train the initial running scheduling scheme generation model based on the sample data pool, reward function and scheduling action space to obtain the target running scheduling scheme generation model.
[0112] The scheduling determination module 404 is used to obtain the target running schedule of the microservice system based on the current running status of the microservice system and the model and scheduling action space generated according to the target running schedule scheme.
[0113] In one embodiment, the sample generation module 402 is further configured to determine the current running state, the scheduling action adopted, the running state after scheduling, and the scheduling reward value of the microservice system at each scheduling node based on historical running information and reward function; and to obtain a sample data pool by treating each scheduling node as a sample.
[0114] In one embodiment, the runtime scheduling scheme generation model includes an online Q-network model and a target Q-network model. The model training module 403 is further configured to use the initial runtime scheduling scheme generation model as the runtime scheduling scheme generation model in training; obtain the current training samples from the sample data pool; for the current training samples, obtain the Q-value of the online Q-network model of the online Q-network model in training and the Q-value of the target Q-network model in training; the online Q-network model in training and the target Q-network model in training are respectively the online Q-network model and the target Q-network model of the runtime scheduling scheme generation model in training; optimize the new runtime scheduling scheme generation model in training based on the Q-value of the online Q-network model, the Q-value of the target Q-network model, and the scheduling reward value in the current training samples, and return to execute the step of obtaining the current training samples from the sample data pool until a target runtime scheduling scheme generation model that meets the preset training objective is obtained.
[0115] In one embodiment, the above-mentioned microservice system operation scheduling scheme determination device is further configured to determine the scheduling period corresponding to each sample in the sample data pool based on historical operation information; determine the scheduling evaluation information corresponding to each scheduling period; and determine the priority weight of each sample in the sample data pool based on the scheduling evaluation information.
[0116] In one embodiment, the model training module 403 is further configured to determine the training sample from the sample data pool based on the priority weight of each sample in the sample data pool.
[0117] In one embodiment, the model training module 403 is further configured to input the current running state and the scheduling action adopted in the current training sample into the online Q-network model in training to obtain the Q value of the online Q-network model; based on the post-scheduling running state, scheduling action space and online Q-network model in training, to obtain the predicted scheduling action for the post-scheduling running state; and input the post-scheduling running state and the predicted scheduling action in the current training sample into the target Q-network model in training to obtain the Q value of the target Q-network model.
[0118] In one embodiment, the initial acquisition module 401 is further configured to determine the reward function for the operation scheduling of the microservice system according to the preset scheduling target; and to determine the scheduling action space for the operation scheduling of the microservice system according to the preset scheduling constraints.
[0119] The aforementioned microservice system's operation scheduling scheme determines that each module in the device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0120] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for determining a microservice system's runtime scheduling scheme. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0121] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0122] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0123] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0124] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for determining the execution scheduling scheme of a microservice system, characterized in that, The method includes: Initialize the duel dual-depth Q-network model to obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system; The system acquires historical operation information of the microservice system, and determines the current operation status, scheduling action, post-scheduling operation status, and scheduling reward value of the microservice system at each scheduling node based on the historical operation information and the reward function; it treats each scheduling node as a sample to obtain a sample data pool; it determines the scheduling period corresponding to each sample in the sample data pool based on the historical operation information; it determines the scheduling evaluation information corresponding to each scheduling period; and it determines the priority weight of each sample in the sample data pool based on the scheduling evaluation information. The initial runtime scheduling scheme generation model is used as the runtime scheduling scheme generation model in training. Based on the priority weight of each sample in the sample data pool, the current training sample is determined from the sample data pool. For the current training sample, the Q-value of the online Q-network model and the Q-value of the target Q-network model in training are obtained respectively. The online Q-network model and the target Q-network model in training are respectively the online Q-network model and the target Q-network model of the runtime scheduling scheme generation model in training. Based on the Q-value of the online Q-network model, the Q-value of the target Q-network model, and the scheduling reward value in the current training sample, a new runtime scheduling scheme generation model in training is obtained through optimization. The process of obtaining the current training sample from the sample data pool is then repeated until a target runtime scheduling scheme generation model that meets the preset training objective is obtained. The runtime scheduling scheme generation model includes an online Q-network model and a target Q-network model. Based on the current running status of the microservice system, and based on the target running schedule generation model and the scheduling action space, the target running schedule of the microservice system is obtained.
2. The method according to claim 1, characterized in that, For the training samples in this instance, the process of obtaining the Q-values of the online Q-network model during training and the target Q-network model during training includes: The current running state and scheduling action used in the current training sample are input into the online Q-network model during training to obtain the Q value of the online Q-network model. Based on the post-scheduling running state in the current training samples, the scheduling action space, and the online Q-network model in the training, a predicted scheduling action is obtained for the post-scheduling running state. The scheduled running state and the predicted scheduling action from the current training samples are input into the target Q-network model during training to obtain the Q-value of the target Q-network model.
3. The method according to any one of claims 1 to 2, characterized in that, The determination of the reward function and scheduling action space for the execution scheduling of the microservice system includes: Based on the preset scheduling objectives, determine the reward function for the execution scheduling of the microservice system; Based on preset scheduling constraints, the scheduling action space for the operation scheduling of the microservice system is determined.
4. A device for determining the operation scheduling scheme of a microservice system, characterized in that, The device includes: The initial acquisition module is used to initialize the duel dual-depth Q network model, obtain the initial runtime scheduling scheme generation model, and determine the reward function and scheduling action space for the runtime scheduling of the microservice system. The sample generation module is used to acquire historical operation information of the microservice system, and determine the current operation status, scheduling action, post-scheduling operation status, and scheduling reward value of the microservice system at each scheduling node based on the historical operation information and the reward function; treat each scheduling node as a sample to obtain a sample data pool; determine the scheduling period corresponding to each sample in the sample data pool based on the historical operation information; determine the scheduling evaluation information corresponding to each scheduling period; and determine the priority weight of each sample in the sample data pool based on the scheduling evaluation information. The model training module is used to generate a model from the initial runtime scheduling scheme as the runtime scheduling scheme generation model in training; determine the current training sample from the sample data pool according to the priority weight of each sample in the sample data pool; for the current training sample, obtain the Q value of the online Q-network model and the Q value of the target Q-network model in training; the online Q-network model and the target Q-network model in training are respectively the online Q-network model and the target Q-network model of the runtime scheduling scheme generation model in training; optimize a new runtime scheduling scheme generation model in training based on the Q value of the online Q-network model, the Q value of the target Q-network model, and the scheduling reward value in the current training sample, and return to execute the step of obtaining the current training sample from the sample data pool until a target runtime scheduling scheme generation model that meets the preset training objective is obtained; the runtime scheduling scheme generation model includes the online Q-network model and the target Q-network model; The scheduling determination module is used to obtain the target operation scheduling scheme of the microservice system based on the current running state of the microservice system, the target operation scheduling scheme generation model, and the scheduling action space.
5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.
7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.