PDU load distribution system based on reinforcement learning

Abstract

The application discloses a PDU load distribution system based on reinforcement learning, relates to the technical field of data center resource management, and realizes efficient scheduling of PDU load, server migration and start-stop through the combination of a multi-dimensional execution cost model and a digital twin environment, can balance energy consumption and performance in different load scenarios by using offline pre-training and hierarchical control, and continuously optimizes through online learning under concept drift detection; in addition, after superimposing security cost and fault tolerance overhead, key business and sensitive data can be preferentially protected, security risks and downtime losses can be effectively reduced, and finally, efficient, safe and scalable data center management in multidimensional coordination is achieved. Through batch migration and phased start-stop, resource impact and performance jitter caused by large-scale operations are further avoided, flexible responses can be made under multi-dimensional risks, and the robustness and sustainability of scheduling under heterogeneous loads can be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data center resource management, in particular to a PDU load distribution system based on reinforcement learning. BACKGROUND

[0002] With the rapid aggregation of energy-intensive businesses such as cloud computing, artificial intelligence inference and training, and big data analysis in super-large-scale data centers, the computing load presents the trends of peak transience, waveform non-stationarity, and business type diversification. The power consumption of the server cabinet often doubles with the surge of second-level traffic, and the power supply side must coordinate the PDU (Power Distribution Unit) phase balance, standby power switching, and bus redundancy within milliseconds. At the same time, the cold and hot channels, the liquid cooling pump speed, and the UPS energy storage scheduling also need to be linked in real time. The traditional heuristic scheduling with CPU utilization or task queue depth as a single dimension has been difficult to balance power peak suppression, energy utilization rate improvement, and stable demand adaptation to AI training long-tail load. In recent years, deep reinforcement learning (RL) has been used for resource scheduling, but most researches only use server power consumption or electricity as reward signals, ignoring the phase line capacity differences in PDU topology, the discharge depth of backup batteries, the safety isolation of key businesses, and the physical and operation limitations of multi-copy fault tolerance overhead, resulting in slow policy convergence speed and easy to break through the power risk threshold when landing in real scenarios, making it difficult to meet the high reliability requirements of "zero failure and zero data loss" in financial and government-level carrying environments.

[0003] After searching, in the Chinese invention patent with the application publication number CN109324875A, a data center server power consumption management and optimization method based on reinforcement learning is disclosed. The reinforcement learning method is used to solve the power consumption management and optimization problem of the data center. By continuously observing the load arrival, load distribution and power consumption information of the random system of the data center, sequential decisions are made. That is, according to the observed state at each time, an action is selected from the available action set to make a decision. The decision maker makes a new decision based on the new observed state, and repeats accordingly. The present application can optimize the load distribution strategy of the data center online without any prior knowledge, thereby reducing the overall operating power consumption of the data center.

[0004] The existing scheduling framework cannot quantify the compound cost of "instantaneous impact of virtual machine migration / server start-stop on the power supply link" and "critical business security redundancy" at the same time, resulting in a lack of effective autonomous decision-making mechanism for the RL agent when facing power supply side fluctuations, load mode drift or security level upgrade. When multiple GPU training tasks in the same cabinet are batch migrated to adjacent cabinets at night to save cooling energy consumption, and the algorithm does not consider the instantaneous margin and phase imbalance of the target PDU loop, it often causes phase overload or current impact; if the financial core business virtual machine is migrated to a node with only a single backup and in the maintenance window at this time, the fault tolerance level will drop sharply. Once the PDU overload triggers a circuit breaker or the host dual machine fails, the result will be AI task interruption, transaction system SLA violation and source station log data damage, which will then trigger a chain of compensation and brand reputation loss.

[0005] Therefore, there is an urgent need for an intelligent scheduling method that can embed power topology restrictions, security isolation levels and fault tolerance redundancy overheads in a multi-dimensional execution cost model, and can adaptively retrain when load concept drift occurs, to achieve the coordinated optimization of data center energy consumption, performance and security and reliability. SUMMARY

[0006] (1) Technical problems solved

[0007] To overcome the shortcomings of the prior art, the present application provides a PDU load allocation system based on reinforcement learning, which realizes efficient scheduling of PDU load, server migration and start-stop through the combination of a multi-dimensional execution cost model and a digital twin environment, balances energy consumption and performance in different load scenarios using offline pre-training and hierarchical control, and continuously optimizes through online learning under concept drift detection; in addition, after superimposing security costs and fault tolerance overheads, critical business and sensitive data can be prioritized to effectively reduce security risks and downtime losses, ultimately achieving efficient, secure and scalable data center management in multiple dimensions. At the same time, through batch migration and phased start-stop, further resource impact and performance jitter caused by large-scale operations are avoided, and flexible responses can be made under multi-dimensional risks; the technical problems described in the background art are solved.

[0008] (2) Technical solutions

[0009] To achieve the above purpose, the present application is implemented by the following technical solutions: a PDU load allocation system based on reinforcement learning, comprising: when a data center load fluctuation is detected, calling a real-time monitoring component to obtain original monitoring data, quantifying the energy consumption and delay of virtual machine migration and server start-stop in depth through a multi-dimensional execution cost model, and outputting a dynamic cost sequence;

[0010] After receiving the dynamic cost sequence and the original monitoring data, the PDU power distribution topology, the server cluster and the typical load scenario are trained in the digital twin environment for multiple rounds of reinforcement learning offline training, and the pre-trained strategy is selected for hierarchical scheduling;

[0011] After successfully loading the pre-trained strategy, the upper layer reinforcement learning controller generates a set of macro migration and start-stop instructions, and the lower layer flexible scheduling module performs batch execution based on multi-dimensional execution cost and safety limit, and records the specific scheduling feedback to the system log;

[0012] If the concept drift measure exceeds the set threshold or the strategy yield drops significantly, small-scale online learning is performed based on the digital twin environment and the latest scheduling feedback, and a resource allocation scheme is deployed in advance using the load prediction model;

[0013] In the original multi-dimensional execution cost model, safety cost and fault tolerance overhead are injected, and reinforcement learning strategies are prioritized in offline pre-training through safety-related penalties or rewards. In the hierarchical scheduling and online adaptive phase, redundant, encrypted or batch migration is prioritized for key servers or sensitive businesses.

[0014] Preferably, the multi-dimensional execution cost model sets independent dimensions for virtual machine migration, server start-stop and PDU switching, and quantifies the time delay, energy consumption and business performance impact of each dimension with weights to dynamically measure the comprehensive cost of each scheduling action in the current state and write it to the monitoring database.

[0015] Preferably, the deployed monitoring components collect server utilization, PDU load, power-on duration and network bandwidth in fixed time windows, and map the collected results to the cost model in real time to dynamically update cost data; based on the bandwidth safety factor and the temperature safety factor, generate the available margin threshold as the basis for subsequent batch scheduling judgment.

[0016] Preferably, the constructed digital twin environment completely maps the PDU power distribution topology, server hardware configuration, network bandwidth limit and typical business load curve of the real data center, and uses performance incentive coefficients and cost penalty coefficients to guide reinforcement learning agents to generate pre-trained strategies in offline simulation.

[0017] Preferably, multiple load scenarios are simulated and trained in the digital twin environment for multiple rounds of reinforcement learning, the cost sensitivity is adjusted during offline training to amplify the negative reward of high-cost actions, and the strategy stability coefficient is used to regularize the difference between new and old strategies, outputting pre-trained strategies that balance energy efficiency and performance, and performance evaluation results.

[0018] Preferably, the upper-layer reinforcement learning controller first filters out instructions with peak costs according to real-time costs and priority labels after generating the instruction set based on the pre-trained strategy, and then executes the instructions in batches according to the scheduling amplification coefficient and flexibility control parameters by the lower-layer module; wherein, before generating the instruction set, instructions with costs exceeding a threshold are filtered out based on the multi-dimensional execution cost model, and execution priorities are assigned to the remaining instructions.

[0019] Preferably, the lower-layer scheduling module executes instructions in the instruction set in batches according to bandwidth margins and server temperature margins, and returns scheduling feedback containing actual execution time, power consumption changes and performance jitter to the upper-layer controller after each batch is completed, together with an updated cost vector for use in the online adaptive phase.

[0020] Preferably, the difference between the actual system state and the expected state is monitored to calculate the environmental drift degree.

[0021] When the drift degree exceeds a preset threshold, a drift alert is generated, and the trigger timestamp and main deviation indicators are recorded, triggering subsequent online training or transfer learning actions.

[0022] Preferably, the online training process selects a number of recent scheduling cycles to form a micro training data set, uses a learning rate to incrementally update the old strategy to obtain a new strategy, and redeploys after completing a quick regression test in the digital twin environment.

[0023] The micro training data set includes the current state, the action generated by the upper-layer reinforcement learning controller, the multi-dimensional execution cost, and the actual revenue.

[0024] Preferably, historical load sequences and recent monitoring data are called to predict future load for a number of periods using a long short-term memory network, and when the average absolute percentage error of the prediction deviation exceeds a set threshold, the time series prediction model is fed back to iteratively update the parameters, and the available computing and power resources are preconfigured in advance accordingly.

[0025] Preferably, a safety cost parameter and a fault tolerance overhead parameter are added to the original multi-dimensional execution cost model to form a comprehensive cost model, wherein the safety weight coefficient and the fault tolerance weight coefficient control the priority of key business migration, redundant copy synchronization and sensitive node power management.

[0026] Preferably, safety positive rewards and fault tolerance negative penalties are added to the reward function during offline simulation to guide the scheduling strategy to prioritize the safety isolation of sensitive businesses and the high-availability redundancy of key nodes. If the host of the sensitive business has not completed encryption isolation or backup of the copy, the shutdown operation is delayed, and the log is recorded in the execution feedback.

[0027] Preferably, after introducing additional offset related to safety alarm, critical node redundancy state, during online self-adaptation, dynamically adjust safety and fault tolerance weight coefficient according to safety alarm frequency and fault trigger number, to maintain preset balance target between energy consumption, performance and reliability, and feedback actual execution result to digital twin environment again.

[0028] (III) Beneficial Effects

[0029] The application provides a PDU load distribution system based on reinforcement learning, which has the following beneficial effects:

[0030] The present scheme is connected through steps one to five, combined with multi-dimensional cost function C(a i ,s t ) and deep interaction with digital twin environment Omega, realizes the overall management of PDU load distribution, server start and stop and safety fault tolerance, and achieves the following beneficial effects:

[0031] Step one quantifies the time delay, energy consumption and performance jitter of virtual machine migration, server start and stop into multi-dimensional cost function C(a i ,s t ), and introduces real-time monitoring, so that the cost of each action is accurately described; step two embeds the multi-dimensional execution cost into the offline training process, simulates various load scenarios using digital twin environment Omega, and iterates the pre-training strategy multiple times to ensure that the reinforcement learning agent can master the spontaneous avoidance ability of high-cost actions before real deployment;

[0032] Step three adopts a hierarchical control architecture: the upper layer reinforcement learning controller Π top generates macro load instructions based on the pre-training strategy, and the lower layer module performs flexible scheduling and batch execution to avoid the impact of one-time large-scale migration or frequent start and stop on the system. This hierarchical mechanism not only balances energy consumption and performance, but also continuously collects scheduling feedback during execution to provide high-precision data for online self-adaptation in step four.

[0033] Step four, when facing non-stationary load, captures environmental changes through concept drift measurement, and quickly adjusts the strategy in the digital twin environment Omega combined with small-scale online training, which is agile and stable, allowing the system to remain efficient in long-term operation. Step five adds safety cost function and fault tolerance overhead function to the original multi-dimensional cost function C(a i ,s t ) to obtain the comprehensive cost function C'(a i ,s t), not only can impose security / fault-tolerant rewards or restrictions in offline training (step two) and hierarchical scheduling (step three), but also can dynamically increase the weight according to the demand in online adaptation (step four), so as to achieve fine control of key server redundancy and sensitive business isolation.

[0034] By unifying the state s t , the action a i and the cost function C'(a i , s t ) are coupled with each other: the offline simulation in the early stage lays the policy foundation for the hierarchical scheduling in the later stage, the hierarchical execution feedback also feeds back to the online adaptation, and the addition of the security and fault-tolerant factors gives the system as a whole more resilience and expands the policy adaptation range; not only can the continuous promotion of energy consumption optimization be maintained, but also can the multi-dimensional risks (performance jitter, environmental drift, security threats) be flexibly responded to, so as to achieve comprehensive control effect. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a schematic diagram of the PDU load distribution system based on reinforcement learning of the application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0037] Please refer to Figure 1 , the application provides a PDU load distribution system based on reinforcement learning, which comprises:

[0038] Step one, when the load fluctuation of the data center is monitored, the real-time monitoring component is called to deeply collect the indicators such as server utilization, PDU load, power supply start-up time and network bandwidth, and the non-linear calculation of the action cost is performed according to the energy consumption, delay and performance jitter weight in the multi-dimensional cost function C(a i , s t ), the comprehensive cost data form is dynamically updated, the real-time evaluation and quantitative description of virtual machine migration or server start-stop are realized, and the generated time series cost information is provided to the subsequent offline pre-training and online adaptation as a key input;

[0039] The step one comprises the following contents:

[0040] Step 101, establishing a multi-dimensional execution cost model

[0041] Step 101 defines a unified multi-dimensional execution cost function for various execution actions in the data center (including virtual machine migration, server start-up and shutdown, and PDU switching) based on historical operation and maintenance data and predictive analysis results.

[0042] Let a i This represents the i-th type of action that can be performed in a data center environment, such as migrating a virtual machine to a target host or enabling a hibernating server; let s t This provides a comprehensive description of the current system status, including server utilization, PDU remaining power margin, and network bandwidth load.

[0043] Let E(a i ,s t ) indicates that in state s t Next, execute action a i The expected additional energy consumption; let M(a) i ,s t ) represents the potential performance migration overhead or service jitter caused by the action; let L(a) i ,s t The cost factor (a) represents the operational delay caused by the action (the time from the issuance of the instruction to its effective date). Based on the three key cost factors mentioned above, a multidimensional cost function C(a) can be constructed under dimensionless conditions. i ,s t The details are as follows:

[0044]

[0045] Among them: ζ1, ζ2, and ζ3 are positive weighting coefficients, all of which are between 0 and 1, and the sum of the three is 1. They are used to balance the relative importance of energy consumption, performance jitter, and time delay.

[0046] φ1, φ2, and φ3 are action sensitivity coefficients, with values ​​greater than 0. They are used to match the dimensions according to the average cost of different action categories, and to control the amplification or attenuation of each cost factor in exponential or logarithmic transformations.

[0047] Multidimensional cost C(a) i ,s t The higher the returned value, the greater the overall execution cost of the action in the current state; by breaking down the three key dimensions of energy consumption, performance, and latency, it is easier to identify high-risk or high-cost actions and accurately characterize the cost of the action; the parameter system's universality and portability: through the weighted coefficient ζ i and sensitivity coefficient φ i Its adjustable properties enable it to adapt to data centers of different sizes and business needs, achieving multi-scenario reuse of a single function structure.

[0048] Step 102: Perform real-time monitoring and dynamic updates.

[0049] After establishing the multi-dimensional cost function C(a i ,s t ), step 102 further deploys a real-time monitoring component inside the data center to dynamically collect and update the key parameters defined in step 101;

[0050] The monitoring probe collects hardware layer data such as server CPU utilization, network I / O usage, and power-on duration;

[0051] The load analysis engine tracks the real-time performance state and service delay of each virtual machine and converts it into statistical quantities required for migration overhead M(a i ,s t ) and operation delay L(a i ,s t ), integrates the power monitoring module output, and calculates and summarizes the estimated value of additional energy consumption (a i ,s t ) generated during action execution.

[0052] The incremental energy consumption, performance jitter, and latency information obtained by monitoring are continuously input into the multi-dimensional cost function C(a i ,s t ) of step 101 to obtain the latest execution cost value; the latest multi-dimensional cost function C(a i ,s t ) and the current system state s t are saved together to form a cost time series dataset that evolves over time, which is used for subsequent offline pre-training and online adaptive learning calls; through real-time updating, the execution cost of each action under different load conditions can be dynamically described, avoiding decision bias caused by history or static configuration alone. When the scale or hardware type of the data center expands, new cost factors can be included in the multi-dimensional cost function C(a i ,s t ) system by adding or upgrading monitoring components, which has good flexibility and maintainability.

[0053] Step two, when the execution cost data and real-time monitoring records are obtained, load the PDU power distribution topology, server group, and typical load mode in the digital twin environment Ω, use reinforcement learning agents to perform offline training on multiple rounds of simulation interaction, balance energy consumption, performance, and multi-dimensional cost function C(a i ,s t ) overhead through a nonlinear objective function, and select a pre-trained strategy to complete strategy performance evaluation and solidification, providing an accurate and feasible macro decision template for subsequent hierarchical scheduling;

[0054] The step two includes the following contents:

[0055] Step 201, build a digital twin environment and load the execution cost model

[0056] According to the system state s output by step one t , action a i and its corresponding multi-dimensional cost function C(a i , s t ), a digital twin environment Ω highly consistent with the real data center is built in the virtual simulation platform;

[0057] Define the complete PDU power distribution topology, compute node (server cluster), network link attributes in the digital twin environment Ω, and set the corresponding load injection module to simulate the working load under various typical business scenarios (such as daytime peak load, night low load, AI training high fluctuation load, etc.);

[0058] Combine the real-time monitoring data of step one with the historical load pattern to generate the initial state set of the digital twin environment Ω, where each state s t contains server utilization, PDU power supply margin, network occupancy rate, etc.;

[0059] Use the action set containing different actions a i to strictly reproduce the impact of executing actions in the digital twin environment Ω, and call the multi-dimensional cost function C(a i , s t ) to evaluate the comprehensive cost of energy consumption, delay and performance jitter generated by each action in the simulation environment; Build an offline data interaction interface so that the reinforcement learning agent can obtain the current state s t at any time when interacting with the digital twin environment Ω and calculate the corresponding comprehensive cost C(a i , s i based on action a t ;

[0060] Reproduce the PDU topology, server cluster and load characteristics through the digital twin environment Ω to make the training and verification phase as close to the production environment as possible, reducing the risk of trial and error in the real system. Use the multi-dimensional cost function C(a i , s t ) directly in the digital twin environment Ω to evaluate the real cost of each action, ensuring that offline simulation can accurately reflect the operation cost in subsequent deployment, providing a reliable basis for policy training. High-cost actions can be accurately identified in the digital twin environment, so that frequent migration or low-yield switch behavior can be effectively suppressed during simulation training.

[0061] Step 202, offline pre-training and policy verification

[0062] The immediate reward function R(s t ,a i ) of the reinforcement learning agent is defined by comprehensively considering the positive incentive of system performance improvement and the negative penalty of execution cost, as follows:

[0063]

[0064] wherein, α is a performance incentive coefficient, greater than 0, used for dimension matching and balancing convergence speed, and can amplify the reward of the overall system performance index Λ(s t ) of the current state s t ); Λ(s t ) can be obtained by comprehensively measuring the data center throughput, task completion rate, etc., and the greater the value, the better the performance, and the specific value can be obtained after linear normalization and weighting; ∈ is a cost penalty coefficient, greater than 0, used to suppress high-cost actions; δ is a cost sensitivity, greater than 0, used for dimension matching and amplifying the influence of C(a i ,s t ) in logarithmic or exponential operations.

[0065] In the digital twin environment Ω, starting from the state s t , the agent tries different actions a i , and iteratively updates the strategy according to the above immediate reward R(s t ,a i ); by simulating real load fluctuations through the monitoring data collected in step one, the agent can learn the optimal or suboptimal load distribution method in various situations such as high load and low load, network surplus and network congestion; in the later stage of training convergence, by increasing the cost sensitivity δ, the influence of execution cost can be further amplified, guiding the agent to reduce high-cost actions with limited performance rewards.

[0066] After training is completed, the cumulative reward, average execution cost, and load balancing degree of the agent in each scenario are counted to obtain a number of candidate strategies; the strategy with the highest comprehensive score in energy efficiency and SLA performance is selected as the pre-trained strategy, and the corresponding performance evaluation result Θ is generated, including the average migration frequency, overall energy consumption, and response time under the strategy, etc., and the obtained pre-trained strategy and performance evaluation result Θ are output together,

[0067] wherein, the SLA performance refers to the key quality indicators agreed in the service level agreement (SLA) for measuring whether the system or service meets the contract requirements; including availability, response delay, and throughput, etc.

[0068] Large-scale migration or testing in real data center environment will bring high risk and cost, step 202 completes a large number of simulation iterations in the digital twin environment Ω, which can quickly explore the better strategy and avoid affecting the production business. By embedding the nonlinear amplification method in the reinforcement learning objective function R(s t ,a i ), the agent can consciously avoid operations with limited contribution to system revenue but high execution cost, improving stability and energy efficiency during subsequent deployment, and efficiently suppressing low-yield high-cost actions; once the simulation training converges, a pre-trained strategy can be generated, and the output strategy can be directly applied in the subsequent hierarchical control architecture, which can greatly shorten the online debugging cycle and obtain good scheduling efficiency in the initial stage.

[0069] Step three, when deploying the pre-trained strategy to the actual data center, the upper layer reinforcement learning controller generates migration or start-stop instruction set {a top ′} according to the macro load allocation scheme Π i filtered in step two, and hands it over to the lower layer scheduling module to dynamically execute in batches according to the real-time monitoring information in step one and multi-dimensional cost function C(a i ,s t ), and generates a traceable execution feedback for online adaptive subsequent calls by segmenting control and safety limit test to smoothly land virtual machine migration and server start-stop.

[0070] The step three includes the following contents:

[0071] Step 301, upper layer reinforcement learning controller deployment and global load allocation

[0072] Load the pre-trained strategy output in step two to the upper layer reinforcement learning controller Π top , which refers to the decision-making module deployed at the top of the data center scheduling architecture, running the reinforcement learning strategy; wherein the upper layer reinforcement learning controller Π top (s t ) can generate macro load allocation decisions when given the current system state s t , that is, give high-level instruction set {a i} to each PDU load, server group and virtual machine migration direction;

[0073] Place the performance evaluation result Θ generated in step two in the reference database of the upper layer reinforcement learning controller Π top , so as to compare the actual effect and detect the deviation during strategy execution. The multi-dimensional cost function C(a i ,s t) to input the latest real-time monitoring information (server utilization, PDU load, power-on duration, etc.) to the upper-layer reinforcement learning controller Π top to ensure that the controller is aware of the current system's possible high-cost operations;

[0074] When the upper-layer reinforcement learning controller Π top calculates a set of action instructions {a i}, it performs re-filtering based on the multi-dimensional cost function C(a i , s t ) to automatically reduce the priority of actions with high costs or unclear benefits, in order to avoid large-scale or frequent operations at the source;

[0075] The set of action instructions {a top ′} finally retained in the upper-layer reinforcement learning controller Π i is defined as the global load distribution instructions, such as migrating several virtual machines to designated host computers and enabling or disabling specific server groups; the corresponding set of action instructions {a i ′} along with its corresponding priority label is output to the lower-layer scheduling module of step 302 for batch or segmented implementation of specific operations.

[0076] During execution, if the lower-layer scheduling module (step 302) returns execution delay or partial operation failure information, the upper-layer reinforcement learning controller Π top will perform rapid simulation evaluation in combination with the digital twin environment Ω constructed in step two to determine whether temporary adjustment of the strategy is needed; this intermediate feedback not only allows the controller to fine-tune the current strategy, but also provides more realistic operation data for subsequent step four: online adaptive learning and continuous iterative optimization.

[0077] In use, the pre-trained strategy is directly used for scheduling in the real environment, significantly shortening the online optimization time and enabling the initial strategy to have a good energy consumption and performance compromise, which can ensure the global optimal tendency of macro decision-making; by real-time referencing the multi-dimensional cost function C(a i , s t ), control instructions with excessive overhead can be preferentially excluded or delayed during action generation, reducing the execution burden of the lower layer. Once the lower-layer execution result does not match the expectation, the strategy can be quickly tested and corrected in the digital twin environment Ω, forming a rapid test-adjustment closed loop. In traditional systems, the problem of excessive migration or frequent start-stop is often discovered only in the execution phase, and the proposed solution performs pre-selection based on the multi-dimensional cost function C(a i , s t ) in the upper layer, which can significantly reduce execution shocks.

[0078] Step 302, lower-layer flexible scheduling and multi-stage execution

[0079] The lower layer scheduling module obtains the action instruction set {a i ′} output by the upper layer and reads its corresponding priority label;

[0080] According to the dynamic monitoring data (such as network bandwidth BW t , server temperature Temp t , etc.) in step one, it is determined whether the current system has the ability to perform simultaneous operations of multiple instructions {a i ′}. If it cannot be executed simultaneously, it is scheduled in order of priority or dependency. For example, the system first estimates the resource demand of each migration or start-stop instruction a i ′ on network bandwidth and host cooling, and reads the current available bandwidth BW avail = current total bandwidth BW cap - used bandwidth BW t and the temperature margin ΔT avail (s) that the server can tolerate = safe temperature upper limit Temp max - current temperature Temp t (s). Then the system adds up the bandwidth requirements of all instructions and compares them with BW avail × μ bw (bandwidth safety factor); at the same time, for each target server, the temperature change ΔT i caused by the instruction is compared with ΔT avail (s) × μ tmp (temperature safety factor). Only when both network and temperature conditions are met for all instructions, the system determines that it can be executed in parallel; otherwise, the instructions will be split into smaller batches to ensure that each operation is executed within the safe margin of bandwidth and temperature control. In this way, performance jitter or service interruption caused by bandwidth overload or overheating of the machine room can be avoided.

[0081] For a large number of virtual machines that need to be migrated or multiple servers that need to be enabled, a batch and segment execution strategy is adopted:

[0082]

[0083] Where: k represents the number of operations in the current batch, such as the number of virtual machines that need to be migrated;

[0084] ω and γ are the scheduling amplification factor and flexibility control parameter respectively, both of which are positive real numbers, used to determine the size of each batch execution; when γ > 1, the size of the subsequent batch execution gradually increases with the batch number, and vice versa.

[0085] Through the above batch control function, a small amount of migration or start-stop test can be performed first to verify whether its impact on server load and network I / O is within the simulation range of step two, and then the execution effect is gradually increased batch by batch.

[0086] After the completion of each batch or each segment of execution, the current system state s t is monitored to determine whether it reaches or exceeds the pre-set safety limit (such as the maximum available margin of PDU load, the critical value of network occupancy), and if it approaches or exceeds, the subsequent operation is suspended and the upper layer reinforcement learning controller Π top is informed.

[0087] The execution process data (including actual migration duration, server power-on time, success / failure log) is synchronized to the digital twin environment Ω in step two and the next online adaptive learning for reference.

[0088] If the upper layer instruction contains an operation of shutting down multiple idle or low-load servers, the lower layer module first checks whether the current task migration is completed to ensure that the power-off does not occur before the migration is completed; when it is confirmed that the power-off can be performed, the servers are sequentially shut down in small batches to avoid excessive current surge. The lower layer module refers to the execution unit deployed in the data center control architecture, which is responsible for executing the upper layer scheduling instructions.

[0089] Large-scale migration or start-stop actions are divided into multiple segments, and the system impact caused by each segment of execution is observed, which can effectively prevent resource bottlenecks and performance jitter; by continuously monitoring the key indicators in the state s t and comparing them with the safety threshold, the operation size can be flexibly controlled during the execution process, which can ensure the protection of SLA and device health.

[0090] The upper layer reinforcement learning controller Π top uses the pre-trained strategy output in step two to integrate the multi-dimensional execution cost data C(a i ,s t ) in step one when generating global load allocation instructions, reducing high-cost actions from the source; the lower layer flexible scheduling assigns the execution order according to the real-time monitored safety limit and multi-segment batch strategy, ensuring smooth transition of the system during large-scale migration, start-stop or PDU switching; the instructions generated by the upper layer controller are gradually implemented at the lower layer, and the execution process data at the lower layer is fed back to the upper layer and the digital twin environment Ω, further providing real scene feedback for online adaptive training in step four.

[0091] Therefore, the combination of pre-trained strategy and batch / segmented scheduling mechanism significantly improves the safety and efficiency in real data center environment, avoiding the common defects of relying solely on macro-optimization while ignoring the impact of the execution layer, and achieving global optimization ability and execution smoothness.

[0092] Step four, when the concept drift measure CD(Δ(s t ) exceeds the threshold or the policy performance drops significantly, perform fast small-scale online training on the digital twin environment Ω with the actual scheduling feedback collected in step three, incrementally fine-tune the reinforcement learning policy, and combine the prediction model to predict the medium and short-term load, pre-arrange virtual machine migration or standby power supply activation scheme, and put the updated policy into the real environment to maintain global optimization of multi-dimensional cost and performance indicators under non-stationary load;

[0093] The step four includes the following contents:

[0094] Step 401, environment offset monitoring and concept drift detection

[0095] After the completion of the lower-layer flexible scheduling in step three, compare the actual system performance of each execution batch With the expected state verified in the digital twin environment Ω in step two Collect the error of each key indicator (such as server utilization, PDU power supply margin, network occupancy) after load allocation;

[0096] Let Δl(s τ ) represent the difference vector between the actual system state and the ideal or expected state at time τ, to capture the cumulative deviation and deviation change speed in the recent period, define the sliding time window length as L; when evaluating the drift at the current time t, consider the difference evolution in the interval [t-L, t] to give the following measure function:

[0097]

[0098] In the formula: Δ(s τ ) represents the difference between the actual state and the expected state of the system at time τ, which is a vector, used to measure the deviation of multi-dimensional load indicators (such as CPU utilization, PDU load, network occupancy, etc.) at that time;

[0099] W represents a configurable positive definite symmetric weighting matrix, used to allocate difference weights between each indicator in the vector, for example, CPU utilization deviation can be given a higher weight, while temperature or network indicators are given a secondary weight;

[0100] [Δ(s τ )] T W[Δ(s τ )] can be regarded as a quadratic form under this weighting matrix, used to measure the comprehensive impact of multi-dimensional deviation.

[0101] represents the first order derivative of the deviation vector with respect to time τ, which is used to reflect the speed of change of the deviation over time; κ is a drift speed sensitive coefficient, which is greater than 0, and is used to control the degree of attention to the rate of change of CD(Δ τ ) over time;

[0102] β is a drift overall sensitivity coefficient, which is greater than 0, and L is a sliding time window length;

[0103] If the obtained offset degree CD(Δ t ) exceeds the offset threshold τ cd , it can be considered that a significant drift occurs, and when the drift monitoring module detects that CD(Δ t )>τ cd , the subsequent online training or transfer learning action is triggered.

[0104] If the detected offset degree CD(Δ t ) is greater than the offset threshold τ cd , a drift alarm is generated, and details such as a trigger timestamp, a main offset indicator, etc. are recorded; the drift alarm is also sent to step 402 to start the targeted policy fine-tuning process, and step 403 is notified to adjust the policy in advance in combination with the load prediction mechanism.

[0105] In use, by continuously monitoring the difference between the actual and expected states, the environment changes can be quickly captured, avoiding the use of outdated policies for a long time under non-stationary loads, and the policy mismatch can be identified in real time; only when the offset degree CD(Δ t ) value exceeds the offset threshold τ cd , the subsequent learning is performed, which can reduce unnecessary training overhead and maintain system stability.

[0106] Step 402, small-scale online training and transfer learning

[0107] After receiving the drift alarm, data is extracted from the real running logs in the recent several execution cycles and the execution feedback collected in step three to form a micro training data set The micro training data set includes the current state , the action a top ′ generated by the upper reinforcement learning controller Π i , the multi-dimensional execution cost , and the actual revenue (SLA performance, energy consumption) and other information.

[0108] The micro training data set is imported in the digital twin environment Ω , the pre-trained strategy output in step two is used as the initial point, and small-scale online training or transfer learning is performed:

[0109]

[0110] where: Π old is the original policy function, e.g. the upper reinforcement learning controller Π top is the current used policy; η is the learning rate, which can be adjusted according to the drift amplitude and system stability requirement;

[0111] is the loss function of reinforcement learning or deep learning, which is used to minimize the estimation error and cost penalty on the micro training data set ; a composite loss function based on policy gradient is adopted to train the upper reinforcement learning controller:

[0112]

[0113] where the first term maximizes the performance incentive aA(s) and cost penalty e(e δC(a,s) -1) by calculating the negative log probability weighted reward on the state-action-return triplets (s, a, r) in the data set D;

[0114] The second term introduces the Kullback-Leibler divergence regularization between the old policy Π old and the new policy Π, with the weight coefficient λ used to smooth the policy update to avoid unstable decision-making due to one-time large-scale adjustment, thus ensuring that the policy can continuously improve the overall system revenue while maintaining continuity and stability with the verified policy during the offline pre-training and online fine-tuning phases; in this process, the execution cost function C(a i ,s t ) in step one is referenced to evaluate the action cost, ensuring that the policy after transfer learning does not sacrifice sensitivity to high-cost actions;

[0115] After training, the policy Π new is quickly verified in the digital twin environment Ω (the number of simulations is less than the complete training in step two), and if the revenue is significantly improved and the system volatility is controllable, the trained policy Π new is replaced in the upper controller; if the new policy is found to be unstable or limited by insufficient data sets during the verification process, the trained policy Π old is maintained and only part of the actions are incrementally updated.

[0116] After the concept drift alarm, small-scale online training is performed to timely correct the part of the policy that does not match the new environment, avoiding high cost and long latency caused by full-scale retraining; the digital twin environment Ω is used to quickly verify the policy Π new , which can eliminate high-risk solutions before real system application, ensuring business continuity in data centers; real execution logs The latest environment is reflected, and efficient simulation fine-tuning is performed in the digital twin environment Ω, so that online training can quickly complete convergence and not disturb the production system, and micro training data + digital twin double linkage is realized.

[0117] Step 403, prospective strategy layout combined with prediction model

[0118] While the concept drift monitoring is in parallel with the small-scale online training, the historical load sequence and the recent monitoring data are called, and a prospective algorithm of a time series prediction model (such as a long short-term memory network LSTM) is applied to obtain a load trend estimation of the next T time periods The multi-dimensional cost function C(a i ,s t ) in step one is combined to evaluate the possible virtual machine migration or server start-stop actions in advance, and sufficient available servers or load offline windows are prepared in priority;

[0119] The predicted load trend estimation is input into the strategy Π updated (or retained) in step 402, so that scheduling decisions in the next period of time are simulated in the digital twin environment Ω; if it is found that the upcoming high load interval may trigger a large number of migrations or device starts, preheating or resource preparation is performed in batches in the real system in advance to avoid concentrated operations near the high load, and prospective strategy layout is realized; the prospective strategy layout is submitted to the upper-layer reinforcement learning controller Π top together with the execution log, and whether it further alleviates the system fluctuation is observed in the concept drift detection in the next stage;

[0120] If the prediction deviation is large, that is, the error index between the load trend estimation and the observed load of the actual arrival period exceeds the preset threshold, the difference is fed back to the time series prediction model, and the prediction parameters are iteratively updated to improve the subsequent accuracy.

[0121] By grasping the load variation trend in advance, the system can complete resource preparation or redundant device offline operation during the low load period, and can actively avoid emergency scheduling during the load peak period, greatly reducing the risk of sudden migration and start-stop during the peak period. When dealing with non-stationary load, not only passive online learning can be done, but also more prospective strategy layout can be done under the support of prediction data.

[0122] Through the constructed online adaptive mechanism, quick response to real-time environmental changes and forward-looking prevention ability to future fluctuations are ensured, and four-dimensional deep integration of monitoring-learning-prediction-execution is realized. Through such deep cooperation, not only is the concept of drift detection, online training, load prediction and other links simply stacked, but also each link is mutually reinforced, so that the optimal state of PDU load distribution and energy consumption optimization is continuously maintained in the long-term operation, providing effective technical support for sustainable operation of large-scale data centers in non-stationary load environment.

[0123] Step five, when the data center needs to strengthen security protection or improve fault tolerance level, security cost i ,s t and fault tolerance overhead are added in the multi-dimensional cost function C(a ,s to form a comprehensive cost function C'(a i ,s t ), and the reinforcement learning strategy is preferentially screened through security-related penalties or rewards in offline pre-training, and the key servers or sensitive services are preferentially migrated in the hierarchical scheduling and online adaptive phase. Finally, under the premise of meeting the energy consumption and performance requirements, high-dimensional cooperation of data security and redundant fault tolerance is realized.

[0124] The step five includes the following contents:

[0125] Step 501, security and fault tolerance overhead are included in the multi-dimensional execution cost model

[0126] Inherit the multi-dimensional cost function C(a i ,s t ) in step one, further define the security cost function and the fault tolerance overhead function respectively measure the influence degree of the execution action a i on the system security policy and fault tolerance policy in the state s t ;

[0127] The additional load brought by encryption, authentication and isolation operations can be represented; It can be used to describe the additional resources consumed by fault tolerance redundancy deployment (such as multiple copy synchronization, key node dual machine hot backup);

[0128] Security and fault tolerance costs are included in the original cost model, and the comprehensive cost function C'(a i ,s t ) is defined:

[0129]

[0130] Where: C(ai ,s t ) is the original energy consumption, delay and performance jitter multi-dimensional cost (defined in step one);

[0131] ρ1,ρ2 are security and fault tolerance weight coefficients, both of which are between 0 and 1, and determine the priority amplification degree of security and fault tolerance; when or is larger, the fault tolerance weight coefficient will inhibit the selection probability of high-risk or low-fault-tolerance actions in subsequent decision-making;

[0132] According to the monitoring component in step one, the collection of security alarm, vulnerability detection, key node availability and other information is added, and the values of fault tolerance weight coefficient and fault tolerance overhead function are updated in real time, so that the comprehensive cost function C'(a i ,s t ) reflects the latest security and fault tolerance state at each moment; with the original monitoring mechanism and the newly added security alarm interface, the dynamic tracking of security vulnerabilities, key node redundancy and other states is realized, and the security and fault tolerance requirements can be perceived in real time.

[0133] Step 502, introduce security / fault tolerance restrictions or rewards in offline pre-training and hierarchical scheduling

[0134] In the digital twin environment Ω, the newly defined comprehensive cost function C'(a i ,s t ) is evaluated;

[0135] The negative benefits or positive incentives related to security and fault tolerance are included in the objective function of reinforcement learning, such as giving additional positive rewards for necessary security operations (such as high-priority encryption) and imposing exponential penalties on unsafe or non-fault-tolerant actions, so that the agent learns to balance security and fault tolerance in the offline training phase;

[0136] When generating macro load allocation scheme, the upper layer reinforcement learning controller Π top gives priority to the security state of key servers and whether the redundant nodes are complete: if an action is easy to cause security weakness or fault tolerance deficiency, its execution cost will be automatically adjusted higher based on the comprehensive cost function C'(a i ,s t ), and the scheduling priority will be reduced;

[0137] The lower layer flexible scheduling module also refers to the safety fault-tolerant indicators when batch migration or start-stop devices: if the host of sensitive business has not completed the encryption isolation or backup copy, delay the shutdown operation, and record the log to the execution feedback, so as to fine-tune the subsequent online adaptation. Thus, without rebuilding independent safety policy training, more cautious batch or batch scheme can be carried out for safety and fault-tolerant key operations when layered scheduling, avoiding hidden dangers to system safety, and ensuring the execution of high-priority security requirements.

[0138] Step 503, online adaptive reinforcement of security and fault tolerance

[0139] When performing concept drift detection in step four, additional offsets related to security alerts and critical node redundancy status are introduced. For example, when the overall system load is still normal but high-density network attacks or multi-node hardware failures occur, the security dimension weight in the offset CD(Δ(s t ) is quickly triggered to rise, so that the subsequent online training focuses on security countermeasures.

[0140] If the security alarm is frequent or the redundancy node is insufficient, resulting in a large increase in the fault tolerance overhead function , small-scale online training or transfer learning is enabled in step four;

[0141] In the form of a micro training data set Recent real execution logs are collected and included in safety fault-tolerant indicators, the punishment coefficient for unsafe actions is increased or the incentive value for redundant operations is increased, to ensure that the agent quickly converges to a new strategy that balances safety and performance.

[0142] Combined with the time series prediction model in step four, if it is predicted that there will be a potential attack peak or critical load extreme fluctuation in the future period, more redundant servers are pre-configured or more stringent authentication and isolation measures are started, and a temporary security mute window is set if necessary to complete backup; the actual execution result is fed back to the digital twin environment Ω, reserving interfaces for subsequent higher-level safety or fault-tolerant expansion.

[0143] When the data center environment mutates mainly due to security or fault tolerance factors, the concept drift detection mechanism can quickly capture the risk, trigger online updates, keep the strategy and the environment synchronized evolution, and dynamically adapt to the security / fault tolerance requirements; through the pre-prevention of attack or failure peaks, key data isolation or multi-copy deployment can be completed before the main load is overheated, which can improve the overall resilience of the system. Step 501 couples the security and fault tolerance dimensions into the original multi-dimensional execution cost model, so that all execution actions can be quantified in the same framework. Step 502 inserts a security / fault tolerance related reward or penalty mechanism in the offline pre-training and hierarchical scheduling link (corresponding to steps two and three), so that the agent can consider security compliance and fault tolerance stability while optimizing energy consumption and performance; Step 503 integrates security and fault tolerance index weight in concept drift detection and small-scale online training for online adaptive scenarios (corresponding to step four), which can respond to sudden scenes of high-risk security situation or insufficient redundancy, and cooperate with load prediction for forward-looking deployment.

[0144] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0146] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment of the present application according to actual needs.

[0148] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A PDU load distribution system based on reinforcement learning, characterized in that: include, When data center load fluctuations are detected, the real-time monitoring component is invoked to obtain raw monitoring data. The energy consumption and latency of virtual machine migration and server startup and shutdown are deeply quantified through a multi-dimensional execution cost model, and a dynamic cost sequence is output. After receiving the dynamic cost sequence and raw monitoring data, multiple rounds of offline reinforcement learning training are conducted on the PDU power distribution topology, server cluster and typical load scenarios in the digital twin environment, and pre-trained strategies are selected for hierarchical scheduling. After successfully loading the pre-training strategy, the upper-layer reinforcement learning controller generates a macro-level transfer and start / stop instruction set, and the lower-layer flexible scheduling module executes the instructions in batches based on multi-dimensional execution costs and safety limits, and records the specific scheduling feedback to the system log. If the concept drift metric exceeds the set threshold or the strategy returns drop significantly, a small-scale online learning process is executed based on the digital twin environment and the latest scheduling feedback, and a resource allocation plan is deployed in advance using a load prediction model. Security costs and fault tolerance overheads are injected into the original multidimensional execution cost model. In offline pre-training, security-related penalties or rewards are used to prioritize the selection of reinforcement learning strategies. In the hierarchical scheduling and online adaptive phases, critical servers or sensitive businesses are prioritized for redundancy, encryption, or batch migration.

2. The PDU load distribution system based on reinforcement learning according to claim 1, characterized in that: The multidimensional execution cost model sets independent dimensions for virtual machine migration, server startup and shutdown, and PDU switching, and quantifies the time latency, energy consumption, and business performance impact of each dimension to dynamically measure the comprehensive cost of each scheduling action in the current state and write it into the monitoring database.

3. The PDU load distribution system based on reinforcement learning according to claim 2, characterized in that: The deployed monitoring components collect server utilization, PDU load, power-on time, and network bandwidth in fixed time windows, and map the collected results to the cost model in real time to dynamically update the cost data. A usable margin threshold is generated based on the bandwidth safety factor and the temperature safety factor, which serves as the basis for subsequent batch scheduling.

4. The PDU load distribution system based on reinforcement learning according to claim 3, characterized in that: The constructed digital twin environment fully maps the PDU power distribution topology, server hardware configuration, network bandwidth limitations, and typical business load curves of a real data center. In offline simulation, performance incentive coefficients and cost penalty coefficients are used to guide the generation of pre-training strategies for reinforcement learning agents.

5. The PDU load distribution system based on reinforcement learning according to claim 4, characterized in that: In the digital twin environment, multiple rounds of reinforcement learning simulation training are performed on various load scenarios. During offline training, the cost sensitivity is adjusted to amplify the negative rewards of high-cost actions. The difference between the old and new strategies is constrained by the policy stability coefficient. The pre-trained strategy and performance evaluation results that take into account both energy efficiency and performance are output.

6. The PDU load distribution system based on reinforcement learning according to claim 5, characterized in that: After the upper-level reinforcement learning controller calls the pre-training strategy to generate the instruction set, it first filters out the cost peak instructions based on the real-time cost and priority label, and then hands them over to the lower-level module to execute in batches according to the scheduling amplification factor and flexibility control parameters. Before generating the instruction set, instructions with costs exceeding a threshold are filtered out based on the multidimensional execution cost model, and the remaining instructions are assigned execution priorities.

7. The PDU load distribution system based on reinforcement learning according to claim 6, characterized in that: The lower-level scheduling module executes the instructions in the instruction set in batches according to the bandwidth margin and server temperature margin. After each batch is completed, the scheduling feedback, which includes the actual execution time, power consumption changes and performance jitter, along with the updated cost vector, is returned to the upper-level controller for use in the online adaptive phase.

8. The PDU load distribution system based on reinforcement learning according to claim 7, characterized in that: Monitor the difference between the actual system state and the expected state to calculate the environmental drift. When the drift exceeds a preset threshold, a drift alarm is generated, and the trigger timestamp and main offset indicators are recorded to trigger subsequent online training or transfer learning actions.

9. The PDU load distribution system based on reinforcement learning according to claim 8, characterized in that: The online training process selects several recent scheduling cycles to form a mini training dataset, uses the learning rate to incrementally update the old strategy to obtain the new strategy, and redeploys it after completing a rapid regression test in the digital twin environment. The mini training dataset includes the current state, the actions generated by the upper-level reinforcement learning controller, multidimensional execution costs, and actual benefits.

10. The PDU load distribution system based on reinforcement learning according to claim 9, characterized in that: By calling historical load sequences and recent monitoring data, and combining them with a long short-term memory network, the load for several future periods is predicted. When the average absolute percentage error of the prediction deviation exceeds a set threshold, the data is fed back to the time series prediction model to iteratively update the parameters, and available computing and power resources are pre-allocated accordingly.

11. The PDU load distribution system based on reinforcement learning according to claim 10, characterized in that: A comprehensive cost model is formed by adding security cost parameters and fault tolerance overhead parameters to the original multidimensional execution cost model. The priority of critical business migration, redundant copy synchronization and sensitive node power management is controlled by security weight coefficient and fault tolerance weight coefficient.

12. The PDU load distribution system based on reinforcement learning according to claim 11, characterized in that: During offline simulation, a positive security reward and a negative fault tolerance penalty are added to the reward function to guide the scheduling strategy to prioritize the security isolation of sensitive services and the high availability and redundancy of critical nodes. If the host hosting the sensitive business has not yet completed encryption isolation or backup, the shutdown operation will be delayed, and the logs will be recorded in the execution feedback.

13. The PDU load distribution system based on reinforcement learning according to claim 12, characterized in that: After introducing additional offsets related to security alarms and critical node redundancy status, during online adaptation, the safety and fault tolerance weight coefficients are dynamically adjusted according to the frequency of security alarms and the number of fault triggers to maintain a preset balance between energy consumption, performance and reliability, and the actual execution results are fed back to the digital twin environment.

Images