Virtual machine thermal migration management method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention disclose a virtual machine live migration management method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining historical operation data of each virtual machine in a cloud platform environment; constructing a learning task framework composed of a state space, an action space and a reward function; training a state value function, a guide strategy and an execution strategy in the learning task framework based on the historical operation data; the state value function is used for evaluating a state long-term value, the guide strategy is used for planning a target state, and the execution strategy is used for generating a specific migration action; integrating the trained state value function, the guidance strategy and the execution strategy to obtain a migration strategy generation model; and generating a live migration strategy of the virtual machine based on the migration strategy generation model. A migration strategy generation model is obtained through training, and a complex migration strategy can be learned from historical operation data, so that a live migration strategy is dynamically generated, and intelligence and flexibility of migration decision making are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of virtual machine hot migration technology, and in particular to a virtual machine hot migration management method, a virtual machine hot migration management device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Cloud computing enables users to use IT infrastructure flexibly and scalably by allocating resources on demand. Virtual machine live migration, a key technology in cloud computing, allows virtual machines to be migrated to other physical nodes in their entirety without interrupting business operations, thereby improving resource utilization and ensuring high system availability.

[0003] Traditional hot migration methods mainly rely on predefined rules and thresholds, which have problems such as poor flexibility, low resource utilization and high operation and maintenance costs. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a virtual machine hot migration management method, a virtual machine hot migration management device, an electronic device, and a computer-readable storage medium that overcome or at least partially solve the above problems.

[0005] To address the aforementioned problems, a first aspect of this invention provides a virtual machine hot migration management method, the method comprising: Obtain historical runtime data for each virtual machine in the cloud platform environment; A learning task framework consisting of a state space, an action space, and a reward function is constructed. The state space is used to represent the running state of each virtual machine, the action space includes all feasible transition operations, and the reward function is used to evaluate the overall effect of the transition operations. Based on the historical operational data, a state value function, a guidance strategy, and an execution strategy are trained within the learning task framework; the state value function is used to evaluate the long-term value of a state, the guidance strategy is used to plan the target state, and the execution strategy is used to generate specific transition actions. The trained state-value function, guidance policy, and execution policy are integrated to obtain the transfer policy generation model; Based on the migration strategy generation model, a hot migration strategy for virtual machines is generated.

[0006] According to a second aspect of the present invention, a virtual machine hot migration management apparatus is provided, the apparatus comprising: The runtime data acquisition module is used to acquire historical runtime data of each virtual machine in the cloud platform environment; The task framework construction module is used to construct a learning task framework consisting of a state space, an action space, and a reward function. The state space is used to represent the running state of each virtual machine, the action space includes all feasible transition operations, and the reward function is used to evaluate the overall effect of the transition operations. The strategy model training module is used to train a state value function, a guidance strategy, and an execution strategy within the learning task framework based on the historical running data; the state value function is used to evaluate the long-term value of a state, the guidance strategy is used to plan the target state, and the execution strategy is used to generate specific transition actions. The strategy model integration module is used to integrate the trained state-value function, guidance policy and execution policy to obtain the transfer policy generation model. The migration strategy generation module is used to generate a hot migration strategy for virtual machines based on the migration strategy generation model.

[0007] According to a third aspect of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the virtual machine hot migration management method as described in any of the preceding embodiments.

[0008] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein when executed by a processor, the computer program implements the steps of the virtual machine hot migration management method as described in any of the preceding embodiments.

[0009] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This invention discloses a method, apparatus, electronic device, and storage medium for virtual machine hot migration management. The method includes: acquiring historical operating data of each virtual machine in a cloud platform environment; constructing a learning task framework consisting of a state space, an action space, and a reward function; training a state value function, a guidance strategy, and an execution strategy within the learning task framework based on the historical operating data; the state value function is used to evaluate the long-term value of a state, the guidance strategy is used to plan a target state, and the execution strategy is used to generate specific migration actions; integrating the trained state value function, guidance strategy, and execution strategy to obtain a migration strategy generation model; and generating a hot migration strategy for the virtual machine based on the migration strategy generation model. By constructing a learning task framework defined by a state space, action space, and reward function, and using historical data to train the state value function, guidance strategy, and execution strategy, the migration strategy generation model can autonomously learn complex optimization rules based on historical operating data. It can make more accurate and adaptive migration decisions based on the real-time dynamic changes in the cloud platform environment, thus achieving automation and intelligence in the migration decision-making process. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the steps of a virtual machine hot migration management method provided in an embodiment of the present invention; Figure 2 This is a flowchart of another virtual machine hot migration management method provided in an embodiment of the present invention; Figure 3 This is an architectural block diagram of a virtual machine hot migration management method provided in an embodiment of the present invention; Figure 4 This is a flowchart of a virtual machine hot migration management method provided in an embodiment of the present invention; Figure 5 This is a structural block diagram of a virtual machine hot migration management device provided in an embodiment of the present invention. Detailed Implementation

[0011] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] Traditional hot migration methods mainly rely on predefined rules and thresholds, which have problems such as poor flexibility, low resource utilization and high operation and maintenance costs.

[0013] One of the core concepts of this invention is that by constructing a learning task framework defined by a state space, an action space, and a reward function, and by using historical data to train the state value function, the guidance strategy, and the execution strategy, the migration strategy generation model can autonomously learn complex optimization rules based on historical running data. It can make more accurate and adaptive migration decisions based on the real-time dynamic changes in the cloud platform environment, thus realizing the automation and intelligence of the migration decision-making process.

[0014] Reference Figure 1 The diagram illustrates a flowchart of a virtual machine hot migration management method provided by an embodiment of the present invention. The method specifically includes the following steps: Step 101: Obtain historical running data for each virtual machine in the cloud platform environment; This invention relates to a policy-guided offline reinforcement learning method applied to virtual machine hot migration management. It utilizes reinforcement learning algorithms to autonomously learn the optimal migration strategy and dynamically adjusts migration decisions based on the real-time environment, thereby improving the migration success rate.

[0015] This invention is applicable to virtual machine hot migration scenarios in data centers. It is applied in data center computing clusters, where the implementation of virtual machine hot migration management requires at least one data center server, and a cloud platform to provide various parameters such as network resources for the virtual machines deployed on each server in the cluster.

[0016] A virtual machine (VM) is a logical computer system simulated on a physical server using virtualization technology, possessing a complete set of hardware system functions. The entire runtime state of a VM is encapsulated in one or more files. This encapsulation is a prerequisite for enabling hot migration, because a VM can be transferred from one physical host to another as a complete, portable unit.

[0017] Historical running data for each virtual machine is collected mainly from host resource metrics, virtual machine status data, network topology data, storage performance data, historical migration records, and special scenario data.

[0018] Host resource metrics include CPU utilization, memory usage, disk I / O, network bandwidth, NUMA topology, and cache hit rate. Virtual machine status data includes vCPU configuration, memory ballooning, dirty page rate, and workload characteristics. Network topology data includes inter-host latency, bandwidth margin, VXLAN / VLAN configuration, and RDMA support. Storage performance data includes shared storage latency (e.g., iSCSI / NFS), IOPS, and storage protocols. Historical migration records include migration success rate, time taken, interruption time, and failure reasons (e.g., memory overflow, network timeout). Special scenario data includes vGPU status, VRAM usage, encryption protocols involved in migration, and compliance tags for GPU / FPGA accelerated scenarios.

[0019] In this embodiment of the invention, acquiring historical operational data of each virtual machine in the cloud platform environment is the data foundation for achieving intelligent migration decisions. A multi-dimensional, large-scale dataset is collected and integrated, encompassing host resource metrics (such as CPU and memory utilization), virtual machine status (such as dirty page rate), network performance (such as latency and bandwidth), and historical migration records (such as success rate and time consumption). This massive amount of historical data accurately records the past operational status and maintenance results of the cloud platform, providing indispensable training samples for subsequently building offline reinforcement learning models, enabling them to learn from historical experience.

[0020] Step 102: Construct a learning task framework consisting of a state space, an action space, and a reward function; the state space is used to represent the running state of each virtual machine, the action space includes all feasible transfer operations, and the reward function is used to evaluate the overall effect of the transfer operations. In the training process of the transfer policy generation model in this embodiment of the invention, the guided policy and the execution policy are learned separately using only the data in the dataset in a supervised manner. The training process is divided into adaptation processing, guided policy training, and execution policy training. Adaptation processing includes the design of the state space, action space, and reward function.

[0021] The state space is a structured vector composed of parameters after data collection and preprocessing. It represents the sum of all possible environmental information observable by the migration strategy generation model in this embodiment of the invention. It is a structured vector composed of preprocessed cloud platform parameters. This vector aims to comprehensively characterize the system's health, load, and performance, mainly including: host resource metrics: such as CPU utilization, memory usage, dirty page rate, disk I / O, network bandwidth utilization, etc.; virtual machine state data: such as virtual CPU utilization, memory allocation, workload characteristics, etc.; network state data: such as network latency between physical hosts, available bandwidth, etc.; and storage performance data: such as shared storage access latency, IOPS, etc.

[0022] The action space defines the set of all executable operations that the migration strategy generation model in this embodiment of the invention can perform after observing a state. It includes discrete actions and continuous actions. Discrete actions represent the migration decision logic, including migrate / not migrate, target host selection (e.g., Host_1, Host_2, ...), while continuous actions represent fine-grained control parameters of the migration process, including migration bandwidth allocation ratio (0.1~1.0), memory compression intensity, etc.

[0023] The reward function, based on the migration strategy generation model in this embodiment, generates migration actions and provides a numerical feedback (reward or penalty) to evaluate the effectiveness of the action. It is typically a weighted sum of multiple reward components, including a base reward and an optimization reward. The base reward is based on the final migration result, such as +1.0 for successful migration and -10.0 for failed migration (business interruption). The optimization reward is related to the global optimization goals of the cloud platform after migration, such as rewards for reduced migration time (seconds) and resource balance.

[0024] In this embodiment of the invention, system observation dimensions (such as CPU load, dirty page rate, network latency, etc.) are explicitly defined through the state space, all executable migration operations (such as whether to migrate, target host selection, bandwidth allocation, etc.) are delineated through the action space, and a quantitative evaluation standard for decision-making effectiveness is established through the reward function (comprehensively considering migration success rate, time consumption, resource balance, etc.). This framework transforms the migration decision-making process from a static process dependent on manual rules into a dynamic learning problem that can be driven by data and continuously optimized, laying the foundation for subsequent autonomous model learning to achieve resource optimization.

[0025] Step 103: Based on the historical running data, train the state value function, the guidance strategy, and the execution strategy within the learning task framework; the state value function is used to evaluate the long-term value of the state, the guidance strategy is used to plan the target state, and the execution strategy is used to generate specific transition actions. The core task of the state-value function is evaluation and prediction. Given a system state (i.e., a vector in the state space), it outputs a numerical value to predict the long-term cumulative expected return that can be obtained starting from that state and following the current policy. It does not directly determine actions but rather scores the state. A higher score indicates a state that is more beneficial to the system in the long run (e.g., more balanced, better performance, lower risk). Training relies on state transition sequences from historical running data, learning by continuously narrowing the difference between the predicted value of the current state and the "actual" target value.

[0026] The core task of the bootstrapping strategy is to set a goal. Given the current system state, it is responsible for planning and outputting an ideal target state. For example, based on the current high load state of the cluster, it plans an ideal target state of "load balancing across all hosts." The recommended target state should, as far as possible, maximize the evaluation result of the state value function V. The recommended target state should not deviate too far from the successful strategies reflected in historical data, thereby ensuring that the recommended goal is feasible and safe.

[0027] The core task of the execution strategy is to generate specific actions. Given the current state and the target state, it is responsible for outputting a specific, executable migration action. It receives the goal set by the bootstrapping strategy and translates it into an actual action, such as: "Migrate virtual machine VM001 from host_A to host_B and allocate 50% of the bandwidth." Its training involves learning to reproduce actions that successfully achieve transitions from similar states from offline data.

[0028] In this embodiment of the invention, a state value function is first trained using historical data, enabling it to accurately assess the long-term value of any system state and provide value guidance for decision-making. Then, a guidance strategy is trained, enabling it to plan a target state with high long-term value from the current state under the guidance of the state value function, thus completing strategic decision-making. At the same time, an execution strategy is trained, enabling it to generate specific transition actions that can achieve state transition under any given current state and target state, thus completing tactical execution.

[0029] Step 104: Integrate the trained state-value function, guidance policy, and execution policy to obtain the transfer policy generation model; The transfer strategy generation model is trained through offline reinforcement learning and integrates three components: a state-value function, a guiding policy, and an execution policy. During runtime, the guiding policy first plans an ideal target state based on the real-time system state, and then the execution policy generates specific transfer actions based on the current state and the target state.

[0030] In this embodiment of the invention, the state value function serves as a value assessment module, providing a decision-making basis for the guidance strategy; the guidance strategy, as a planning module, is responsible for generating the target state based on the real-time state; and the execution strategy, as an action generation module, outputs specific migration actions based on the current state and the target state provided by the guidance strategy. This integrated migration strategy generation model retains the specialized functions of each component, providing a complete solution for subsequent online intelligent migration.

[0031] Step 105: Generate a hot migration strategy for the virtual machine based on the migration strategy generation model.

[0032] A hot migration strategy is the final decision output by the migration strategy generation model, representing the specific set of operational instructions for completely migrating virtual machines from the source physical host to the target host. It includes not only macro-level decision instructions (such as "whether to migrate" and "which specific target host to migrate to"), but also micro-level control parameters (such as the proportion of network bandwidth dynamically allocated for this migration task). Based on in-depth analysis of the real-time status of the cloud platform (such as host load, network conditions, and virtual machine performance), it is generated after comprehensively balancing multiple optimization objectives such as migration success rate, impact on business performance, and resource balance. The ultimate goal is to automate and optimize the virtual machine migration process.

[0033] In this embodiment of the invention, by inputting the real-time collected system status (such as host CPU load, memory dirty page rate, network latency, etc.) into the deployed migration strategy generation model, its internal collaborative working mechanism is triggered: first, the guiding strategy analyzes the current status and plans the optimal target status, and then the execution strategy generates a complete hot migration strategy that includes specific migration actions (such as target host selection) and control parameters (such as bandwidth allocation) based on the combination of the current status and the target status.

[0034] Reference Figure 2 The diagram illustrates a flowchart of another virtual machine hot migration management method provided by an embodiment of the present invention. The method specifically includes the following steps: Step 201: Obtain historical running data for each virtual machine in the cloud platform environment; In this embodiment of the invention, acquiring historical runtime data of each virtual machine in the cloud platform environment is the data foundation for realizing intelligent migration decisions. A multi-dimensional, large-scale dataset is collected and integrated, encompassing host resource metrics (such as CPU and memory utilization), virtual machine status (such as dirty page rate), network performance (such as latency and bandwidth), and historical migration records (such as success rate and time consumption).

[0035] Reference Figure 3 The diagram illustrates the architecture of a virtual machine hot migration management method provided by an embodiment of the present invention. Figure 3 The demonstration shows that the hot migration management center, which is responsible for the hot migration management of virtual machines in the data center computing cluster, receives the running data of each virtual machine in the cluster provided by the cloud platform, including network status, resource utilization, migration records and effects, and trains and deploys models based on the data to perform hot migration scheduling and management of virtual machines on each server in the cluster.

[0036] Step 202: Construct a learning task framework consisting of a state space, an action space, and a reward function; the state space is used to represent the running state of each virtual machine, the action space includes all feasible transition operations, and the reward function is used to evaluate the overall effect of the transition operations. In this embodiment of the invention, constructing a learning task framework consisting of a state space, an action space, and a reward function essentially formalizes the complex problem of virtual machine hot migration management into a reinforcement learning task. The state space explicitly defines system observation dimensions (such as CPU load, dirty page rate, network latency, etc.), the action space delineates all executable migration operations (such as whether to migrate, target host selection, bandwidth allocation, etc.), and the reward function establishes a quantitative evaluation standard for decision-making effectiveness (comprehensively considering migration success rate, time consumption, resource balance, etc.).

[0037] In some embodiments, the action space includes whether to migrate and target host selection; the target host selection is determined by the migration strategy generation model through offline learning of the reward function and host resource information in the state space.

[0038] This invention clarifies the core actions of model decision-making. Traditional methods typically require a pre-verification round (such as querying a resource database) to confirm the availability of the target host when selecting it. However, in this invention, the target host selection is determined through offline learning of the model using the migration strategy. This means that the model does not query in real-time during decision-making, but rather autonomously learns the rule that "selecting a target host will only yield a high reward when it has sufficient resources" through repeated analysis of the reward function (i.e., the evaluation criterion for the decision's consequences) and host resource information in the state space during long-term offline training.

[0039] In some embodiments, the reward function is obtained by weighted summation of a base reward, a performance reward, and an optimization reward; the base reward is determined based on the migration result; the performance reward is related to the efficiency indicators of the migration process; and the optimization reward is related to the degree of improvement in the overall resource status of the cloud platform after the migration is completed.

[0040] The reward function in this embodiment of the invention employs a multi-objective weighted fusion method. The basic reward is directly related to the success or failure of the migration (positive reward for success, severe penalty for failure), ensuring the basic safety line of the business and the bottom line of the model's decision-making. The performance reward focuses on the quality of the migration process; it is negatively correlated with efficiency indicators (such as migration time and business downtime), incentivizing the model to find faster migration solutions with less impact on the business. The optimization reward focuses on the long-term value and overall benefits of the migration, positively correlated with the improvement of the overall resource status of the cloud platform after migration (such as cluster load balancing and energy consumption), guiding the model to make strategic decisions that not only solve current problems but also improve global resource utilization.

[0041] Step 203: Based on the historical running data, train the state value function, the guidance strategy, and the execution strategy within the learning task framework; the state value function is used to evaluate the long-term value of the state, the guidance strategy is used to plan the target state, and the execution strategy is used to generate specific transition actions; In this embodiment of the invention, the purpose of training the state value function, guidance strategy, and execution strategy is to combine the learning task framework with collected historical data, and to train three components that each perform their own functions and work collaboratively through offline reinforcement learning. First, the state value function is trained using historical data, enabling it to accurately assess the long-term value of any system state and provide value guidance for decision-making. Then, the guidance strategy is trained, enabling it to plan a target state with high long-term value from the current state under the guidance of the state value function, thus completing strategic decision-making. At the same time, the execution strategy is trained, enabling it to generate specific transition actions that can achieve state transitions under any given current state and target state, thus completing tactical execution.

[0042] In some embodiments, step 203 may include the following sub-steps: Sub-step S11: Preprocess the historical running data and construct a dataset based on the preprocessed historical running data; In this embodiment of the invention, the raw, heterogeneous historical running data obtained from the cloud platform is transformed into a standardized dataset that can be used for offline reinforcement learning model training.

[0043] First, a data cleaning process removes outliers and erroneous data caused by sensor malfunctions or monitoring jitter (such as invalid readings with CPU utilization exceeding 100%), and intelligently completes missing values ​​for key features (e.g., proactively detecting and completing network latency). Next, the cleaned data undergoes standardization and structuring, including normalizing numerical features (such as CPU / memory utilization), encoding categorical features (such as storage type), and aligning and synchronizing information from different data sources (such as host monitoring, network logs, and storage records) using a unified clock reference. Finally, by downsampling and aggregating high-frequency monitoring data (e.g., aggregating second-level data into minute-level granularity), key trend information is preserved while significantly reducing data size and computational complexity, thus constructing a high-quality, low-noise offline dataset for subsequent model training.

[0044] In some embodiments, step S11 may include the following sub-steps: Sub-step S111: Clean the historical running data and fill in any missing values ​​in the historical running data; Sub-step S112 involves standardizing the numerical features and encoding the categorical features in the cleaned running data. Sub-step S113 involves synchronizing the standardized running data and the encoded running data with a clock, and then downsampling the synchronized running data.

[0045] Data preprocessing includes data cleaning, data standardization and encoding, time alignment, and sampling. The data cleaning stage removes instantaneous peaks caused by sensor malfunctions (e.g., CPU utilization > 100%), filters data noise, and triggers proactive detection and completion when key features (e.g., network latency) are missing. The data standardization and encoding stage performs Min-Max normalization on metrics such as CPU / memory and One-Hot encoding on storage types (NVMe / HDD). The time alignment and sampling stage synchronizes different data sources (e.g., host monitoring and storage logs) using a unified clock (PTP / NTP calibration), performs timestamp alignment, aggregates high-frequency data (e.g., data collected per second) into a 1-minute granularity, and reduces computational overhead through downsampling.

[0046] In this embodiment of the invention, data cleaning is mainly responsible for identifying and removing outliers, isolated points, and obviously erroneous data (e.g., invalid readings with CPU utilization exceeding 100%) caused by sensor failures, transmission errors, or instantaneous system jitter. Meanwhile, missing value completion addresses information gaps caused by data stream interruptions; for key features (such as network latency between hosts), an active detection mechanism is triggered to obtain real-time data for filling in the missing values.

[0047] The cleaned raw data is transformed into a unified numerical format that machine learning models can efficiently process. Numerical feature standardization (such as Min-Max normalization) scales metrics with vastly different dimensions and ranges (such as CPU utilization, memory usage, and network bandwidth) to a uniform interval (such as [0,1]), preventing certain features from dominating model training due to their excessively large values. Categorical feature encoding converts non-numerical categorical variables (such as whether the storage type is NVMe or SATA) into binary vectors that the model can understand.

[0048] Clock synchronization utilizes protocols such as NTP / PTP to unify data from different servers and monitoring components, which may exhibit slight clock drift, onto a precise timeline. This ensures that a snapshot of the state at a given "moment" is composed of all data occurring at the same time, which is crucial for accurately reconstructing the system state. Subsequently, downsampling processing aggregates high-frequency, fine-grained data (e.g., once per second) into lower-frequency, coarse-grained data (e.g., an average value per minute). This significantly reduces the total amount of data without losing key trend information, effectively lowering the computational load and storage overhead for subsequent model training.

[0049] Sub-step S12: Based on the dataset, train the state value function, guidance strategy, and execution strategy within the learning task framework.

[0050] In this embodiment of the invention, a dataset is constructed, and within a defined reinforcement learning task framework (state space, action space, reward function), the three core components are trained separately but collaboratively through a decoupled offline reinforcement learning process.

[0051] The training process begins by training the state-value function using state transition sequences from the dataset. Through temporal difference learning, the system learns to accurately assess the long-term value of any given state. Subsequently, based on the learned state-value function, a guidance policy is trained. This policy, through a joint objective combining value maximization and behavioral constraints, learns how to plan a high-value and feasible target state from the current state. Simultaneously, an execution policy is trained. Through conditional behavior cloning, the policy learns to generate specific transfer actions that achieve the given state transition, given both the current and target states. This decoupling and step-by-step training of "value assessment," "target planning," and "action execution" prepares the components for integrating a powerful and intelligent transfer policy generation model.

[0052] In some embodiments, step S12 may include the following sub-steps: Sub-step S121: Sample a state transition sequence from the dataset; the state transition sequence includes the current state of each virtual machine, the migration action performed, and the target state; Sub-step S122: Determine the reward obtained from performing the migration action using the reward function based on the performed migration action; Sub-step S123 involves training the state value function with the objective of minimizing the difference between the predicted value of the state value function for the current state and the predicted value of the reward plus the predicted value for the target state.

[0053] The goal of the guiding policy is to instruct the execution policy on which state to proceed to. To achieve this, a state-value function V(s): S→R is trained. V is trained using only (s, s') samples from an offline dataset. To approximate the optimal value function in the dataset, an asymmetric L2 loss is obtained by using predictable regression with different weights.

[0054]

[0055] The purpose of this formula is to train the state-value function V(s) to more accurately predict the long-term expected reward starting from the current state s. V(s') - V(s) is the core of the temporal difference error, calculating the difference between the "predicted value for the target state s'" and the "predicted value for the current state s". r is the immediate reward obtained from the dataset, representing the immediate feedback given by the environment after taking an action in state s. The training objective of the state-value function V is to make the predicted value of V(s) as close as possible to r + γV(s'), where γ is a discount factor that weighs the current and future rewards. Therefore, its loss function is to minimize (V(s) - (r + γV(s')))².

[0056] Maximizing the guiding policy relative to V(s) may lead to incorrect generalization states in the execution policy. To mitigate this problem, a behavioral constraint is added to the learning objective of the guiding policy. The guiding policy is represented as... : S→S, where It is a behavior guidance policy estimated from the dataset. The weight α plays a trade-off role, balancing the guidance space for high returns with the space in which the execution policy should have the correct generalization ability.

[0057]

[0058] The purpose of this formula is to train the guiding policy g(s) so that it learns to recommend a high-value target state s' based on the current state s. φ (s') (Value maximization term) s' is the target state recommended by the guiding strategy. V φ(s') is the evaluation value of the state-value function with respect to s', encouraging and guiding the policy g to recommend target states that V deems to have high long-term value. For example, recommending a state where "all hosts have balanced CPU load". αlog g μ (s'|s) (behavioral constraint or entropy regularization term), g μ (s'|s) is the probability that policy g recommends target state s' in the current state s, log g μ (s'|s) is its logarithmic probability. α is a hyperparameter used to balance the two objectives of "value maximization" and "behavioral constraints".

[0059] In this embodiment of the invention, training samples with temporal correlations are extracted from the preprocessed dataset. Each sample is a state transition sequence, specifically including: the current state, which is the complete state of the cloud platform system at a certain historical moment; the migration action performed, which is the specific migration operation taken by the system in the current state (e.g., "migrate virtual machine VM001 to host B"); and the target state, which is the new system state observed at the next moment after performing the above migration action.

[0060] By inputting the executed transition action and its resulting target state into a predefined reward function, the immediate reward for that state transition is calculated. This reward value comprehensively reflects the immediate effects of this transition action in terms of success rate, performance overhead, and resource optimization.

[0061] The temporal difference learning method is employed, which minimizes a loss function: the predicted value V(s), which is the long-term value predicted by the state-value function for the current state. The target value r+γV(s') is a target value that is closer to the true value, consisting of an immediate reward and a discounted target state predicted value, where γ is the discount factor. The training process involves continuously adjusting the parameters of the state-value function V to make its predicted value V(s) for the current state as close as possible to the target value.

[0062] In some embodiments, step S12 may include the following sub-steps: Sub-step S124: Estimate a historical guidance strategy from the dataset; Sub-step S125: Evaluate the value of the target state output by the historical guidance strategy through the state value function; Sub-step S126 involves training the guidance strategy with the goal of maximizing the evaluation value of the state value function for the target state output by the historical guidance strategy and minimizing the deviation between the target state and the target state output by the historical guidance strategy.

[0063] In this embodiment of the invention, an empirical strategy representing historical decision-making patterns is inferred from an offline dataset that records a large number of successful and failed experiences. This historical guidance strategy is a probability distribution obtained from the data through statistical learning regarding "what target state is ultimately tended to be reached in the past states". The value of the historical target state is evaluated by using a trained state-value function (V) to measure the long-term effect of historical decisions. The long-term value V(s') of the target state recommended by the historical guidance strategy is calculated using the state-value function V.

[0064] The goal of training a new guidance policy is to develop a novel policy. The target state recommended by the new guidance policy should yield a higher long-term value, as evaluated by the state-value function, than that of historical policies. Simultaneously, the target state recommended by the new guidance policy *g* should not deviate excessively from the target states recommended by historical guidance policies.

[0065] In some embodiments, step S12 may include the following sub-steps: Sub-step S127: Extract the migration actions that were successfully performed in the preset current state and target state from the dataset; Sub-step S128 involves training the execution strategy to maximize the probability of selecting the successful transfer action under the same preset current state and target state.

[0066] Since the execution policy has strong generalization capabilities, reinforcement learning through a supervised learning framework is adopted. During the training process, π performs supervised learning by maximizing the probability of actions in a given state and the next state, thereby deriving the following objective.

[0067]

[0068] This formula is a mathematical expression of the training objective of the execution policy. θ represents the model parameters of the execution policy π (e.g., the weights of a neural network). The training objective is to find an optimal set of parameters θ that maximizes the value of the expected term in the formula. The expected value (E_{(s, a, s')∈D}) represents the expected value, which is the average of all samples in dataset D. (s, a, s')∈D represents each state transition sequence sampled from dataset D, where s is the current state, a is the successful transition action performed in state s, and s' is the target state reached after performing action a. The logarithmic probability (logπ) θ (a|s,s')),π θ (a|s, s') is the execution policy model. The inputs are the current state s and the target state s', and the output is the probability distribution over all possible actions. That is, it predicts the probability of each action a being chosen given s and s'. π θ(a|s, s') specifically refers to the probability that the model assigns to the successful action 'a' that actually occurred in the dataset. `log` is the logarithmic operation, which transforms multiplication into addition, making the optimization process more stable and efficient.

[0069] In this embodiment of the invention, successful transfer cases proven by historical experience are filtered and extracted from an offline dataset. The specific transfer actions executed under a particular combination of current and target states that ultimately lead to successful transfer are identified. The training optimization objective is to maximize the probability that the execution policy selects the recorded successful transfer action when encountering the same combination of current and target states. Once the guiding policy plans a target state, the execution policy can generate the most likely specific and precise transfer actions to successfully achieve that target from historical experience.

[0070] Step 204: Integrate the trained state-value function, guidance policy, and execution policy to obtain the transfer policy generation model; In this embodiment of the invention, the core lies in the systematic integration of three independently trained components—the state value function, the guidance strategy, and the execution strategy—to construct a unified and operable intelligent decision-making entity migration strategy generation model. The state value function, as a value assessment module, provides the decision-making basis for the guidance strategy; the guidance strategy, as a planning module, is responsible for generating the target state based on the real-time state; and the execution strategy, as an action generation module, outputs specific migration actions based on the current state and the target state provided by the guidance strategy.

[0071] Step 205: Generate a hot migration strategy for the virtual machine based on the migration strategy generation model.

[0072] In this embodiment of the invention, the trained and integrated migration strategy generation model is applied to a real cloud platform environment to achieve a closed loop from real-time state awareness to intelligent decision output. By inputting the real-time collected system state (such as host CPU load, memory dirty page rate, network latency, etc.) into the deployed migration strategy generation model, its internal collaborative working mechanism is triggered: first, the guiding strategy analyzes the current state and plans the optimal target state; then, the execution strategy generates a complete hot migration strategy that includes specific migration actions (such as target host selection) and control parameters (such as bandwidth allocation) based on the combination of the current state and the target state.

[0073] In some embodiments, step 205 may include the following sub-steps: Sub-step S21: Obtain the real-time running data of each virtual machine, and generate a hot migration strategy for the virtual machine based on the real-time running data of each virtual machine through the migration strategy generation model.

[0074] In this embodiment of the invention, real-time operational data of each virtual machine (such as CPU load, dirty page rate, network latency, etc.) is first continuously acquired. After undergoing the same preprocessing process as in the training phase, this data constitutes a current system state vector that the model can understand. Subsequently, this real-time state vector is input into the deployed migration strategy generation model. Internally, the guiding strategy first performs macro-planning based on the current state and outputs a high-value target state. Then, the execution strategy receives the current state and the target state and generates specific, executable migration actions required to achieve the target (e.g., "migrate virtual machine VM001 to host B and allocate 40% of the bandwidth").

[0075] In some embodiments, step 21 may include the following sub-steps: Sub-step S211: Input the current state of each virtual machine in the acquired real-time running data into the guidance strategy of the migration strategy generation model, and output the recommended target state; Sub-step S212: Input the current state and the recommended target state into the execution strategy of the migration strategy generation model, and output a virtual machine hot migration strategy that includes migration actions and the target host.

[0076] The trained migration strategy model is deployed to the cloud platform, where the system status is monitored in real time, and migration decisions are output based on the strategy model. In actual processing, this includes the output of a bootstrapping strategy and an execution strategy. After the virtual machine hot migration management center deploys the strategy model, it obtains the bootstrapping strategy, i.e., the next optimal state s2, based on the real-time monitored system state s1 and the current strategy model. This optimal state is a set of states including system resource utilization, network status, etc. When the virtual machine hot migration management center obtains the next optimal state s2, the model obtains the execution strategy for that state. This execution strategy is a set of execution actions, including migration actions, migration targets, bandwidth allocation ratios, etc.

[0077] In this embodiment of the invention, the real-time collected and preprocessed cloud platform system state is input into a pre-trained guidance strategy. Based on value judgments learned during offline training, the guidance strategy analyzes the current state and outputs a recommended target state s2 with high long-term value. This target state is an ideal future state of the system, indicating the strategic objective that the migration operation needs to achieve.

[0078] The current state s1 and the recommended target state s2 are input into the execution policy (π). The execution policy translates the macro-level strategic goal into specific, actionable instructions. Based on the "state-target-action" mapping learned by imitating a large number of successful experiences in offline training, a complete virtual machine hot migration policy that can be immediately delivered to the cloud platform for execution is output. This policy includes specific migration actions and the target host.

[0079] In some embodiments, after outputting the virtual machine hot migration policy, the following steps are further included: The virtual machine hot migration policy is compared with the preset operation and maintenance rules for security verification; if the virtual machine hot migration policy conflicts with the operation and maintenance rules, the virtual machine hot migration policy is rejected.

[0080] In this embodiment of the invention, the virtual machine hot migration strategy generated by the model is compared and verified with preset operation and maintenance rules. These preset operation and maintenance rules typically include affinity / anti-affinity rules: such as "two critical services must / must never be deployed on the same physical host"; resource guarantee rules: such as "reserve a minimum amount of resources for a specific virtual machine"; business criticality rules: such as "virtual machines marked as 'core business' are prohibited from automatic migration"; and host status rules: such as "hosts marked as 'to be maintained' cannot be used as migration targets". If a generated migration strategy is detected to conflict with any preset rule (e.g., the strategy attempts to place two anti-affinity services together, or migrate a virtual machine to a host to be maintained), the strategy will be automatically rejected and its execution will be prevented. By using rule constraints, catastrophic decisions (such as interrupting core business) triggered by the model are fundamentally prevented, greatly improving the reliability and trustworthiness of the entire intelligent management system, enabling it to be safely applied to demanding production environments.

[0081] Step 206: Obtain the migration results of executing the virtual machine hot migration strategy to update the dataset; In this embodiment of the invention, the final result data of each hot migration strategy generated and executed by the model is collected, including key indicators such as migration success or failure, total time consumption, business interruption time, and changes in resource status before and after migration. This practically validated migration result data is expanded into the original dataset, enabling the dataset to continuously reflect the latest dynamic characteristics of the cloud platform and the true effectiveness of model decisions. This provides a crucial source of new knowledge for subsequent periodic retraining or incremental learning of the model, ensuring that the migration strategy generation model can continuously adapt to environmental changes, correct erroneous decisions, and learn from new successful experiences, thereby achieving continuous performance improvement.

[0082] Step 207: Based on the updated dataset, optimize and train the migration strategy generation model, and evaluate the performance of the optimized and retrained migration strategy generation model with the current migration strategy generation model.

[0083] In this embodiment of the invention, the migration strategy generation model is first optimized and trained using the expanded and updated dataset. Based on the existing model, full or incremental learning is performed using a fusion of old and new data, enabling the model to absorb the latest operational experience and adapt to changes in the cloud platform environment. After optimization training, the newly trained model version is compared with the currently online model version under a unified evaluation benchmark. Key indicators include migration success rate, resource optimization degree, and decision-making efficiency. This evaluation mechanism ensures that only new models proven to have equivalent or better performance are deployed, effectively preventing model performance degradation and guaranteeing continuous optimization of cloud platform management decisions and the overall stability and reliability of the system.

[0084] Reference Figure 4 The diagram illustrates a flowchart of a virtual machine hot migration management method provided by an embodiment of the present invention. Figure 4 The demonstration showcases the complete process of virtual machine hot migration management: by acquiring historical running data of each virtual machine, offline reinforcement learning technology is used to train three core components in parallel: state value function, bootstrapping policy, and execution policy; then the trained components are integrated into a migration policy generation model, which is used to generate hot migration policies during online decision-making; finally, by executing the policy and collecting migration results, the performance evaluation and continuous optimization loop of the model is completed.

[0085] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0086] Reference Figure 5 The diagram illustrates a structural block diagram of a virtual machine hot migration management device provided in an embodiment of the present invention. The device includes: The runtime data acquisition module 301 is used to acquire historical runtime data of each virtual machine in the cloud platform environment; The task framework construction module 302 is used to construct a learning task framework consisting of a state space, an action space, and a reward function; the state space is used to represent the running state of each virtual machine, the action space includes all feasible transition operations, and the reward function is used to evaluate the overall effect of the transition operations. The strategy model training module 303 is used to train a state value function, a guidance strategy, and an execution strategy within the learning task framework based on the historical running data; the state value function is used to evaluate the long-term value of a state, the guidance strategy is used to plan a target state, and the execution strategy is used to generate specific transition actions. The strategy model integration module 304 is used to integrate the trained state-value function, guidance strategy and execution strategy to obtain the transfer strategy generation model; The migration strategy generation module 305 is used to generate a hot migration strategy for virtual machines based on the migration strategy generation model.

[0087] In some embodiments, the policy model training module 303 includes: The runtime data preprocessing submodule is used to preprocess the historical runtime data and construct a dataset based on the preprocessed historical runtime data; the model training submodule is used to train the state value function, guidance strategy and execution strategy within the learning task framework according to the dataset.

[0088] In some embodiments, the model training submodule includes: A state transition sequence acquisition unit is used to sample state transition sequences from the dataset; the state transition sequence includes the current state of each virtual machine, the executed migration action, and the target state; a migration reward determination unit is used to determine the reward obtained from executing the migration action through the reward function based on the executed migration action; a state value function training unit is used to train the state value function with the objective of minimizing the difference between the predicted value of the state value function for the current state and the predicted value of the reward plus the predicted value of the target state.

[0089] In some embodiments, the model training submodule includes: The historical guidance strategy acquisition unit is used to estimate a historical guidance strategy from the dataset; the target state value determination unit is used to evaluate the value of the target state output by the historical guidance strategy through the state value function; and the guidance strategy training unit is used to train the guidance strategy with the goal of maximizing the evaluation value of the target state output by the historical guidance strategy by the state value function and minimizing the deviation between the target state and the target state output by the historical guidance strategy.

[0090] In some embodiments, the model training submodule includes: The migration action acquisition unit is used to extract the migration actions that are successfully executed in the preset current state and target state from the dataset; the execution strategy training unit is used to train the execution strategy with the goal of maximizing the probability of the execution strategy selecting the successful migration action in the same preset current state and target state.

[0091] In some embodiments, the migration strategy generation module 305 includes: The hot migration strategy generation submodule is used to obtain the real-time running data of each virtual machine, and generate a hot migration strategy for the virtual machine based on the real-time running data of each virtual machine through the migration strategy generation model.

[0092] In some embodiments, the hot migration strategy generation submodule includes: The target state determination unit is used to input the current state of each virtual machine in the acquired real-time running data into the guidance strategy of the migration strategy generation model and output the recommended target state; the hot migration strategy output unit is used to input the current state and the recommended target state into the execution strategy of the migration strategy generation model and output the virtual machine hot migration strategy that includes the migration action and the target host.

[0093] In some embodiments, after outputting the virtual machine hot migration policy, the method further includes: The migration strategy verification module is used to perform security verification between the virtual machine hot migration strategy and the preset operation and maintenance rules; if the virtual machine hot migration strategy conflicts with the operation and maintenance rules, the virtual machine hot migration strategy is rejected.

[0094] In some embodiments, the runtime data preprocessing submodule includes: The data cleaning and completion unit is used to clean the historical operating data and complete the missing values ​​in the historical operating data; the data standardization unit is used to standardize the numerical features and encode the category features in the cleaned operating data; the data synchronization unit is used to synchronize the standardized operating data and the encoded operating data with a clock and to downsample the clock-synchronized operating data.

[0095] In some embodiments, the apparatus further includes: The migration result acquisition module is used to acquire the migration results of executing the virtual machine hot migration strategy in order to update the dataset; the strategy model optimization module is used to optimize and train the migration strategy generation model based on the updated dataset, and to evaluate the performance of the optimized and retrained migration strategy generation model with the current migration strategy generation model.

[0096] In some embodiments, the action space includes whether to migrate and target host selection; the target host selection is determined by the migration strategy generation model through offline learning of the reward function and host resource information in the state space.

[0097] In some embodiments, the reward function is obtained by weighted summation of a base reward, a performance reward, and an optimization reward; the base reward is determined based on the migration result; the performance reward is related to the efficiency indicators of the migration process; and the optimization reward is related to the degree of improvement in the overall resource status of the cloud platform after the migration is completed.

[0098] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0099] This invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the virtual machine hot migration management method embodiments described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0100] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the virtual machine hot migration management method embodiments described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0101] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A virtual machine hot migration management method, characterized in that, The method includes: Obtain historical runtime data for each virtual machine in the cloud platform environment; A learning task framework consisting of a state space, an action space, and a reward function is constructed. The state space is used to represent the running state of each virtual machine, the action space includes all feasible transition operations, and the reward function is used to evaluate the overall effect of the transition operations. Based on the historical operational data, a state value function, a guidance strategy, and an execution strategy are trained within the learning task framework; the state value function is used to evaluate the long-term value of a state, the guidance strategy is used to plan the target state, and the execution strategy is used to generate specific transition actions. The trained state-value function, guidance policy, and execution policy are integrated to obtain the transfer policy generation model; Based on the migration strategy generation model, a hot migration strategy for virtual machines is generated.

2. The virtual machine hot migration management method according to claim 1, characterized in that, The step of training the state value function, guidance strategy, and execution strategy within the learning task framework based on the historical operational data includes: The historical operation data is preprocessed, and a dataset is constructed based on the preprocessed historical operation data; Based on the dataset, the state value function, guidance policy, and execution policy are trained within the learning task framework.

3. The virtual machine hot migration management method according to claim 2, characterized in that, The step of training the state-value function, guidance policy, and execution policy within the learning task framework based on the dataset includes: A state transition sequence is obtained by sampling from the dataset; the state transition sequence includes the current state of each virtual machine, the migration action performed, and the target state; The reward obtained from performing the migration action is determined by the reward function based on the performed migration action. The state value function is trained with the objective of minimizing the difference between the predicted value of the state value function for the current state and the predicted value of the reward plus the predicted value for the target state.

4. The virtual machine hot migration management method according to claim 2, characterized in that, The step of training the state-value function, guidance policy, and execution policy within the learning task framework based on the dataset includes: Estimate a historical guidance strategy from the dataset; The value of the target state output by the historical guidance strategy is evaluated using the state value function. The guidance strategy is trained with the goal of maximizing the evaluation value of the state value function for the target state output by the historical guidance strategy, and minimizing the deviation between the target state and the target state output by the historical guidance strategy.

5. The virtual machine hot migration management method according to claim 2, characterized in that, The step of training the state-value function, guidance policy, and execution policy within the learning task framework based on the dataset includes: Extract the migration actions that were successfully performed in the preset current state and target state from the dataset; The execution strategy is trained with the goal of maximizing the probability of selecting the successful migration action under the same preset current state and target state.

6. The virtual machine hot migration management method according to claim 1, characterized in that, The process of generating a hot migration strategy for a virtual machine based on the migration strategy generation model includes: The system acquires real-time running data of each virtual machine and generates a hot migration strategy for the virtual machines based on the migration strategy generation model using the real-time running data of each virtual machine.

7. The virtual machine hot migration management method according to claim 6, characterized in that, The step of generating a virtual machine hot migration strategy based on the real-time running data of each virtual machine through the migration strategy generation model includes: The current state of each virtual machine in the acquired real-time running data is input into the guidance strategy of the migration strategy generation model, and the recommended target state is output. The current state and the recommended target state are input into the execution strategy of the migration strategy generation model, and the output is a virtual machine hot migration strategy that includes migration actions and the target host.

8. The virtual machine hot migration management method according to claim 7, characterized in that, After outputting the virtual machine hot migration strategy, the following is also included: The virtual machine hot migration strategy is compared with the preset operation and maintenance rules for security verification. If the virtual machine hot migration policy conflicts with the operation and maintenance rules, the virtual machine hot migration policy shall be rejected.

9. The virtual machine hot migration management method according to claim 2, characterized in that, The preprocessing of the historical operational data includes: The historical operation data is cleaned, and missing values ​​in the historical operation data are filled in; The numerical features in the cleaned operational data are standardized, and the categorical features are encoded. The standardized running data and the encoded running data are clock-synchronized, and the clock-synchronized running data is downsampled.

10. The virtual machine hot migration management method according to claim 2, characterized in that, The method further includes: Obtain the migration results of the virtual machine hot migration strategy to update the dataset; Based on the updated dataset, the migration strategy generation model is optimized and trained, and the performance of the optimized and retrained migration strategy generation model is evaluated with that of the current migration strategy generation model.

11. The virtual machine hot migration management method according to claim 1, characterized in that, The action space includes whether to migrate and target host selection; the target host selection is determined by the migration strategy generation model through offline learning of the reward function and host resource information in the state space.

12. The virtual machine hot migration management method according to claim 1, characterized in that, The reward function is obtained by weighted summation of the basic reward, performance reward, and optimization reward; the basic reward is determined based on the migration results; the performance reward is related to the efficiency indicators of the migration process; and the optimization reward is related to the degree of improvement in the overall resource status of the cloud platform after the migration is completed.

13. A virtual machine hot migration management device, characterized in that, The device includes: The runtime data acquisition module is used to acquire historical runtime data of each virtual machine in the cloud platform environment; The task framework construction module is used to construct a learning task framework consisting of a state space, an action space, and a reward function. The state space is used to represent the running state of each virtual machine, the action space includes all feasible transition operations, and the reward function is used to evaluate the overall effect of the transition operations. The strategy model training module is used to train a state value function, a guidance strategy, and an execution strategy within the learning task framework based on the historical running data; the state value function is used to evaluate the long-term value of a state, the guidance strategy is used to plan the target state, and the execution strategy is used to generate specific transition actions. The strategy model integration module is used to integrate the trained state-value function, guidance policy and execution policy to obtain the transfer policy generation model. The migration strategy generation module is used to generate a hot migration strategy for virtual machines based on the migration strategy generation model.

14. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the virtual machine hot migration management method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the virtual machine hot migration management method as described in any one of claims 1-12.

Images