Server hybrid deployment and management system based on virtualization technology

Abstract

The invention relates to the technical field of computer server resource management and network virtualization, and particularly discloses a server hybrid deployment and management system based on a virtualization technology, which comprises a uniform resource abstraction layer, a global resource sensing and modeling unit, a strategy-driven intelligent arrangement engine and a heterogeneous resource execution adapter. The uniform resource abstraction layer performs standardized abstraction on a physical server, a virtual machine and a container; the global resource sensing and modeling unit integrates the resource data and constructs a global resource model; a strategy-driven intelligent arrangement engine performs multi-objective optimization solution based on strategies and constraints to generate a scheduling scheme; the heterogeneous resource execution adapter translates the scheme into bottom executable atomic operations. According to the invention, unified management and intelligent cooperative scheduling of heterogeneous computing resources are realized, and the resource utilization efficiency and the system automation level are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer server resource management and network virtualization, and particularly relates to a server hybrid deployment and management system based on virtualization technology. BACKGROUND

[0002] In the field of information technology, a data center is a core infrastructure supporting various business applications, and the resource management and scheduling efficiency thereof is directly related to the reliability, cost-effectiveness and business agility of services. With the rapid development of cloud computing, edge computing and other technologies, the computing resource forms inside the data center are increasingly diversified, forming a complex environment in which physical servers, virtual machines and containers coexist.

[0003] The deployment and scheduling management of server resources is a key link in the operation and maintenance of a data center. The core goal is to efficiently and intelligently allocate and adjust computing, storage and network resources according to the dynamic changes of business loads, so as to guarantee service performance and optimize resource utilization.

[0004] Using independent management systems or tools to monitor and deploy physical machines, virtual machines and containers respectively leads to the dispersion of state information of heterogeneous resources and the fragmentation of management perspectives. This fragmented management mode makes it difficult for operation and maintenance personnel to obtain a unified global resource view and effectively evaluate the overall load and capacity conditions across resource types. When dynamic adjustment or fault migration of resources is needed, due to the lack of collaborative scheduling mechanism across heterogeneous resources, resource allocation is often rigid and response is slow, making it difficult to achieve globally optimal resource orchestration and elastic scaling according to business priority and real-time load. SUMMARY

[0005] The purpose of the present application is to provide a server hybrid deployment and management system based on virtualization technology to solve the technical contradictions of fragmented management of heterogeneous resources, lack of global view and difficulty in collaborative scheduling across resources in the prior art.

[0006] The present application provides a server hybrid deployment and management system based on virtualization technology, which comprises a unified resource abstraction layer, a global resource perception and modeling unit, a policy-driven intelligent orchestration engine and a heterogeneous resource execution adapter.

[0007] The unified resource abstraction layer is used for standardized abstract description of three kinds of heterogeneous computing resources in the data center, i.e. physical servers, virtual machines and containers. The abstraction layer defines a unified resource description model, which includes a computing capability descriptor, a memory capacity descriptor, a storage performance descriptor, a network bandwidth descriptor and a resource state descriptor. The computing capability descriptor is quantitatively characterized by a standardized performance benchmark score, the memory capacity descriptor is recorded in units of gigabytes, the storage performance descriptor includes two indicators of read-write operation times per second and sequential read-write bandwidth, the network bandwidth descriptor is recorded in units of megabits per second, and the resource state descriptor is used to identify the ready, busy, fault or maintenance state of the resource. The abstraction layer continuously collects real-time performance indicators and state data of the above-mentioned heterogeneous resources through a lightweight agent program deployed on each physical server, and formats and packages the data according to the unified resource description model to generate standardized resource description objects.

[0008] The global resource perception and modeling unit is connected to the unified resource abstraction layer and is used to receive and integrate the standardized resource description objects reported by the unified resource abstraction layer, and to build and maintain a global and real-time resource topology and state model. The unit includes a resource topology discovery module, a multi-dimensional load aggregation calculation module and a capacity prediction module. The resource topology discovery module automatically identifies and records the host and slave relationship between physical servers, virtual machines and containers, forming a resource relationship map reflecting the physical carrying and virtualization hierarchy. The multi-dimensional load aggregation calculation module performs weighted aggregation calculation on the computing, memory, storage and network loads based on the received resource description objects, according to different logical dimensions such as resource pool, service cluster or physical rack, to generate a multi-dimensional load heat map. The capacity prediction module predicts the future resource demand trend in a specific time window based on the time series of historical load data, using an autoregressive integrated moving average model, and outputs a capacity warning signal.

[0009] The policy-driven intelligent orchestration engine is the decision-making core of the system, and its input end is connected to the global resource perception and modeling unit, and the output end is connected to the heterogeneous resource execution adapter. The engine has a built-in policy library, a constraint solver, and a benefit evaluator. The policy library preloads multiple resource scheduling strategies, including load balancing strategy, energy saving and consumption reduction strategy, high availability strategy, and business priority guarantee strategy. When receiving a resource adjustment request or a pre-warning triggered by the capacity prediction module, the constraint solver performs multi-objective optimization solving based on the activated strategy, the global resource topology and state model, and the preset hard constraint conditions. The hard constraint conditions include but are not limited to the anti-affinity rules of virtual machines and containers, the resource upper limit of physical servers, and the network delay threshold. The solving process is guided by minimizing the objective function value, which takes into account multiple factors such as resource fragmentation rate, overall energy consumption, risk cost of violating service level agreement, and migration overhead. The benefit evaluator is used to simulate and score the candidate scheduling scheme generated by the constraint solver, and finally selects the scheme with the highest score to generate specific resource orchestration instruction sequences.

[0010] The heterogeneous resource execution adapter is connected between the policy-driven intelligent orchestration engine and the underlying resource management interfaces of various types, and is used to translate and adapt the unified resource orchestration instruction sequences issued by the engine into atomic operation commands that can be recognized and executed by the underlying heterogeneous resources. The adapter includes an instruction translator, an atomic operation executor, and a rollback manager. The instruction translator maintains an operation mapping table across resource types, and maps abstract instructions such as "migrate workload A" into application programming interface call sequences for virtual machine live migration or container rescheduling commands according to the actual type of workload A, which is either a virtual machine or a container. The atomic operation executor is responsible for sequentially executing the translated atomic commands in a transactional manner and monitoring the execution results of each step. The rollback manager automatically triggers a predefined rollback process to restore the system state to the state before the instruction sequence execution when it detects that any atomic operation has failed, ensuring the consistency of the operation.

[0011] Further, the lightweight agent program deployed in the unified resource abstraction layer works with an event-driven architecture. The agent program has an embedded index collector, an event filter, and a compression uploader. The index collector polls the performance counters and state information of the host resources at a frequency of 1 per second. The event filter compares the raw data collected with the threshold and detects changes, and only when the index value exceeds the preset threshold or the state changes, it is determined as a valid event. The compression uploader performs differential compression on the valid event data and sends it to the central controller of the unified resource abstraction layer through a reliable data transmission protocol.

[0012] Further, the multi-dimensional load aggregation calculation module in the global resource perception and modeling unit adopts a hierarchical weighting algorithm. Firstly, at the physical server level, the loads of all virtual machines and containers running on the physical server are summed up according to resource types, and the reserved overhead of the physical server itself is added, to obtain the real load of the physical server. Subsequently, at the resource pool or cluster level, the loads of all physical servers belonging to the logical range are accumulated. In the calculation process, the memory and storage loads are directly summed up, and the computing and network loads are aggregated after being given different weight coefficients according to business characteristics, to more accurately reflect the resource pressure under mixed loads.

[0013] Further, the constraint solver in the policy-driven intelligent orchestration engine adopts an optimization solving process based on a genetic algorithm. In the initialization of the process, a group of scheduling schemes satisfying the basic constraints are randomly generated as the initial population. In each generation evolution, part of the scheduling decisions in different schemes are exchanged through the crossover operation, and the deployment position of a certain decision is randomly adjusted through the mutation operation. Subsequently, the benefit evaluator calculates the fitness score of each newly generated individual. The fitness score is the reciprocal of the objective function value. The algorithm iteratively performs the selection, crossover, mutation and evaluation steps until the optimal fitness score of the population of the last 10 generations no longer improves or reaches the maximum iteration number of 1000 times, and the individual with the highest fitness score in the current generation is decoded as the final resource scheduling scheme.

[0014] Further, the rollback manager in the heterogeneous resource execution adapter implements an operation log-based reverse recovery mechanism. Before executing the resource orchestration instruction sequence, the rollback manager first records the current state snapshot of all target resources and generates a globally unique transaction identifier. Subsequently, the atomic operation executor records each atomic command and the result state after the execution of the command in the operation log, and associates it with the transaction identifier. When rollback is needed, the rollback manager retrieves the operation log according to the transaction identifier, sends the reverse operation commands corresponding to the original atomic commands in the reverse order of the execution order, and restores the state of all affected resources to the recorded snapshot state.

[0015] Further, the system further comprises a policy learning and optimization feedback loop. The loop continuously collects scheduling decisions issued by the policy-driven intelligent orchestration engine and the actual benefit data after the execution of the scheduling decisions, to form a decision effect sample library. The offline training module periodically reads the data in the sample library, fine-tunes the policy parameters in the policy library using a deep reinforcement learning algorithm, to make them more suitable for changes in the actual operation environment and business mode of the data center, thereby realizing continuous and autonomous evolution of the intelligent level of system scheduling.

[0016] Compared with the prior art, the present application has the following advantages: 1. The application establishes a standardized description model and data collection channel for the three heterogeneous resources of physical servers, virtual machines and containers through the design of a unified resource abstraction layer, fundamentally solving the problem of information dispersion and inconsistent formats in traditional management mode. The unified resource view provided by this layer enables the upper management system to understand and operate all types of computing resources in a consistent manner, laying a solid data foundation for global collaborative scheduling.

[0017] 2. The policy-driven intelligent orchestration engine constructed in the application converts the resource scheduling problem into a mathematical problem of seeking multi-objective optimal solution under multiple constraint conditions. By integrating multiple business policies and using intelligent optimization algorithms for automatic solution, the system can comprehensively consider multiple dimensions such as load balancing, energy efficiency, reliability and business priority, and automatically generate a resource orchestration scheme close to the global optimal. This replaces the traditional scheduling mode which relies on manual experience, has slow response and is difficult to balance the interests of multiple parties, greatly improving the scientificity, automation level and overall efficiency of resource scheduling.

[0018] 3. The heterogeneous resource execution adapter and rollback manager designed in the application effectively shields the technical differences of different virtualization platforms and management interfaces at the bottom layer, providing reliable and consistent operation execution guarantee. The instruction translation mechanism ensures that high-level policies can be delivered to the bottom layer without loss, and the rollback mechanism based on transactions and logs ensures the safety of any complex orchestration operation, even if some steps fail, it can also be automatically recovered, avoiding system state chaos or business interruption caused by operation failure, significantly enhancing the operation reliability and robustness of the system in production environment.

[0019] 4. The policy learning and optimization feedback loop introduced in the application enables the system to have the ability to learn and continuously optimize from historical decision-making effects. By continuously adjusting policy parameters to adapt to dynamically changing business load and environment, the system can gradually approach better scheduling performance, realizing the evolution from static rule configuration to dynamic intelligent adaptation, ensuring the effectiveness and advanced nature of the system during long-term operation. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is the overall technical scheme architecture diagram of the server hybrid deployment and management system based on virtualization technology proposed by the application.

[0021] Figure 2 is the core principle framework diagram of the policy-driven intelligent orchestration engine in the application.

[0022] Figure 3 is the logical flow framework diagram of the unified resource abstraction layer and global resource perception and modeling unit in the application.

[0023] Figure 4is a multi-level interaction relationship and data flow diagram of the heterogeneous resource execution adapter in the application.

[0024] Figure 5 is a principle framework diagram of the policy learning and optimization feedback closed loop in the application. DETAILED DESCRIPTION

[0025] Embodiment 1: Please refer to the attached Figures 1 to 5 This embodiment elaborates the specific technical implementation of the server hybrid deployment and management system based on virtualization technology. Please refer to the attached Figure 1 The system is composed of five core parts: unified resource abstraction layer, global resource perception and modeling unit, policy-driven intelligent orchestration engine, heterogeneous resource execution adapter, and policy learning and optimization feedback closed loop. These parts are connected through well-defined data interfaces and communication protocols, forming a complete automated closed loop from bottom resource perception, global state modeling, intelligent decision-making to secure execution.

[0026] The unified resource abstraction layer is the cornerstone of the direct interaction between the system and the underlying heterogeneous physical resources. This layer is deployed on each physical server in the data center, and its core is a lightweight agent program. The agent program works with an event-driven architecture, which contains three sub-components: index collector, event filter and compression uploader. The index collector collects real-time performance data of the host physical server and all virtual machines and containers running on it at a fixed frequency of once per second through the performance counters exposed by the operating system kernel, the management interfaces of the virtualization management program, and the monitoring application programming interfaces of the container runtime engine. These data specifically include the percentage of CPU core usage, the used memory capacity in gigabytes, the disk read and write operation frequency per second, the disk sequential read and write bandwidth in megabits per second, and the network interface receive and send bandwidth in megabits per second. At the same time, the index collector also collects the state identifiers of the resources, which include four discrete states: ready, busy, fault or maintenance.

[0027] The event filter continuously receives the raw data stream from the metric collector. It maintains a set of configurable dynamic threshold rules internally, which defines the upper and lower threshold of normal fluctuation range for each performance metric. The event filter compares each secondly collected metric value with the corresponding threshold in real time, and calculates the absolute difference between the current sampling value and the last sampling value. Only when a certain metric value continuously exceeds the threshold for 3 consecutive sampling periods, or the state identifier of a certain resource changes, the event filter determines that the data is a valid event. This design greatly reduces the transmission of invalid data and reduces the communication overhead of the system. The compression uploader is responsible for processing valid event data. It uses a differential compression algorithm to compare the current valid event data with the last uploaded baseline data, and only encodes and transmits the changed data part. The compressed data packet is sent to the central controller node of the unified resource abstraction layer through the reliable data transmission protocol based on the transmission control protocol. After receiving the data packets from thousands of agent programs in the network, the central controller node decompresses and reorganizes them.

[0028] The core responsibility of the central controller is to format the received heterogeneous data into standardized resource description objects according to the pre-defined unified resource description model. The model is a structured data pattern that includes 5 core description sub-fields. The computing power descriptor does not directly record the number of central processing unit cores or the frequency, but quantitatively represents it through a standardized performance benchmark score. This score is derived from a comprehensive score obtained by running a set of unified benchmark tests when each physical server is initialized into the network. This score can more fairly compare the actual computing power of different architecture processors. The memory capacity descriptor records the total memory capacity and the currently allocated or used memory capacity of the physical server or virtual machine in gigabytes.

[0029] The storage performance descriptor is a composite field that includes two key indicators, namely the number of read and write operations per second of the storage device and the sequential read and write bandwidth per second in megabytes. The network bandwidth descriptor records the theoretical maximum bandwidth and the currently measured available bandwidth of the network interface in megabytes per second. The resource state descriptor directly records the ready, busy, fault, or maintenance state code reported by the agent program. The central controller instantiates the corresponding resource description object for each manageable resource entity in the data center, whether it is a physical server, a virtual machine, or a container, and continuously updates the field values in the object as the agent program reports. These standardized resource description objects constitute the only true source of resource perception for the upper layer system.

[0030] The global resource perception and modeling unit is connected to the central controller of the unified resource abstraction layer, and its core task is to integrate all resource description objects, build and maintain a global, real-time, and topologically related resource state model. Please refer to the attached Figure 3The unit is realized by the resource topology discovery module, the multi-dimensional load aggregation calculation module, and the capacity prediction module. The resource topology discovery module actively queries all resource description objects from the unified resource abstraction layer, and automatically identifies the hierarchical relationship between resources by analyzing the associated attributes in the objects. For example, it determines which physical server the virtual machine runs on by parsing the host identifier field recorded in the virtual machine description object; similarly, it determines the host relationship of the container by the identifier of the virtual machine or physical node recorded in the container description object. These relationships are stored in a resource relationship graph database, which clearly reflects the complete bearing and subordinate hierarchy from the physical rack to the physical server, to the virtual machine, and finally to the container.

[0031] The input of the multi-dimensional load aggregation calculation module is the real-time refreshed resource description object set and the resource relationship graph. The module uses a hierarchical weighting algorithm for load aggregation calculation. The algorithm first aggregates at the physical server level. For a given physical server, the module finds all virtual machines and containers running on it from the resource relationship graph, and then obtains the real-time load data of these virtual machines and containers from the resource description object set, including four dimensions of computing, memory, storage, and network. At the physical server level, the loads of the memory and storage dimensions are summed directly, that is, the used amounts of all sub-loads are added. However, for the computing and network loads, direct summation may not accurately reflect the real pressure of physical resources, because different types of businesses have different competition patterns for central processors and network resources. Therefore, the algorithm introduces a weight coefficient for computing and network load.

[0032] The weight coefficient is dynamically set according to the main business type running on the physical server, for example, a business mainly based on scientific computing is given a higher weight for computing load, and a business mainly based on data throughput is given a higher weight for network load. The system reservation overhead of the physical server itself, such as the central processor and memory occupied by the virtualization management program, will also be included in the load of the server. After weighted summation and reservation overhead superposition, the real four-dimensional load vector of the physical server is finally obtained.

[0033] After the load calculation at the physical server level is completed, the module continues to aggregate the logical dimensions. The system administrator can define multiple logical groups in advance, such as a business-based resource pool, a project-based user cluster, or a physical location-based rack group. The multi-dimensional load aggregation calculation module adds up the load vectors of all physical servers belonging to the same logical group in each dimension according to the resource relationship graph and the grouping configuration. For example, the total computing load of the resource pool is calculated by adding up the weighted computing load values of all physical servers in the pool. Finally, the module generates a series of multi-dimensional load heat map data indexed by logical groups. These heat map data not only contain the absolute values of each dimension, but also contain the utilization percentage relative to the total capacity of the logical group, providing a quantitative basis for the global resource view.

[0034] The capacity prediction module focuses on trend analysis based on historical data. The module continuously extracts key resource indicators from the global resource model, such as the total CPU utilization and total memory usage of the entire data center or a specific resource pool, forming time series data. The module uses an autoregressive integrated moving average model to analyze these time series. This model analyzes the autocorrelation, difference stationarity, and moving average characteristics of historical data to fit the internal law of the data. Specifically, the model first performs a difference operation on the original time series to make it stationary, and then establishes a joint equation of autoregressive and moving average terms. Through learning from historical data, the model estimates the best parameters of the equation. In the prediction phase, the module uses the established model to extrapolate the resource demand curve for a specific time window, such as 12 hours or 24 hours in the future. When the prediction curve shows that the resource demand at a certain time point in the future will exceed the preset capacity threshold by 95%, the capacity prediction module generates a capacity warning signal. This signal contains the resource type, logical range, predicted time point, and severity level of the overrun, and is immediately sent to the policy-driven intelligent orchestration engine.

[0035] The policy-driven intelligent orchestration engine is the intelligent decision center of the system. Please refer to the attached Figure 2 The engine receives real-time resource models, multi-dimensional load heat maps, and capacity warning signals from the global resource perception and modeling unit, as well as resource adjustment requests from external management platforms or applications. The engine is composed of three components: a policy library, a constraint solver, and a benefit evaluator. The policy library is an extensible repository that predefines multiple resource scheduling policy templates, such as load balancing, energy saving, high availability, and business priority guarantee. Each policy clearly defines its optimization target and preference, such as the load balancing policy aiming to minimize the standard deviation of computing load among all physical servers, and the energy saving policy aiming to maximize the integration of workloads to put as many physical servers as possible into low-power sleep state while meeting business performance.

[0036] When the engine is triggered, the constraint solver starts working. Its first task is to determine the hard constraints that need to be satisfied by this scheduling decision. These conditions include the resource upper limit of physical servers, i.e. a single server cannot host more workloads than its CPU core number, memory capacity, etc. The anti-affinity rule of virtual machines and containers, i.e. workloads belonging to the same high-availability group cannot be deployed on the same physical server. The network latency threshold, i.e. some latency-sensitive services require their workloads to be deployed within a server group whose network round-trip latency is less than 2 ms. After the constraints and optimization objectives are clear, the constraint solver formalizes the resource scheduling problem as a multi-objective optimization problem and defines a comprehensive objective function to solve it. The objective function aims to find the optimal workload placement scheme, which needs to consider multiple factors at the same time.

[0037] The specific form of the objective function can be expressed as: minimize the resource fragmentation rate penalty term plus the overall energy consumption penalty term plus the risk cost of violating service level agreement plus the migration overhead penalty term. Among them, the resource fragmentation rate penalty term is quantified by calculating the size and distribution of the remaining resource blocks on all physical servers, encouraging the formation of large blocks of continuous available resources. The overall energy consumption penalty term estimates the instantaneous power consumption of each server according to its CPU utilization rate through the server power consumption model and sums it up. The risk cost of violating service level agreement is a function related to the priority of the workload and its current performance state, and the performance decline of high-priority workloads will cause this cost to rise sharply. The migration overhead penalty term estimates the total amount of data migration required to execute the scheduling scheme, including memory migration and storage migration, and converts it into the interference cost that may be caused to the business.

[0038] To solve this complex optimization problem, the constraint solver adopts an optimization process based on genetic algorithm. When the process is initialized, 100 initial scheduling schemes that satisfy all hard constraints are randomly generated, each defining where each workload should be placed on which physical server, and these 100 schemes constitute the initial population. Each scheme is called an individual. In each generation of evolution, the algorithm performs selection, crossover, and mutation operations. The selection operation is based on the fitness score of the individual, which is calculated by the benefit evaluator, and it is the inverse of the value of the above objective function, i.e. the smaller the value of the objective function, the higher the fitness score. Individuals with higher fitness scores have a higher probability of being selected into the next generation. The crossover operation randomly selects two selected individuals as parents and exchanges their deployment decisions for a certain part of the workload to produce new child individuals. The mutation operation randomly changes the deployment location of a certain workload in a certain individual with a small probability to ensure the diversity of the population. The newly generated child individuals also need to be checked for hard constraint conditions, and invalid individuals that do not meet the constraints are removed.

[0039] The benefit evaluator plays a key role in each generation of the genetic algorithm. For each individual in the population, i.e. each candidate scheduling scheme, the benefit evaluator does not actually execute the scheme, but rather reason about it in a sandbox simulation environment. It simulates the execution of all workload migration and placement operations defined in the scheme based on the current global resource model, and then utilizes a suite of refined models to predict the system state after the scheme execution. The model estimates the network bandwidth occupancy and duration during the migration process, predicts the performance of workloads on the new locations, calculates the new resource fragmentation rate and overall energy consumption. Finally, the benefit evaluator computes the comprehensive cost of the simulation result according to the objective function, and translates it into a fitness score to feed back to the constraint solver. The genetic algorithm iteratively performs the selection, crossover, mutation and evaluation steps until the fitness score of the best individual in the population of the last 10 generations improves by less than 1 / 1000, or the total number of iterations reaches 1000, the algorithm terminates. At this point, the individual with the highest fitness score in the current generation is decoded, i.e. the final selected resource scheduling scheme is obtained. The scheme is converted into a sequence of specific, executable resource orchestration instructions, e.g. migrate virtual machine A from physical server X to physical server Y, scale up container group B by 2 instances on physical server Z.

[0040] The heterogeneous resource execution adapter is responsible for safely and reliably executing the abstract orchestration instructions generated by the policy-driven intelligent orchestration engine on the underlying heterogeneous infrastructure. Please refer to the attached Figure 4 As an execution intermediary, the adapter connects the upper-layer unified decision engine with the lower-layer diverse resource management interfaces, including the application programming interfaces of virtualization hypervisors, the application programming interfaces of container orchestration platforms, and the out-of-band management controller interfaces. The adapter is internally composed of an instruction translator, an atomic operation executor and a rollback manager.

[0041] The instruction translator maintains a detailed cross-resource type operation mapping table. The mapping table defines how each abstract orchestration instruction is decomposed into a series of underlying atomic operation commands based on the specific type of the target workloads. For example, for the abstract instruction migrate workload A, the instruction translator first queries the global resource model to confirm that workload A is a kernel-based virtual machine instance. Then, it looks up the standardized procedure for virtual machine live migration in the mapping table, which is decomposed into a sequence of atomic operations including pre-creating the virtual machine configuration on the target host, establishing a migration transport channel, iterating through the memory pages, briefly pausing the source virtual machine, synchronizing the last memory differences, resuming the virtual machine on the target host, and releasing the source host resources. For the same abstract instruction, if workload A is a container, the mapped sequence of atomic operations is completely different, possibly including pulling the container image on the target node, creating a new container instance, switching service traffic to the new instance, and terminating the old instance. The instruction translator ensures that the high-level policy intent is losslessly and accurately translated into the underlying understandable operation language.

[0042] The atomic operation executor receives the sequence of atomic commands from the instruction translator. It adopts a transactional execution manner to ensure that the entire orchestration operation either completely succeeds or completely fails, avoiding intermediate states. Before the execution starts, the atomic operation executor collaborates with the rollback manager to create a globally unique transaction identifier for this orchestration operation. Then, the executor sequentially executes each atomic command. After executing each command, for example, calling the migration application programming interface of the virtualization hypervisor, the executor synchronously waits and confirms the return result of the application programming interface call. Only after the current atomic command is confirmed to be successfully executed, the next command is executed. With each successful execution step, the executor records the detailed information of the command, the execution timestamp, the resource state snapshot after execution, and the transaction identifier into the operation log.

[0043] The rollback manager is a key component to achieve operation safety. It implements a reverse recovery mechanism based on operation log. Before the atomic operation executor starts to execute, the rollback manager has recorded all target resource states before execution according to transaction identification. During the execution of the atomic operation, the rollback manager monitors the execution result in real time. Once the atomic operation executor reports that a certain atomic command fails to execute, for example, the virtual machine migration fails due to network interruption, the rollback manager is triggered immediately. It retrieves all successfully executed atomic command records since the beginning of the transaction according to the transaction identification of the current failed command from the operation log. Then, the rollback manager generates and sends reverse operation commands corresponding to the original commands in the opposite direction of the execution order. For example, for the successfully created virtual machine configuration command, the reverse operation is to delete the configuration; for the successfully executed memory page synchronization command, the reverse operation is to clean up the temporary data that has been transmitted. The rollback manager drives the atomic operation executor to execute these reverse commands until the state of all affected resources is restored to the initial recorded snapshot state. This mechanism ensures that even if there is a partial failure in a complex multi-step operation, the system can automatically and cleanly roll back, ensuring the stability of the business environment and the consistency of the data.

[0044] The policy learning and optimization feedback loop is an intelligent module for the system to achieve continuous self-optimization. Please refer to the attached Figure 5 , which runs continuously as a background process. Its core is to continuously collect decision input, output and actual effect data after execution of the policy-driven intelligent orchestration engine, forming a decision effect sample library. Each sample contains the following data: global resource state snapshot when triggering scheduling decision, capacity warning signal content, scheduling strategy type adopted, final scheduling scheme generated by the constraint solver, predicted score of the scheme by the benefit evaluator, actual results of executing the scheme by the heterogeneous resource execution adapter, and actual system benefit indicators collected through the unified resource abstraction layer within a period of time after the scheme is executed, such as real load balancing degree change, energy consumption change, and number of service level agreement violation events.

[0045] The offline training module is started periodically, for example, every 24 hours, and reads a large number of samples accumulated in the decision effect sample library in recent period. The module fine-tunes the strategy parameters in the strategy library using a deep reinforcement learning algorithm. Specifically, it regards the entire scheduling system as an agent, the global resource state as the environment state, and the generated scheduling scheme as the action. The actual benefit improvement brought about by the execution of the scheme is regarded as the reward. The training module constructs a deep neural network as the strategy function, the input of which is the encoded global resource state and business demand features, and the output is the probability distribution of different scheduling actions. The training process adjusts the weight parameters of the neural network by comparing the actions taken by the agent, i.e., the current strategy, in the historical state, i.e., the historical scheduling decisions, with the actual rewards, i.e., the actual benefits, obtained. The goal is to make the actions output by the strategy function obtain higher long-term cumulative rewards. After training, the new strategy parameters are updated to the corresponding strategy template in the strategy library. For example, the initial setting of the weight of different load dimensions in the load balancing strategy may be adjusted to a value that is more in line with the current business mix characteristics according to historical data. The load threshold for triggering server consolidation in the energy saving and consumption reduction strategy may also be dynamically adjusted according to the actual business tide law. Through this closed loop, the system can learn autonomously from historical experience, enabling the scheduling strategy to continuously adapt to the dynamic changes in the actual operating environment and business mode of the data center, thereby realizing the continuous autonomous evolution of the scheduling intelligence level without the need for frequent manual adjustment of strategy parameters.

[0046] Embodiment 2 provides another specific implementation of the server hybrid deployment and management system based on virtualization technology, focusing on distributed and hierarchical optimization of the global resource perception and modeling unit and the strategy-driven intelligent orchestration engine in the super-large-scale data center scenario to address the scalability and real-time decision-making challenges brought about by massive resource management.

[0047] In a super-large-scale environment, the central controller of the unified resource abstraction layer can become a performance bottleneck. Therefore, the embodiment is designed as a distributed architecture. The data center is divided into multiple logical areas, and a regional controller is deployed in each area. The regional controller is responsible for managing the lightweight agent programs on all physical servers in the region, receiving and processing the data reported by the agent programs, and generating a set of standardized resource description objects within the region. The global metadata coordinator is responsible for maintaining the registration information of all regional controllers and synchronizing the resource association relationships across regions, such as the logical connection relationships between virtual machines and containers involved in the distributed applications deployed across regions. The regional controllers communicate with each other through a high-speed data center network and exchange necessary resource summary information.

[0048] The global resource perception and modeling unit also adopts a hierarchical design. In each logical region, a regional resource modeling node is deployed, which integrates the resource description objects provided by the regional controllers to build a regional-level resource topology and load heat map. The regional resource modeling node has complete resource topology discovery, load aggregation calculation, and capacity prediction capabilities, but its field of view is limited to the region. The global resource aggregation node is responsible for integrating the summary information from all regional resource modeling nodes. The global resource aggregation node does not store the full details of all resources, but maintains a global resource summary view that records the overall resource capacity, current overall utilization, distribution of key business loads, and inter-regional network link status and bandwidth utilization of each region. When global scheduling decisions are needed across regions, the global resource aggregation node provides the macro-situation information required for decision-making.

[0049] The policy-driven intelligent orchestration engine evolves into a two-level decision-making system in this embodiment. A regional orchestration engine is deployed in each region to handle scheduling requests that occur within the region and do not involve cross-regional resource adjustment. The regional orchestration engine has a complete policy library, constraint solver, and benefit evaluator, but its decision variables and constraint conditions are limited to resources within the region. The decision-making cycle of the regional orchestration engine is shorter, for example, running once every minute, to quickly respond to load fluctuations within the region. Strategic decisions that involve cross-regional workload migration, cross-regional resource allocation, or affect global goals are handled by the global orchestration engine.

[0050] The input to the global orchestration engine comes from the summary view of the global resource aggregation node and the local decision plans and resource requirements reported by the regional orchestration engines. The global orchestration engine is used to solve higher-level optimization problems, such as balancing overall load across different regions and selecting the optimal deployment region for cross-regional applications.

Claims

1. A server hybrid deployment and management system based on virtualization technology, characterized in that, include: The unified resource abstraction layer is used to standardize and abstract the description of three types of heterogeneous computing resources in the data center: physical servers, virtual machines, and containers, and generate standardized resource description objects. The global resource awareness and modeling unit is connected to the unified resource abstraction layer and is used to receive and integrate the standardized resource description objects reported by the unified resource abstraction layer in order to build and maintain a global, real-time resource topology and state model. The strategy-driven intelligent orchestration engine, whose input is connected to the global resource awareness and modeling unit, is used to make intelligent decisions based on the global resource topology and state model, and generate a sequence of resource orchestration instructions. A heterogeneous resource execution adapter connects the policy-driven intelligent orchestration engine to various underlying resource management interfaces. It is used to translate and adapt the resource orchestration instruction sequence issued by the policy-driven intelligent orchestration engine into atomic operation commands that can be recognized and executed by the underlying heterogeneous resources.

2. The server hybrid deployment and management system based on virtualization technology according to claim 1, characterized in that, The unified resource abstraction layer defines a unified resource description model, which includes a computing power descriptor, a memory capacity descriptor, a storage performance descriptor, a network bandwidth descriptor, and a resource status descriptor. The unified resource abstraction layer continuously collects real-time performance indicators and status data of the heterogeneous computing resources through lightweight agent programs deployed on each physical server, and formats and encapsulates them according to the unified resource description model to generate the standardized resource description object.

3. The server hybrid deployment and management system based on virtualization technology according to claim 2, characterized in that, The global resource perception and modeling unit includes a resource topology discovery module, a multi-dimensional load aggregation calculation module, and a capacity prediction module; the resource topology discovery module is used to automatically identify and record the host and subordinate relationships between physical servers, virtual machines, and containers to form a resource relationship graph. The multidimensional load aggregation calculation module is used to perform weighted aggregation calculations on four types of loads—computing, memory, storage, and network—according to different logical dimensions based on the received resource description object, and generate a multidimensional load heatmap; the capacity prediction module is used to predict the resource demand trend within a specific future time window based on the time series of historical load data using an autoregressive integral moving average model, and output a capacity warning signal.

4. The server hybrid deployment and management system based on virtualization technology according to claim 3, characterized in that, The strategy-driven intelligent orchestration engine has a built-in strategy library, constraint solver, and benefit evaluator. The strategy library pre-sets multiple resource scheduling strategies. The constraint solver, upon receiving a resource adjustment request or an early warning triggered by the capacity prediction module, performs multi-objective optimization based on the activated strategy, the global resource topology and state model, and preset hard constraints to generate candidate scheduling schemes. The benefit evaluator simulates and scores the candidate scheduling schemes generated by the constraint solver, and selects the scheme with the highest score to generate a specific resource orchestration instruction sequence.

5. The server hybrid deployment and management system based on virtualization technology according to claim 4, characterized in that, The heterogeneous resource execution adapter includes an instruction translator, an atomic operation executor, and a rollback manager; the instruction translator is used to maintain an operation mapping table across resource types and translate abstract orchestration instructions into atomic operation command sequences for specific resource types; The atomic operation executor is used to execute the translated atomic commands sequentially in a transactional manner and monitor the execution results; the rollback manager is used to automatically trigger a predefined rollback process when an atomic operation execution failure is detected, restoring the system state to the state before the instruction sequence was executed.

6. The server hybrid deployment and management system based on virtualization technology according to claim 5, characterized in that, The lightweight agent deployed in the unified resource abstraction layer adopts an event-driven architecture; the lightweight agent embeds an indicator collector, an event filter, and a compressed uploader; the indicator collector is used to poll and collect the performance counters and status information of the host resources at a frequency of once per second; The event filter is used to perform threshold comparison and change detection on the collected raw data. Only when the indicator value exceeds the preset threshold or the status changes is it determined to be a valid event. The compressed uploader is used to perform differential compression processing on the valid event data and then send it to the central controller of the unified resource abstraction layer through a reliable data transmission protocol.

7. The server hybrid deployment and management system based on virtualization technology according to claim 6, characterized in that, The multidimensional load aggregation calculation module employs a hierarchical weighted algorithm; the execution process of the hierarchical weighted algorithm is as follows: First, at the physical server level, the load of all virtual machines and containers running on it is summed according to resource type, and then added to the reserved overhead of the physical server itself to obtain the actual load of the physical server; then, at the resource pool or cluster level, the load of all physical servers belonging to that logical scope is accumulated. During the calculation process, memory and storage loads are directly summed, while computation and network loads are aggregated after being assigned different weight coefficients according to business characteristics.

8. The server hybrid deployment and management system based on virtualization technology according to claim 7, characterized in that, The constraint solver adopts an optimization solution process based on a genetic algorithm; the execution process of the optimization solution process based on a genetic algorithm is as follows: during initialization, a set of scheduling schemes that satisfy the basic constraints are randomly generated as the initial population; In each generation of evolution, some scheduling decisions from different schemes are exchanged through crossover operations, and the deployment position of a certain decision is randomly adjusted through mutation operations; then the fitness score of each newly generated individual is calculated using the benefit evaluator, and the fitness score is the reciprocal of the objective function value; The algorithm iteratively executes selection, crossover, mutation, and evaluation steps until the optimal fitness score of the population no longer increases for 10 consecutive generations or reaches the maximum number of iterations of 1000. The individual with the highest fitness score in the current generation is then decoded as the final resource scheduling scheme.

9. The server hybrid deployment and management system based on virtualization technology according to claim 8, characterized in that, The rollback manager implements a reverse recovery mechanism based on operation logs; the execution process of the reverse recovery mechanism based on operation logs is as follows: Before executing the resource orchestration instruction sequence, the rollback manager first records a snapshot of the current state of all target resources and generates a globally unique transaction identifier. Subsequently, each time the atomic operation executor successfully executes an atomic command, it records the command and its resulting state in the operation log and associates it with the transaction identifier. When a rollback is required, the rollback manager retrieves the operation log based on the transaction identifier and sends the inverse operation commands corresponding to the original atomic commands in the reverse direction of the execution order until the state of all affected resources is restored to the recorded snapshot state.

10. The server hybrid deployment and management system based on virtualization technology according to claim 9, characterized in that, The system also includes a strategy learning and optimization feedback loop; the strategy learning and optimization feedback loop is used to continuously collect the scheduling decisions issued by the strategy-driven intelligent orchestration engine and the actual benefit data after their execution, forming a decision effect sample library; The policy learning and optimization feedback loop includes an offline training module, which periodically reads data from the decision performance sample library and uses a deep reinforcement learning algorithm to fine-tune the policy parameters in the policy library.

Images