A low-cost redundant deployment method and scheduling device supporting availability assurance

Abstract

The present disclosure relates to the technical field of computer, and proposes a low-cost redundancy deployment method and scheduling device supporting availability guarantee, wherein a multi-dimensional fault feature model is constructed by collecting node and link failure probability, and a differentiated strategy set containing hot backup and cold backup is established; a recovery delay weighted expectation model is constructed based on backup priority and link state, and system-level recovery failure probability is quantified; a mixed integer linear programming model is used to globally optimize under the constraints of recovery delay threshold and availability probability, with the goal of minimizing the sum of static activation cost and dynamic recovery load, and generate a ternary mapping result; finally, the ternary mapping result is converted into node affinity, resource quota and routing rule of cloud-native cluster by a controller and automatically executed. The present disclosure realizes accurate balance between resource cost and availability guarantee, effectively avoids link failure risk, and realizes business-level recovery time guarantee.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a low-cost redundant deployment method and scheduling device that supports availability assurance. Background Technology

[0002] In the fields of cloud computing and large-scale distributed systems, to ensure the continuity and high availability of critical business services, the industry typically adopts redundant deployment strategies such as cross-regional active-active deployment, dual-machine hot standby, and elastic scaling based on container orchestration platforms. These technologies, by reserving backup resources or establishing off-site disaster recovery centers, enhance the system's resilience to a certain extent.

[0003] However, traditional active-active or hot-standby architectures often employ full mirroring or a fixed-ratio static resource reservation model, lacking differentiated backup strategies for specific business functions. This results in a large number of backup resources remaining idle for extended periods during non-failure periods, leading to low infrastructure resource utilization and high maintenance costs. Furthermore, existing availability assessment standards largely rely on long-term average availability statistics (such as annual average failure rate), failing to provide refined modeling and constraints for task-level recovery time under single-failure scenarios. Consequently, when facing sudden failures, although statistical indicators may meet requirements, the actual recovery response latency can easily exceed the tolerance threshold for latency-sensitive businesses such as financial transactions and industrial control.

[0004] Furthermore, existing scheduling and recovery schemes typically only focus on the liveness of compute nodes, neglecting the risks of connectivity degradation or interruption in communication links between primary and backup nodes. This can easily lead to actual failures in complex network topologies, where backup nodes are available but data transmission paths are unreachable. Moreover, existing disaster recovery configurations largely rely on manually preset static rules or operation and maintenance scripts, lacking mathematical optimization decision-making mechanisms based on global failure probabilities, link status, and resource costs. This makes it difficult to achieve a precise balance and automated closed-loop control between resource costs and high availability in dynamically changing production environments. Summary of the Invention

[0005] This invention aims to address the technical problem of balancing resource cost and multi-dimensional availability assurance in existing distributed system redundancy deployment schemes. Existing technologies often employ static fixed replica number strategies, lacking refined modeling of network link quality degradation, node differential failure probabilities, and recovery latency under dynamic fault scenarios. This results in low resource utilization when ensuring business continuity or insufficient recovery capabilities under critical fault scenarios.

[0006] The first aspect of this invention provides a low-cost redundant deployment method that supports availability assurance, comprising the following steps: Acquire basic operational data and fault characteristic parameters of the cluster. These fault characteristic parameters include node failure probability and link failure probability, which reflects the risk of connectivity degradation between nodes. The link failure probability is calculated by weighted summation of the network packet loss rate and normalized latency, thus mapping network jitter and packet loss characteristics into quantitative risk indicators.

[0007] A set of optional backup modes is established for each function, and the resource consumption parameters and recovery time parameters corresponding to each backup mode are defined. The backup modes include at least hot backup mode and cold backup mode, which correspond to different resource consumption levels and startup and recovery times, respectively, providing decision-making space for subsequent strategy combinations.

[0008] A fault configuration space encompassing multiple potential failure scenarios is established, and the system-level recovery failure probability is quantified. Specifically, all possible combinations of node failure states are defined as the fault configuration space, and each specific combination of failure states constitutes a fault scenario. Based on the node failure probabilities, the joint probability of each fault scenario is calculated. This joint probability is the combination of the product of the failure probabilities of each node in the failed node set and the product of the survival probabilities of each node in the normal node set. Based on the redundancy coverage principle, it is determined whether the function is completely failed under each fault scenario, and the joint probabilities of all fault scenarios leading to complete functional failure are summed to obtain the system-level recovery failure probability.

[0009] Define a functional-level recovery time metric and calculate the expected recovery latency for the function under different primary / backup strategy combinations. This step introduces a priority parameter to define the order of backup nodes in the recovery sequence and constructs a weighted expected recovery latency model. This model traverses all possible recovery paths, multiplies the recovery time parameter corresponding to each path by its conditional probability of effectiveness, and sums the results to obtain the quantified value of the functional-level recovery time metric. The conditional probability comprehensively considers the probability of primary node failure, the probability of maintaining connectivity between primary and backup nodes, and the joint probability of a high-priority backup node failing while the current backup node is available.

[0010] A mixed-integer linear programming model is constructed and solved to determine a ternary mapping result containing function ID, deployment node ID, and backup strategy ID. The objective function of this model aims to minimize the total system cost, which consists of two parts: the static deployment cost and the expected cost of dynamic resource load under failure scenarios. The expected cost of dynamic resource load is obtained by traversing the failure configuration space and summing the products of the joint probability of each failure scenario and the instantaneous resource load value determined based on resource occupancy parameters for that scenario. The solution process is constrained by the following conditions: the functional-level recovery time index does not exceed a preset recovery latency threshold, and the system-level recovery failure probability meets a preset safety standard. This allows for finding the deployment scheme with the minimum resource overhead while satisfying a strict availability SLA (Service Level Agreement).

[0011] The ternary mapping results are converted into scheduling policies and automatically distributed to the cluster for execution. This step maps deployment node IDs to cluster node affinity configurations to implement physical topology constraints, and backup policy IDs to service mesh routing rules or resource limit configurations. For cold backup mode, resource quotas are set to the minimum consumption value to maintain a healthy state and traffic is blocked; for hot backup mode, the full resource quota required by the business is set and state synchronization is initiated, thereby realizing the transformation from theoretical optimization results to cloud-native deployment configurations.

[0012] A second aspect of the present invention provides a low-cost redundant deployment scheduling device that supports availability assurance, comprising: The data acquisition module is used to obtain basic operational data and fault characteristic parameters of the cluster, and to solidify the topology and connectivity view. This module is responsible for establishing a unified fault detection signal source and calculating the probability of node failure and link failure.

[0013] The redundancy strategy configuration module is used to establish a set of optional backup modes for each function, and output the resource consumption parameters and recovery time parameters of each function in different modes, providing strategy input for the optimization model.

[0014] The availability analysis module is used to determine the recoverability of functions under different failure scenarios and resource configurations, and to quantify the system-level probability of recovery failure. This module assesses the reliability risk of the current deployment scheme by calculating the joint probability of failure scenarios and traversing failure combinations.

[0015] The recovery latency modeling module is used to calculate the expected recovery latency of the function under different primary and backup strategy combinations based on recovery time parameters, backup priority, and fault characteristic parameters, and to verify whether the indicator meets the recovery latency threshold defined by the business.

[0016] The optimized deployment solution module is used to construct and solve mixed-integer linear programming models. This module aims to minimize the sum of node activation cost and expected resource load, and outputs a ternary mapping result containing the function ID, deployment node ID, and backup strategy ID, while satisfying latency and reliability constraints.

[0017] The cloud-native adaptation and deployment module is used to convert the three-element mapping result into cluster node affinity configuration, resource quota configuration and routing rules, and automatically distribute and execute them to realize the deployment plan.

[0018] The user visualization module is used to display resource allocation views and recovery time guarantee compliance views, providing an auditable view of the system.

[0019] This invention incorporates network quality and recovery latency into the redundancy decision-making system by introducing link failure probability and functional recovery time indicators; it achieves joint optimization of static resource cost and dynamic fault risk cost by constructing a mixed integer linear programming model; and it realizes low-cost, highly reliable and latency-guaranteed automated redundancy deployment by mapping the optimization results to cloud-native configuration.

[0020] In summary, the method provided in this disclosure can accurately quantify the balance between resource costs and availability assurance, effectively avoiding the risk of primary / backup failure due to network connectivity degradation. This method, through differentiated combination optimization of cold and hot backup modes, reduces the static holding cost of the system and the dynamic recovery load, while strictly fulfilling the business-level recovery time guarantee (GRT) and system-level reliability commitments. It also achieves an automated closed loop from mathematical optimization decision-making to cloud-native scheduling execution, improving the survivability and operational efficiency of distributed systems in complex fault scenarios.

[0021] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which: Figure 1 A flowchart illustrating a low-cost redundant deployment method supporting availability assurance provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram illustrating the data acquisition modeling and differential redundancy strategy definition provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram illustrating the joint calculation principle of system-level availability and functional-level recovery latency (GRT) provided in an embodiment of this disclosure. Figure 4This diagram illustrates the construction and solution of a mixed-integer linear programming (MILP) optimization model provided in an embodiment of this disclosure. Figure 5 A schematic diagram illustrating the logical principle and solution process of the deployment solver module provided in this embodiment of the disclosure; Figure 6 A schematic diagram of cloud-native platform mapping for the deployment strategy provided in the embodiments of this disclosure; Figure 7 This is a schematic diagram of the automated closed-loop scheduling and adaptive feedback mechanism provided in the embodiments of this disclosure; Figure 8 This is a diagram illustrating a low-cost, redundant deployment system architecture that supports availability assurance, provided as an embodiment of this disclosure.

[0023] The module includes: 10. Data acquisition module; 20. Redundancy strategy configuration module; 30. Availability analysis module; 40. Recovery latency modeling module; 50. Optimization deployment solution module; 60. Cloud-native adaptation and deployment module; and 70. User visualization module. Detailed Implementation

[0024] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0025] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. The acquisition, transmission, storage, use, and processing of data in the technical solution of this disclosure all comply with the relevant provisions of national laws and regulations.

[0026] It should be noted that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solutions disclosed herein. However, they do not mean that the applicant has used or necessarily used such solutions.

[0027] See attached document Figure 1 First, a digital environment model required for optimization is constructed. An initialization process is then performed to establish a node set. With function set The mapping relationship, and for each node Enter resource capacity Resource capacity It is a multi-dimensional vector containing the number of CPU cores, memory size, storage space, and network bandwidth limit. By reading the metadata of the cluster infrastructure, the current topology and connectivity view is solidified. The topology and connectivity view includes the physical role attributes of nodes, their data center location, rack number, and availability zone affiliation information. Topology information is used in subsequent constraints to ensure that the primary and backup nodes meet geographical isolation requirements at the physical level.

[0028] Quantitatively model the reliability of infrastructure, define and obtain the probability of node failure. . Represents a node The probability value of a failure causing service unavailability within the statistics window. The values ​​are not derived from a single assumption, but are calculated based on the node's historical operating logs, hardware health reports (such as SMART information and ECC error counts), and the operating system kernel's stability records. Different values ​​are assigned to nodes with different hardware specifications or service lives. This value allows for the differentiation between high-reliability nodes and ordinary nodes at the model level. Node failure probability. As a basic input parameter, it directly participates in the subsequent joint probability of failure modes. The calculation.

[0029] Establish a link status awareness mechanism, and define and measure the link failure probability. .parameter Used for quantizing master nodes With backup nodes The probability of communication path interruption or quality degradation is assessed by performing mesh point-to-point connectivity detection using distributed network probes deployed on each node of the cluster. The detection metrics cover the packet loss rate of the Internet Control Message Protocol (ICMP), the establishment latency of TCP connections at the transport layer, and the handshake success rate at the application layer. A weighted normalization method is used, where the monitored network packet loss rate is weighted and summed with the normalized latency value, thus mapping network jitter and packet loss characteristics to probability values ​​within the interval [0, 1]. When the probe data shows the node With nodes When there is a high risk of congestion or frequent disconnections in the network paths between them. The value increases. This mechanism ensures that the system calculates the expected recovery delay. At the same time, invalid backup schemes that are healthy on their own but whose transmission paths are unreachable are eliminated, thereby supporting joint modeling of link reachability and node status at the underlying data level.

[0030] Simultaneously, responsible for inputting business constraint parameters and synchronizing with the time base. Obtaining information for each function. Business recovery latency threshold Service recovery latency threshold The maximum allowable downtime for services is defined. To support the auditability of Recovery Time Guarantee (GRT), a unified fault detection signal source and timestamp synchronization mechanism are established. Health check probes and link status monitoring signals from the access infrastructure use the fault detection moment as the timing start point, and calibrate the system clocks of all nodes in the cluster using Network Time Protocol (NTP) or Precision Time Protocol (PTP). Clock synchronization ensures a unified and accurate time observation calibrator when subsequently comparing actual recovery time with model predictions.

[0031] See attached document Figure 2 Then, a set of optional backup modes is established for each function, and the resource consumption parameters and recovery time parameters corresponding to each backup mode are defined. Construct a set of optional backup modes .gather It is not a single, fixed configuration, but rather includes multiple differentiated redundancy strategy options, mainly divided into hot backup mode and cold backup mode. The resource consumption and recovery time of each mode are characterized by quantitative parameters, providing a basis for subsequent optimization solutions.

[0032] For critical business scenarios with stringent service continuity requirements, such as core financial transaction systems, industrial automation control, and signaling plane functions of carrier networks, in the aggregated... This document defines a hot backup mode. In hot backup mode, the backup node is configured to run functional instances concurrently with the primary node. Hot backup mode requires a real-time state synchronization mechanism between the primary and backup nodes, maintaining functional consistency through data mirroring, shared memory access, or state log replay. To accurately reflect the resource cost of this mode, backup resource usage parameters are set for hot backup mode. The resource usage parameter is approximately equal to that of the master node. (Right now While hot backup mode configuration results in higher resource overhead, it ensures that backup nodes are always ready. The recovery time parameter corresponds to the hot backup mode. In this respect, because the process of instance startup and state loading is eliminated, The value is set to the minimum allowed by the system, typically containing only the very short time required for fault detection confirmation and traffic switching.

[0033] For low-to-medium priority functions with a certain tolerance for recovery time, such as offline log analysis, scheduled data aggregation, or auxiliary monitoring components, in the collection The system defines a cold backup mode. In cold backup mode, the backup node does not actively run the processes of the functional instances, but only retains the necessary container images, configuration files, and basic runtime environment dependencies in local storage or a remote repository. When the primary node fails and is confirmed to have failed, the system schedules the backup node to perform a restart operation, including image loading, container creation, network configuration, and application initialization. Correspondingly, backup resource usage parameters are set for cold backup mode. This is a relatively small value, reflecting only storage or minimal maintenance overhead, thus reducing resource consumption in steady state. However, due to the involvement of the complete startup process, the recovery time parameter corresponding to this mode... The efficiency is relatively high. When modeling the system, it is necessary to clearly define the image distribution strategy and the execution efficiency of the preheating script in order to accurately estimate... The specific values ​​should be set to avoid situations where resources are saved but basic availability requirements cannot be met.

[0034] To provide a unified representation of hot backup and cold backup modes in the mathematical model, a primary / standby deployment variable is introduced. . It is a binary decision variable used to represent function. Whether to use a specific deployment combination: i.e., the primary instance is deployed on the node Backup instances are deployed on nodes And backup nodes Adopting strategies ( ). Through variables The system couples the selection of physical nodes with the selection of backup modes.

[0035] In the subsequent optimization process, based on The value of , combined with the corresponding mode and The parameters aim to find a balance between resource cost and recovery latency. For example, for latency-sensitive functions listed in the strong GRT list, the model will tend to select the hot standby mode. Value, to satisfy The hard constraints apply; for other functions, the combination of cold standby modes with lower resource costs is prioritized, provided that the latency threshold is met.

[0036] See attached document Figure 4 A fault configuration space containing multiple potential failure scenarios is established. During system deployment or operation, in-depth scanning and probability calculations for multiple failure scenarios are performed. This is used to exhaustively enumerate or sample all possible combinations of node failure states in the network. (In the fault configuration space) In, each independent fault configuration (in () represents a specific network snapshot, which clearly defines which nodes are in a failed state at the current moment, while the set The nodes in the system are in normal working order.

[0037] Based on the node failure probability provided during the data acquisition phase Configure each fault The joint probability of occurrence is quantified. Since node hardware failures typically exhibit statistical independence, the joint probability... It is defined as the combination of the product of the failure probabilities of each node in the set of failed nodes and the product of the survival probabilities of each node in the set of normal nodes. The calculation formula is defined as follows: ; The above formula provides a weighted risk assessment benchmark for the system, which makes high-probability failure scenarios (such as single node failure) have a greater weight in subsequent optimization decisions, while the impact of extremely low-probability catastrophic scenarios (such as network-wide paralysis) on cost optimization is reasonably converged.

[0038] Introducing binary deployment variables and This maps the relationship between functionality and physical resources. Indication Function Deployed as the primary instance on the node superior, Indication Function Deployed as a backup instance on the node Above. For any given fault configuration traverse all functions This determines whether the system has lost its service capability under this configuration. The decision logic is based on the redundancy coverage principle: it determines whether the system loses its service capability if and only if the function it carries... The nodes of the primary instance and all nodes of the backup instances that support this function belong to the failure set. If this occurs, the function is deemed unrecoverable.

[0039] Define binary state variables When configuring Next function When all deployed nodes (including the primary node and all backup nodes) fail, A value of 1 indicates the function. In this scenario, an irreversible service interruption occurs; otherwise... A value of 0 indicates that at least one live replica (primary or backup) is available for use. Based on this judgment logic, the overall recovery failure probability of the system under the full configuration space is calculated. The calculation formula is as follows: ; Symbols in the formula This represents a logical OR operation, i.e., in fault configuration. Under any one of the following conditions, If recovery is not possible, the scenario is marked as a system-level failure scenario. By summing up the occurrence probabilities of all system-level failure scenarios This yields a macroscopic risk value for system unavailability.

[0040] To ensure the auditability of the project implementation, a standardized recoverability log is generated during the above calculations. This log records the fault configuration fingerprint (corresponding to...) for each window determined to have failed. This includes a detailed list of nodes, the binding relationship between functions and nodes, and specific failure reason codes. (Indicators) When it is used as a core constraint input into the subsequent optimization process, Exceeding the preset security threshold When this happens, the system will reject the current deployment plan and trigger a re-solution, thereby ensuring that the final generated deployment strategy meets the system-level high availability requirements at the probabilistic level.

[0041] See attached document Figure 5 For each individual function Establish functional-level recovery time indicators .index It doesn't simply reflect whether the service is online, but rather quantifies the end-to-end response latency required for the system to detect the fault, decide on a switchover, and complete the backup instance takeover at the moment the primary node fails. For latency-sensitive applications such as financial risk control and industrial real-time control, This determines the boundaries of business losses caused by the failure.

[0042] Based on fundamental probability parameters and policy latency parameters, a mathematical expectation model for recovery latency is constructed. This model not only considers the failover time of a single node but also integrates the execution logic of multi-level backup strategies and the reliability of network paths. Specifically, a priority parameter is introduced. This defines the order of backup node k in the recovery sequence of function j. When the primary node fails, the system recovers the node according to priority. Recovery attempts are made sequentially from highest to lowest priority backup nodes. Recovery requests are only passed to lower priority backup nodes if a high-priority backup node fails or becomes unreachable.

[0043] To accurately calculate the expected delay of this dynamic process, the following weighted expected delay formula for recovery is constructed: ; In this formula, Representative adopts strategy The baseline recovery time during hot or cold standby is composed of image loading time, state synchronization latency, and service startup time. These are the deployment decision variables, which determine the specific primary / backup relationship and strategy selection. Indicates the master node The probability of a failure triggering a recovery process. The item embodies the key constraints of link reachability; it represents the reachability of the link at the master node. The probability that the link between the primary and backup nodes remains connected when a failure occurs. If If the value is high (i.e., the links are extremely prone to simultaneous interruption), the effectiveness weight of the path will be reduced, thus penalizing deployment schemes that are physically isolated but have highly coupled network paths at the model level.

[0044] Multiplication terms in the formula It describes a cascading failure scenario, where all nodes with higher priority than the current node fail. Backup nodes The joint probability of both failures. This indicates the backup node that is currently in turn. The probability of itself being in an available state. Using this formula, the recovery latency modeling module 40 iterates through all possible recovery paths, calculating the physical latency of each path. Multiply the conditional probability of its effectiveness by the product and sum them to obtain the function. Expected recovery time in a statistical sense

[0045] Calculated Recovery latency threshold defined by business logic A comparison is performed to establish hard constraints. For critical services listed in the GRT directory, if a candidate deployment scheme calculates... Exceeded the threshold If the answer is no, then the plan is deemed infeasible.

[0046] See attached document Figure 6 Based on the mathematical framework of Mixed Integer Linear Programming (MILP), an objective function is established to minimize the overall resource cost of the system. The objective function employs a bi-objective weighted optimization strategy, incorporating the static fixed costs of the system deployment phase and the dynamic expected load costs generated during fault recovery in the operational phase into a unified computational model. By solving this objective function, the optimal balance between the number of activated nodes and the reservation of resource redundancy is found, while ensuring system resilience.

[0047] The objective function for minimizing the total cost of the system is defined as follows: ; The first term in the formula This represents the static deployment cost of the system. Among them, It is a binary decision variable used to indicate physical nodes. Whether it has been activated and put into use. When the node... When any function is deployed on the primary or backup instance, The value is 1 if the condition is met, and 0 otherwise. This item reflects the cluster's infrastructure overhead, including server hardware purchase, power-on energy consumption, basic cooling, and fixed expenses for daily operation and maintenance. During optimization, the solver tends to deploy the primary and backup instances of the functionality on fewer physical nodes, thereby shutting down or hibernating redundant nodes to reduce the system's basic operating costs.

[0048] The second term in the formula This represents the expected cost of dynamic resource load under fault scenarios. This item is calculated by traversing the fault configuration space. All possible scenarios Calculate the resource cost of the recovery operation. Parameters It represents the probability of a failure scenario occurring. Indicates a specific failure scenario Next, node In order to fulfill the function The instantaneous resource load value generated by recovery tasks (such as starting containers, loading images, or handling burst traffic). The calculation is based on function The weighted sum of CPU and memory requests required during the recovery phase. The range of the summation. It limits the calculation to only those in the fault scenario. The nodes that are still alive and participating in the recovery.

[0049] The construction logic of this objective function lies in establishing a risk cost model due to the uncertainty of failures. For failure scenarios with a high probability of occurrence (i.e., In larger scenarios, such as single-node failures, the model assigns a high cost weight, forcing the solver to reserve sufficient and low-cost recovery paths for these scenarios; while for extremely low-probability catastrophic scenarios (i.e., In extremely rare scenarios, the model allows the system to bear higher temporary load costs, thus avoiding the long-term idle of large amounts of redundant resources to cope with rare disasters. By adding the static activation cost to the probability-weighted dynamic recovery cost, a deployment scheme that is both economical and flexible can be generated, avoiding resource waste caused by excessive redundancy or recovery blockage caused by insufficient redundancy.

[0050] When performing mixed-integer linear programming, a strict system of linear constraints was constructed to define the feasible region of the solution space. This constraint system not only includes basic logical constraints but also transforms the recovery time model and availability probability model calculated by the aforementioned modules into mathematical inequalities, ensuring that the generated deployment plan is physically executable, meets business SLAs in terms of time, and complies with high availability standards in terms of probability.

[0051] First, logical constraints are established between node activation status and functional deployment variables. To accurately calculate the static cost term in the objective function, the optimization deployment solution module 50 must identify whether each node is active. The optimization deployment solution module 50 introduces node activation constraints, defining the node activation variables. The value depends on whether the node hosts either a primary instance or a backup instance that performs any functionality. The constraint formula is as follows: ; The above constraint formula shows that as long as any function exists... This makes its main deployment variable True (i.e., deployed on the node) ) or backup deployment variables True (i.e., backup is on the node) ), then the node Activation status It must be set to 1; otherwise, if the node If no instances are hosted, then The value is 0. Node activation constraints ensure the accuracy of cost calculation and prevent logical errors such as nodes running idle but not being included in costs or nodes carrying services but not being marked as activated.

[0052] Establish a unique constraint on the master node to ensure that each function... The system must have one and only one running master instance. This is the topological foundation for the normal operation of distributed services, avoiding data inconsistencies caused by multiple master instances or service unavailability due to the absence of a master instance. The constraint formula is as follows: ; The above constraint formula applies to each function In all optional nodes The main deployment variable on The summation is performed, forcing it to equal 1. Mathematically, this eliminates the possibility of multiple deployments or no deployments, ensuring the integrity of the business logic.

[0053] Establish exclusive constraints on backup strategies and standardize resource configuration patterns on backup nodes. For the same function... and the same backup node The system does not allow the simultaneous use of multiple backup strategies (e.g., configuring both cold and hot backups), nor does it allow for different backup strategies for different master nodes. On the same backup node Establish multiple backup relationships. The constraint formula is as follows: ; The above constraint formula traversal function All optional backup modes and all possible master nodes (exclude (Itself), restricting it in nodes Deployment variables on The sum does not exceed 1. Exclusivity constraints ensure that the allocation of backup resources is deterministic and unique, avoiding resource contention and scheduling logic conflicts.

[0054] The calculation results are transformed into hard time constraints, namely, Recovery Time Guarantee (GRT) constraints. This invention sets latency as an insurmountable boundary condition, requiring each function j to meet the expected recovery time under fault scenarios. It must not exceed the business's preset threshold. The constraint formula is as follows: ; The above constraint formula directly affects the solution space selection process; any constraint that leads to a decrease in recovery time... Deployment combinations that exceed the limits (such as selecting backup nodes with excessively long network paths or excessively long cold start times) will be considered infeasible solutions and eliminated. Time constraints ensure that the final output solution strictly meets the commitments made to latency-sensitive services.

[0055] Establish system-level availability probability constraints to control the risk boundary of overall service interruption, comprehensively considering the probability of unrecoverable failure due to the complete failure of all nodes. Risk probability from other potential failure factors (Such as failures caused by resource capacity exceeding limits, etc.), the sum of these failures must not exceed the maximum allowed failure probability of the system. The constraint formula is as follows: ; The above constraint formula limits the unrecoverability of the system at the macroscopic level. It quantifies the catastrophic risks at the physical level; Usually, the minimum value is taken (e.g.) or This corresponds to the industry-standard five-nine or six-nine availability criteria. System-level availability probability constraints force the optimizer to reduce costs by increasing the number of backup nodes or selecting more reliable nodes. The value of this value sets a bottom line between cost and reliability.

[0056] It integrates a mixed-integer programming engine, which serves as the core of the computation, responsible for processing the mathematical model constructed by the aforementioned modules and receiving a set of nodes. Functional Set Failure probability and Resource capacity and recovery time parameters The entire parameter set, including the objective function and constraint matrices, is used as input data. The solution engine performs search and iterative calculations in the multidimensional solution space, aiming to find a set of decision variable values ​​that minimize the objective function while satisfying all the aforementioned constraints. Decision variables include node activation state variables. Primary deployment variables Backup deployment variables and detailed primary and backup strategy combination variables .

[0057] After obtaining the optimal mathematical solution, the output data undergoes engineering transformation and standardized encapsulation. This is because the original mathematical variables... It is a sparse, high-dimensional matrix form, which is not convenient for direct use in subsequent system configuration. Therefore, it is parsed and reconstructed into a standardized ternary mapping structure. The ternary mapping structure is defined as [Function ID, Deployment Node ID, Backup Strategy ID]. Specifically, for each decision variable with a value of 1... Generate a corresponding deployment instruction record: where the function ID corresponds to the variable index. This refers to a specific microservice or network element task; the deployment node ID includes the master node. With backup nodes The physical identifier information clarifies the topological location of the primary / standby relationship; the backup policy ID corresponds to the variable index. This refers to the specific mode (such as hot backup or cold backup) used in the backup relationship and its associated resource configuration parameters.

[0058] The ternary mapping results are incorporated into a version control system, assigning a unique version number and timestamp to each deployment scheme generated by the solution. Version information not only links to the specific deployment location but also includes a snapshot of system parameters at the time of the solution (such as the node failure probability at that time). Link status ) and model-predicted performance metrics (such as calculated function recovery latency) With the probability of system unrecoverability This is all solidified together. The version control system establishes a consistency index for parameters, strategies, and observations, enabling the system to accurately align real-time monitoring data with theoretical predictions at the time of deployment during subsequent operation. For example, when the system subsequently observes the actual time taken for a recovery operation, it can trace back to the recovery time threshold set when the solution was initially deployed using the version number. Compared with the forecast This provides an immutable benchmark for compliance audits of Recovery Time Guarantee (GRT).

[0059] Simultaneously, it is also responsible for generating standardized output of the unrecoverable inventory log. During the solution process, if it is found that certain functions cannot find feasible solutions that meet the conditions under specific failure scenario constraints (e.g., when all delays are satisfied)... (Insufficient resources on the path), or the overall risk probability of the system. Unable to converge to The solver engine will then mark the tight constraints (BindinqConstraints) that lead to infeasible solutions. Infeasibility information will be formatted as log entries containing the function ID, failure cause code (e.g., path unreachable, latency exceeding limits, capacity saturation), and the corresponding failure scenario fingerprint. These log entries will not be issued as deployment commands but will instead be fed back to the operations and maintenance management layer as alarm data, indicating the need to expand resources or adjust SLA thresholds, thus forming an effective information transmission from the mathematical model to operational decisions.

[0060] See attached document Figure 7 First, the deployment node ID is mapped to the Pod Affinity rules and Pod Tolerations configurations of the Kubernetes cluster. To ensure that the calculated fault isolation and link reachability are strictly effective at the physical layer, the affinity rules are configured as hard constraints.

[0061] Secondly, map the backup policy ID to ServiceMesh routing rules or resource limit configurations. For the primary instance Pod, set its resource requests and limits to... The corresponding complete business requirement value. For backup instance Pods, perform differentiated configuration based on the backup policy ID: if the policy is hot backup mode, set the resource quota of the backup Pod to be similar to that of the primary instance. And start the state synchronization process; if the strategy is cold backup mode, then set the resource quota of the backup Pod to the low consumption value required to maintain the minimum liveness state. (For example, only retaining the basic Sidecar agent or a placeholder process with very low weight), or simply issuing Deployment configurations and setting the replica count to 0 (to be scaled up again when a failure is triggered). This mapping mechanism concretizes the low-cost redundancy goal in the mathematical model into a cluster-level CPU and memory quota strategy, thereby improving resource utilization.

[0062] Secondly, leverage the traffic management capabilities of ServiceMesh (such as Istio or Linkerd) to translate backup path policies into routing rules. For each function... Define a VirtualService object based on the priority determined by the optimization model. Build a route list. Under normal operating conditions, the route weight is configured to point to the master node instance 100%, and the outlier detection policy in DestinationRule is configured to reduce the probability of link failure. Higher path associations result in stricter health check thresholds. When the Sidecar agent in the mesh plane detects a primary path interruption or service unreachability, traffic is automatically redirected to the highest-priority available backup subset based on a pre-configured failover policy. In this way, the logical backup relationships in the triple are solidified into mesh-level traffic forwarding rules, ensuring the executability and determinism of the service recovery path.

[0063] A custom controller architecture based on the Kubernetes Operator pattern is adopted to achieve automated synchronization from logical deployment plans to physical cluster states. A dedicated custom resource definition (CRD) is defined, which is a redundant deployment planning resource object. The controller continuously monitors for changes to the CRD object. Once an update to the desired state is detected, the controller initiates the resolution logic, converting the [Function ID, Deployment Node ID, Backup Policy ID] in the ternary mapping into specific resource manifests within the Kubernetes cluster.

[0064] For each function The controller generates or updates two sets of core resource objects: one set of workload objects (such as Deployment or StatefulSet) to host the primary instance, and another set of workload objects to host backup instances. The controller builds a configuration tree of these objects in memory, injecting the node affinity rules, resource quota limits, and tolerance configurations determined in the preceding steps, ensuring that the generated YAML description file strictly corresponds to the constraints in the mathematical model.

[0065] The controller performs atomic policy update operations to ensure service continuity when adjusting the deployment structure. For the migration or redeployment of primary nodes, the controller employs a rolling update strategy, configuring a maximum number of unavailable replicas to 0. It ensures that instances on the new node are fully ready and confirmed healthy by probes before gradually reducing the number of instances on the old node. For configuration changes to backup nodes, the controller uses a parallel creation mechanism, prioritizing creation on new backup nodes. The pull-up function is configured with the corresponding strategy. Pod instances (such as those with cold or hot standby resource quotas). The controller only cleans up old backup resources after the new backup path is network-connected and resource reservation is successful. This scaling-up-down mechanism prevents the risk of redundancy loss during policy switching windows.

[0066] Simultaneously, it is also responsible for synchronizing traffic rule objects at the ServiceMesh layer. Based on the backup relationships in the triplet, the controller generates Istio's VirtualService and DestinationRule objects. For cold backup mode, although the controller distributes the image and configuration to the physical nodes, the weight pointing to the backup subset is initially set to 0 in the traffic rules, placing it in a plug-and-play standby state. For hot backup mode, if the service supports dual-active operation, the controller can configure specific weight ratios. After all configurations are distributed, the controller compares the actual cluster running status (Status) with the expected status (Spec). When the two are completely consistent, it marks the CRD object as synchronized (Synced), completing the closed loop of this automated distribution.

[0067] Furthermore, this method also includes collaborative work during the system operation phase to establish a closed-loop feedback mechanism from state awareness and deviation analysis to strategy correction. This mechanism aims to address the parameter drift problem that occurs in static optimization models in dynamically changing physical environments, ensuring that the recovery time guarantee (GRT) and availability commitment calculated in the aforementioned steps remain effective in actual operation.

[0068] Integrate or interface with cluster-level metric collection components (such as Prometheus or OpenTelemetry probes) to perform high-frequency sampling of key performance indicators for system operation. The collection targets include the liveness status of physical nodes, the resource utilization of container instances (CPU / MemoryUsage), and the round-trip time (RTT) and packet loss rate of inter-node networks. Specifically, the system injects tracking markers into the ServiceMesh data plane to monitor the link quality of cross-node communication in real time, dynamically mapping the monitored network jitter data to updated link failure probability parameters. Simultaneously, monitor the hardware health counters of each node, and dynamically increase the failure probability parameter of the corresponding node when an increase in disk read / write error rate or frequent memory ECC error correction is detected. .

[0069] The system establishes real-time compliance verification logic and performs online simulations. The system periodically utilizes updated parameters. and For the currently effective deployment plan (i.e., the current decision variables) (Status) Recalculate the expected recovery latency for each function With the probability of system unrecoverability The verification logic will recalculate the result. Hard thresholds set by the business Perform comparison (validate constraints) ), and will With the system's maximum allowable failure probability A comparison is performed. If the calculation results show that the expected recovery time of a critical function in the current network environment has exceeded the threshold, or the overall system risk has exceeded the security threshold, the system determines that the current deployment plan has defaulted on the SLA or the model has drifted.

[0070] Once a model drift or SLA default risk is detected, the system triggers an adaptive replanning process. It then updates the latest environmental parameters (…). It is injected into the solver engine as input data.

[0071] While maintaining the stability of unaffected services, incremental optimization or global replanning is initiated to find new optimal deployment triples that satisfy all constraints (especially the aforementioned time and probability constraints). The new optimization results are then automatically executed by the controller through the aforementioned CRD update mechanism to perform Pod migration or backup strategy adjustments (e.g., upgrading cold backups on unstable links to local hot backups, or migrating them to nodes with better network quality), thereby automatically repairing system availability bottlenecks without human intervention.

[0072] By transforming underlying mathematical indicators into a visual management dashboard based on system operational status, an auditable view is provided, displaying resource allocation, GRT compliance, and fault simulation heatmaps. Actual fault recovery times are compared with model predictions to calculate the prediction deviation rate. Historical deviation data is fed back into a closed-loop feedback mechanism for strategy correction, used to adjust the baseline weights of parameter estimations in subsequent rounds, thereby improving the accuracy of the mathematical model's fit to the physical environment.

[0073] See attached document Figure 8This invention provides a low-cost redundant deployment device that supports availability assurance, including a data acquisition module 10, a redundancy policy configuration module 20, an availability analysis module 30, a recovery latency modeling module 40, an optimization deployment solution module 50, a cloud-native adaptation and deployment module 60, and a user visualization module 70. The data acquisition module 10, redundancy policy configuration module 20, availability analysis module 30, recovery latency modeling module 40, optimization deployment solution module 50, cloud-native adaptation and deployment module 60, and user visualization module 70 are connected via a data bus or API interface.

[0074] The data acquisition module 10 is used to acquire basic operational data and fault characteristic parameters of the cluster. The information acquired by the data acquisition module 10 includes the cluster topology and node set. Resource capacity of each node (Including computing, storage, and bandwidth capacity) and the current resource usage of nodes. The data acquisition module 10 solidifies the topology and connectivity view, determining node roles, data center or availability zone distribution, and link tables. Based on fault characteristics, the data acquisition module 10 collects and defines the node failure probability. and link failure probability ,in This is used to reflect the risk of connectivity degradation between nodes. The data acquisition module 10 obtains the fault detection signal source and timestamp through probes or health check mechanisms to establish the scope of recovery observation.

[0075] Redundancy policy configuration module 20 is used for each function Establish a set of optional backup modes The redundancy policy configuration module 20 configures backup modes including hot backup mode and cold backup mode. For hot backup mode, the redundancy policy configuration module 20 sets higher resource consumption parameters. (Approaching master node resource usage) and minimum recovery time parameter This hot backup mode requires both the primary and backup nodes to run functional instances simultaneously and maintain consistency through a state synchronization mechanism. For cold backup mode, the redundancy policy configuration module 20 sets lower resource consumption parameters. and larger recovery time parameters In this cold backup mode, the backup node only retains the necessary environment configuration or image. The redundancy policy configuration module 20 outputs the resource usage and recovery time parameters corresponding to each function to the availability analysis module 30 and the recovery latency modeling module 40.

[0076] The availability analysis module 30 is used to determine the recoverability of functions under different failure scenarios and resource configurations, and to quantify the system-level recovery failure probability. The availability analysis module 30 defines the fault configuration space. Each configuration This represents a set of failed nodes. The availability analysis module 30 calculates the failure modes based on the following formula. Joint probability of occurrence : ; Availability Analysis Module 30 defines binary variables Indicates fault configuration Next function Whether it has completely failed, and calculate the overall probability of system recovery failure based on this. .

[0077] ; The availability analysis module 30 outputs an unrecoverable list log, recording the function ID, primary and backup node information, and the cause of failure.

[0078] The recovery latency modeling module 40 is used to calculate the expected recovery latency of the function under different combinations of primary, backup, and strategy settings, and to verify whether it meets the recovery latency threshold defined by the business. The recovery latency modeling module 40 introduces a functional-level recovery time metric. This is used to measure the response time when the system switches to an available backup instance after a node failure. The recovery latency modeling module 40 is based on decision variables. (Indicates function) Main deployment in Backup deployed at And adopt strategies and backup priority The weighted expected value of the recovery delay is calculated using the following formula: ; The recovery delay modeling module 40 inputs the calculation results as constraints into the optimization deployment solution module 50.

[0079] The optimization deployment solution module 50 is used to construct and solve a mixed-integer linear programming model to determine the optimal deployment scheme. The optimization deployment solution module 50 sets the system's objective function to minimize the sum of node activation cost and expected resource load under failure scenarios. ; in, Indicates whether a node is active. Indicating a fault scenario Next node The resource load generated when undertaking the recovery tasks of functional combinations. Optimize the deployment of the solver module 50 to meet node capacity and functional recovery time targets. The solution is obtained under the constraint of the sum of recovery failure probabilities, and the output is a ternary mapping result containing function ID, deployment node ID and backup strategy ID.

[0080] The cloud-native adaptation and deployment module 60 converts the ternary mapping results output by the optimization deployment solution module 50 into specific scheduling policies. The cloud-native adaptation and deployment module 60 maps deployment node IDs to node affinity configurations and Pod tolerance configurations for the Kubernetes cluster, and maps backup policy IDs to service mesh routing rules or resource limit configurations. The cloud-native adaptation and deployment module 60 automatically distributes the configurations to the cluster for execution through the controller or operator.

[0081] User visualization module 70 provides an auditable view of the system. It displays a resource allocation view, a GRT compliance view, and a fault simulation heatmap, and records detailed data on the recoverable time and the probability of unsuccessful recovery.

Claims

1. A low-cost redundant deployment method supporting availability assurance, characterized in that, Includes the following steps: Acquire basic operational data and fault characteristic parameters of the cluster, including node failure probability and link failure probability reflecting the risk of connectivity degradation between nodes; Establish a set of optional backup modes for each function, and define the resource consumption parameters and recovery time parameters corresponding to each backup mode; Establish a fault configuration space containing multiple potential fault scenarios, calculate the joint probability of each fault scenario based on the node failure probability, and determine the recoverability of the function and quantify the system-level recovery failure probability based on the redundancy coverage principle. Define a functional-level recovery time metric. Based on the recovery time parameter, backup priority, node failure probability, and link failure probability, calculate the expected recovery latency of the function under different primary / backup strategy combinations selected from the set of optional backup modes, and use the expected recovery latency as the quantitative value of the functional-level recovery time metric. A mixed-integer linear programming model is constructed and solved. The objective is to minimize the sum of the node activation cost and the expected resource load under the fault scenario calculated based on the resource occupancy parameters. The constraints are that the functional recovery time index is not higher than the preset recovery latency threshold and the recovery failure probability meets the preset security standard. The ternary mapping result containing the function ID, deployment node ID and backup strategy ID is determined. The ternary mapping result is converted into a scheduling policy, the deployment node ID is mapped to the node affinity configuration of the cluster, the backup policy ID is mapped to the routing rules or resource limit configuration of the service mesh, and the policy is automatically sent to the cluster for execution.

2. The low-cost redundant deployment method supporting availability assurance according to claim 1, characterized in that, The acquisition of basic operational data and fault characteristic parameters of the cluster includes: A fixed topology and connectivity view is used to determine node roles, data center distribution, and link tables; The probability of node failure is calculated based on the node's historical operation logs, hardware health status reports, and operating system kernel stability records. Mesh point-to-point connectivity detection is performed using distributed network probes. A weighted normalization method is used to sum the monitored network packet loss rate with the normalized latency value, and network jitter and packet loss characteristics are mapped to the link failure probability. Establish a unified fault detection signal source and timestamp synchronization mechanism to define the criteria for recovery observation.

3. The low-cost redundant deployment method supporting availability assurance according to claim 1, characterized in that, The set of optional backup modes includes hot backup mode and cold backup mode: The hot backup mode is used for backup nodes and primary nodes to run functional instances simultaneously and maintain consistency through a state synchronization mechanism, setting resource usage parameters close to those of the primary node and minimum recovery time parameters. The cold backup mode is used to keep only the necessary environment configuration or image on the backup node, and sets the resource usage parameters that only reflect storage or maintenance overhead and the recovery time parameters that include the startup process.

4. The low-cost redundant deployment method supporting availability assurance according to claim 1, characterized in that, The quantified system-level recovery failure probability includes: The fault configuration space is defined as the set of all possible combinations of node failure states, where each specific combination of failure states constitutes a fault scenario. Based on the node failure probability, the joint probability of a fault configuration occurring is defined as the combination of the product of the failure probabilities of each node in the set of failed nodes and the product of the survival probabilities of each node in the set of normal nodes. If, under a certain failure scenario, the node carrying the primary instance of the function and all the nodes carrying the backup instances of the function belong to the set of failed nodes, then the function is completely failed. The system-level recovery failure probability is obtained by summing the joint probabilities of all failure scenarios that would cause any one of the functions to fail completely.

5. The low-cost redundant deployment method supporting availability assurance according to claim 1, characterized in that, The expected recovery latency of the computing function under different primary / backup strategy combinations includes: A priority parameter is introduced to define the order of backup nodes in the function's recovery sequence; A weighted expectation model for recovery latency is constructed. The weighted expectation model for recovery latency traverses all recovery paths, multiplies the recovery time parameter corresponding to each path by its conditional probability of taking effect, and sums the results. The conditional probabilities include the probability of the primary node failing, the probability of the link between the primary and backup nodes remaining connected, and the combined probability that all other backup nodes with higher priority than the current backup node have failed while the current backup node itself is in an available state.

6. The low-cost redundant deployment method supporting availability assurance according to claim 1, characterized in that, The objective function of the constructed mixed-integer linear programming model consists of two parts: The first part is the static deployment cost of the system, which is calculated by summing the node activation variables of all activated nodes. The node activation variables indicate whether a physical node is activated and put into use. The second part is the expected cost of dynamic resource load under the fault scenario. By traversing all scenarios in the fault configuration space, the sum of the products of the joint probability of the fault configuration occurring and the instantaneous resource load value generated by the node undertaking the recovery task under the fault scenario is calculated, wherein the instantaneous resource load value is determined according to the resource occupancy parameter.

7. The low-cost redundant deployment method supporting availability assurance according to claim 6, characterized in that, The constraints of the mixed-integer linear programming model include: Node activation constraint: Specifies that the value of the node activation variable depends on whether the node carries a primary instance or a backup instance of any of the functions described. Master node uniqueness constraint: forces the sum of the master deployment variables for each function to be 1 across all available nodes; Backup strategy exclusivity constraint: For the same function and the same backup node, the sum of the set of optional backup modes and the deployment variables of all master nodes shall not exceed 1; Recovery time guarantee constraint: It is required that the expected recovery latency of each function under the fault scenario shall not exceed the preset recovery latency threshold; System-level availability probability constraint: The sum of the probability of recovery failure caused by the complete failure of nodes and the risk probabilities from other potential failure factors must not exceed the maximum failure probability allowed by the system.

8. The low-cost redundant deployment method supporting availability assurance according to claim 1, characterized in that, The specific structure of the ternary mapping result is as follows: Function ID: Refers to a specific microservice or network element task; Deployment Node ID: Contains the physical identification information of the primary and backup nodes, clearly defining the topological location of the primary-backup relationship; Backup Policy ID: Indicates the specific mode used in the backup relationship and its associated resource configuration parameters; The low-cost redundant deployment method that supports availability assurance further includes: incorporating the ternary mapping results into the version control system, assigning a unique version number and timestamp to each deployment scheme generated by the solution, and associating the system parameter snapshot and model prediction performance indicators at the time of solution.

9. A low-cost redundant deployment method supporting availability assurance according to claim 1, characterized in that, The step of converting the ternary mapping result into a scheduling policy includes: Map the deployment node ID to the Pod node affinity rule and configure it as a hard scheduling constraint; For cold backup mode, set the resource quota of the backup instance to the consumption value required to maintain the minimum liveness state or set the number of replicas to 0, and initially set the weight pointing to the backup subset to 0 in the traffic rules; For hot backup mode, the resource quota of the backup instance is set to a value similar to the business requirements of the primary instance, and the status synchronization process is started. A route list is constructed based on the priorities determined by the optimization model, and health check thresholds are associated with paths with a high probability of link failure.

10. A low-cost redundant deployment and scheduling device supporting availability assurance, characterized in that, The method for low-cost redundant deployment supporting availability assurance, applicable to any one of claims 1-9, comprises: The data acquisition module (10) is used to acquire the basic operating data and fault characteristic parameters of the cluster, and to solidify the topology and connectivity view; The redundancy strategy configuration module (20) is used to establish a set of optional backup modes for each function and output the resource consumption and recovery time parameters corresponding to each function. The availability analysis module (30) is used to determine the recoverability of functions under different failure scenarios and resource configurations, and to quantify the probability of system-level recovery failure. The recovery latency modeling module (40) calculates the expected recovery latency under different primary / backup strategy combinations based on the recovery time parameters, and verifies whether it meets the recovery latency threshold defined by the business. The optimization deployment solution module (50) is used to construct and solve a mixed integer linear programming model with the goal of minimizing the sum of cost and expected resource load calculated based on the resource occupancy parameters, and outputs a ternary mapping result containing function ID, deployment node ID and backup strategy ID; The cloud-native adaptation and deployment module (60) is used to convert the ternary mapping result into the cluster's node affinity configuration, resource quota configuration and routing rules, and automatically distribute and execute them; The user visualization module (70) is used to display the resource allocation view and the recovery time guarantee compliance view, providing an auditable view of the system.

Images