Distributed cluster software real-time updating system and method

Abstract

The invention discloses a distributed cluster software real-time updating system and method, and belongs to the technical field of computer software. According to the invention, accurate global gray control is realized, the flow control center ensures the consistency and accuracy of flow switching in a global range, and flow routing confusion caused by inconsistent node states is avoided; second-level global fault isolation and rollback are realized, once any link is abnormal, a flow control center has the capability of issuing a global command at the second level, and the flow of the whole cluster is instantly stripped from a problematic version, which is incomparable by manual or step-by-step rollback, so that the service continuity is greatly guaranteed; and monitoring-decision-execution forms an automatic closed loop, so that human intervention is reduced, the operation and maintenance efficiency and safety are improved, intelligent decision based on real-time data is realized, and the automation and intelligence degrees are improved.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and more specifically to a distributed cluster software real-time update system and method. Background Technology

[0002] With the widespread adoption of cloud computing and microservice architectures, large-scale applications are typically deployed in distributed clusters across hundreds or even thousands of server nodes to provide services. Software version iterations and updates are a common occurrence in operations and maintenance. Traditional update methods, such as rolling updates, blue-green deployments, or canary releases, while reducing service interruptions to some extent, still have the following drawbacks:

[0003] 1) Service interruption or performance degradation: During the process of restarting the instance, the instance will be temporarily unavailable, which may cause user requests to fail or increase latency.

[0004] 2) High demand for resource redundancy: For example, blue-green deployment requires a set of redundant resources that are exactly the same as the production environment, which is expensive.

[0005] 3) Slow rollback speed: When a problem is found in the new version, the traditional rollback operation requires redeploying the old version, which takes a long time and has a significant impact on business.

[0006] 4) Coarse update granularity: Updates are usually performed on the basis of the entire service instance, which cannot achieve finer-grained hot updates of code or functions.

[0007] Therefore, there is an urgent need for a software update technology that can achieve true real-time, smooth, and lossless updates to support continuous business delivery and 24 / 7 high availability requirements. To this end, a distributed cluster software real-time update system and method are proposed. Summary of the Invention

[0008] The technical problem to be solved by this invention is: how to achieve real-time, smooth, and lossless software updates. It provides a distributed cluster software real-time update system that enables automated canary releases and second-level rollbacks without restarting the service process and without the user's awareness.

[0009] The present invention solves the above-mentioned technical problems through the following technical solution, the present invention comprising a management control plane and a data plane;

[0010] The management control plane includes a version management module, a traffic control center, and a status monitoring and coordination module. The version management module is used to receive, verify, and store new version software packages and update policies. The traffic control center is used to receive the update status of node proxy modules and dynamically adjust global or local traffic routing rules according to policies and monitoring data, while triggering a global rollback in case of anomalies. The status monitoring and coordination module is used to collect the update status, health status, and performance indicators of all nodes in real time to obtain global health data, provide data basis for the traffic control center's decision-making, and can trigger rollback commands.

[0011] The data plane includes multiple nodes, each of which is equipped with a node proxy module, a traffic proxy module, and a business service module. The node proxy module is responsible for communicating with the management control plane, receiving and executing instructions. The traffic proxy module is responsible for receiving and executing traffic routing rules from the traffic control center. The business service module is responsible for running software services.

[0012] Furthermore, in the version management module, the update strategy includes the canary release ratio, target node group, and update conditions.

[0013] Furthermore, the node proxy module includes a class loader management unit and a traffic control unit. The class loader management unit is used to create an independent class loader for the new version of the software, so as to realize the isolation and coexistence of multiple versions of code in a single process. The traffic control unit is used to interact with the traffic control center and the local traffic proxy module, report the status, and perform traffic reception and drainage operations for the new and old versions of the software code.

[0014] This invention also provides a method for real-time updating of distributed cluster software, applied to the above-mentioned system, comprising the following steps:

[0015] S1: Upload and Strategy Formulation

[0016] Upload new version software packages and formulate update strategies through the version management module, and perform security scans and dependency verifications on the software packages;

[0017] S2: Distribution and Loading

[0018] According to the update strategy, the version management module distributes the new version software package to the node agent module of the target node group. The node agent module stores the software package in the local cache. The class loader management unit of the node agent module creates a new class loader for the new version software and loads the class files and related resources in it.

[0019] S3: Traffic Coordination and Canary Release

[0020] The node proxy module reports its readiness status to the traffic control center; the traffic control center issues initial canary traffic rules to the node proxy module according to the update strategy; the traffic control unit of the node proxy module notifies the local traffic proxy module to start directing new requests to the thread pool created by the new class loader; the existing thread pool that was processing the old version of the software code continues to process the received existing requests, and enters an idle state after processing is completed; the traffic control center makes decisions and repeatedly executes traffic ratio adjustments based on the global health data summarized by the status monitoring and coordination module, until all traffic is switched to the new version;

[0021] S4: Emptying and Cleaning

[0022] After all traffic is switched, wait for the old version to finish processing and emptying the existing requests, and then the traffic control center will instruct the node proxy module to unload the old version resources.

[0023] S5: Monitoring and Rollback

[0024] During gray-scale or global rollout, if the status monitoring and coordination module detects that the error rate or response delay of the new version request has doubled, it will immediately report the anomaly to the traffic control center. The traffic control center will trigger a rollback and issue emergency rollback rules to the traffic proxy modules across the entire network within seconds, switching all traffic back to the unupdated nodes or the old version threads, and notifying all updated node proxies to uninstall the new version and reload the old version software package.

[0025] Furthermore, in step S2, during the distribution and loading process, the old version of the software code is still running and providing services for the traffic.

[0026] Furthermore, in step S3, the global health data includes CPU utilization, memory utilization, request error rate, and request response time.

[0027] Furthermore, in step S3, the monitoring and rollback are triggered by the status monitoring and coordination module detecting anomalies in real time and immediately reporting them to the flow control center.

[0028] The present invention has the following advantages over the prior art:

[0029] 1. Seamless automated updates: The traffic control center enables fully automated canary releases and traffic ratio adjustments without manual intervention, resulting in extremely high efficiency.

[0030] 2. Global second-level intelligent rollback: Based on global monitoring data, a full network rollback is triggered instantly when an anomaly occurs, greatly shortening the fault recovery time and minimizing the impact on business.

[0031] 3. High efficiency and lossless resource utilization: It realizes the emptying of traffic from the old version, ensuring that user requests are not interrupted, and without the need for full redundant resources.

[0032] 4. Security and Reliability of Release: By forming an automated closed loop of "monitoring-decision-execution", intelligent decision-making based on real-time data is realized, which greatly improves the security and reliability of the release process. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating the structure and interaction of the distributed cluster software real-time update system in an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the overall process of the real-time update method for distributed cluster software in an embodiment of the present invention;

[0035] Figure 3(a) is a timing diagram of class loading and traffic switching within a single node under normal circumstances in an embodiment of the present invention;

[0036] Figure 3(b) is a timing diagram of class loading and traffic switching within a single node under abnormal conditions in an embodiment of the present invention. Detailed Implementation

[0037] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0038] Example 1

[0039] like Figure 1 As shown, this embodiment provides a technical solution: a distributed cluster software real-time update system, including a management control plane and a data plane.

[0040] The management control plane includes the management control plane, the flow control center, and the status monitoring and coordination module. The specific contents of each module are as follows:

[0041] 1. Version Management Module: This module receives, verifies, and stores new version software packages and update strategies. Update strategies include canary release ratios, target node groups, and update conditions.

[0042] 2. The traffic control center, as the brain of global traffic scheduling, receives the updated status of node proxy modules and dynamically adjusts global or local traffic routing rules according to policies and monitoring data to achieve precise canary releases and traffic scheduling, and triggers global rollback in case of anomalies.

[0043] 3. The status monitoring and coordination module collects real-time updates, health status, and performance metrics (such as CPU utilization, memory utilization, request error rate, and request response time) of all nodes. This provides data support for the traffic control center's decision-making and can trigger rollback commands.

[0044] The data plane consists of each node in a distributed cluster, and each node is deployed with:

[0045] 1. The node agent module is responsible for communicating with the management control plane, receiving and executing instructions. This module includes a class loader management unit and a traffic control unit. The class loader management unit is responsible for creating independent class loaders to load new versions of software, achieving isolation and coexistence of multiple versions of code within a single process; while the traffic control unit is responsible for interacting with the traffic control center and the local traffic agent, reporting status and performing local traffic draining operations.

[0046] 2. The traffic proxy module receives and executes traffic routing rules from the traffic control center.

[0047] 3. The business service module runs the software service process and can simultaneously support multiple versions of code instances.

[0048] like Figure 2 As shown, this embodiment also provides a method for real-time updating of distributed cluster software, applied to the above system, including the following steps:

[0049] Step 1: Uploading and Strategy Formulation

[0050] By uploading new version software packages and setting automated update strategies through the version management module, the system performs security scans and dependency checks on the software packages.

[0051] Step 2: Distribution and Loading

[0052] a) The version management module distributes the new version software package to the node agent modules of the target node group according to the update strategy. The node agent modules store the package in their local cache; b) The class loader management unit of the node agent module creates a new class loader for the new version software and loads the class files and related resources within it. At this time, the old version software code is still running and providing services for traffic.

[0053] Step 3: Traffic Coordination and Canary Release

[0054] a) The node proxy module reports its readiness status to the traffic control center; b) The traffic control center issues initial gray-scale traffic rules (e.g., 1%) to the node proxy module according to the update strategy; c) The traffic control unit of the node proxy module notifies the local traffic proxy to start directing new requests to the thread pool created by the new class loader; d) The existing thread pool that was processing the old version of the software code continues to process the received existing requests, and enters an idle state after processing is completed; e) The traffic control center automatically decides and cyclically executes traffic ratio adjustments (e.g., from 1% to 5%, 50%, 100%) based on the global health data summarized by the status monitoring and coordination module, until all traffic is switched (traffic switching refers to the switching of task request processing volume) to the new version.

[0055] Step 4: Emptying and Cleaning

[0056] After all traffic is switched over, once the old version has finished processing and emptying its existing requests, the traffic control center instructs the node proxy module to unload the old version's resources.

[0057] Step 5: Monitoring and Rollback

[0058] If, during a gray-scale or global rollout, the status monitoring and coordination module detects a surge in the new version's request response error rate or a sharp increase in request response latency (the change in request response error rate or request response latency is twice the original, for example, the request response error rate increases from 2% to 4%, or the request response latency increases from 10ms to 20ms), then it immediately reports the anomaly to the traffic control center. The traffic control center triggers an automatic rollback, and within seconds, it issues an emergency rollback rule to all traffic proxies across the network, switching all traffic back to nodes that have not been updated or to the old version threads (if they have not yet been cleaned up), and notifies all updated node proxies to uninstall the new version and reload the old version software package.

[0059] Example 2

[0060] This embodiment further illustrates Embodiment 1 with specific examples.

[0061] Suppose a Java microservice cluster with 100 nodes needs to update the user query interface service (user-service, upgrading from v1.0 to v2.0). The system of this invention will execute a fully automated update process that includes canary releases and automatic rollback capabilities.

[0062] System Deployment:

[0063] Management and control plane: Deployed on a separate server, it includes a version management module, a status monitoring and coordination module, and a traffic control center.

[0064] Data processing plane: 100 servers are deployed in a distributed cluster, with each node deploying: a business service module, a node proxy module, and a traffic proxy module.

[0065] Detailed steps for method execution:

[0066] Part 1: Uploading and Strategy Formulation

[0067] (a) Upload the new version user-service-v2.0.jar through the web interface of the version management module;

[0068] (b) Formulate update strategy: Set up a canary strategy, first release 1% of the traffic to 10 nodes; set up an automatic update strategy, if the request response error rate of version v2.0 is >1% or the P99 (the response time of the 99th request is the P99) delay is >200ms, then automatically trigger a rollback; set up a success condition, after version v2.0 runs stably under 1% traffic for 10 minutes, automatically increase the traffic to 5%, then 50%, 100%, and roll over to all nodes.

[0069] Part Two: Distribution and Loading

[0070] (a) The version management module randomly selects 10 nodes according to the policy and distributes the JAR package to the node agent module on them;

[0071] (b) After the target node proxy module receives the software update package, its class loader management unit immediately creates a new class loader instance (CL_v2.0) and loads the v2.0 JAR package. At this time, the v1.0 and v2.0 code coexist within the node.

[0072] Three: Traffic Coordination and Canary Release

[0073] (a) The flow control unit of the node agent module sends a "ready" signal to the flow control center;

[0074] (b) After receiving the ready signals from all 10 nodes, the traffic control center begins to execute global traffic scheduling. That is, the traffic control center simultaneously issues new routing rules to the traffic proxy modules on all nodes. The rules state: "1% of the query request traffic from external users will be identified by the HTTP header X-Version:v2.0 and routed to the node group marked with version=v2.0 (i.e., the first batch of 10 nodes)". At the same time, the traffic control center commands the traffic control units of these 10 nodes to start importing requests with X-Version:v2.0 into the v2.0 instance for processing.

[0075] 4. Emptying and Cleaning

[0076] (a) The status monitoring and coordination module continuously collects key indicators, including the request response error rate, request response latency, and overall performance of the entire cluster for the v2.0 instances of those 10 nodes.

[0077] (b) The status monitoring and coordination module compares the collected data with the preset strategy: Scenario 1 (Normal): If the indicators are normal within 10 minutes, the status monitoring and coordination module notifies the traffic control center, which then increases the traffic ratio from 1% to 5% and repeats the monitoring steps. This cycle continues until 100% traffic is switched over. When 100% traffic is switched to v2.0, the traffic control center instructs the node proxy module to check the v1.0 instance status. After confirming that all existing requests have been processed, the traffic control center instructs the node proxy module to unload the v1.0 class loader and release resources. The node reports that the cleanup is complete, the traffic control center confirms that the global update is successful, and the process ends. Scenario 2 (Abnormal): When the traffic is switched to 1%, monitoring finds that the request response error rate of v2.0 suddenly rises to 15%, and the P99 latency is as high as 1000ms, and step five is executed.

[0078] 5. Monitoring and Rollback

[0079] (a) The status monitoring and coordination module determines that it is “abnormal” and returns the abnormal indicator to the traffic control unit. The traffic control unit reports “v2.0 status abnormal” to the traffic control center. The traffic control center notifies the version management module of the node status abnormality and automatically triggers a global rollback.

[0080] (b) The traffic control center, acting as the brain of traffic, performs a global rollback operation within seconds (<2 seconds), immediately issuing a global emergency rule to the traffic proxy modules of all nodes: "Immediately reset all traffic routing rules to version=v1.0 and completely remove all routes pointing to v2.0"; at the same time, it sends an instruction to the node proxy modules of all updated nodes to "uninstall v2.0";

[0081] (c) After receiving the instruction, the node proxy module unloads the v2.0 class loader and reclaims its memory resources, and the node state is completely restored to the state before the update.

[0082] The traffic control center aggregates the rollback status of all nodes and ultimately displays "Update failed, automatic rollback successful" on the version management module interface, and issues a warning notification to the developers.

[0083] In summary, the distributed cluster software real-time update system and method described in the above embodiments achieve precise global canary control. The traffic control center ensures the consistency and accuracy of traffic switching globally, avoiding traffic routing chaos caused by inconsistent node states. Furthermore, it achieves second-level global fault isolation and rollback. Once any link detects an anomaly, the traffic control center has the ability to issue a global command within seconds, instantly stripping the traffic of the entire cluster from the problematic version. This is incomparable to manual or step-by-step rollback, greatly ensuring business continuity. It also forms an automated closed loop of "monitoring-decision-execution," reducing human intervention, improving operational efficiency and security, while enabling intelligent decision-making based on real-time data, thus improving the level of automation and intelligence.

[0084] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A distributed cluster software real-time update system, characterized in that, This includes the management control plane and the data plane; The management control plane includes a version management module, a traffic control center, and a status monitoring and coordination module. The version management module is used to receive, verify, and store new version software packages and update policies. The traffic control center is used to receive the update status of node proxy modules and dynamically adjust global or local traffic routing rules according to policies and monitoring data, while triggering a global rollback in case of anomalies. The status monitoring and coordination module is used to collect the update status, health status, and performance indicators of all nodes in real time to obtain global health data, provide data basis for the traffic control center's decision-making, and can trigger rollback commands. The data plane includes multiple nodes, each node is equipped with a node proxy module, a traffic proxy module and a business service module; the node proxy module is responsible for communicating with the management and control plane, receiving and executing instructions. The traffic proxy module is used to receive and execute traffic routing rules from the traffic control center; the business service module is used to run software services.

2. The distributed cluster software real-time update system according to claim 1, characterized in that, In the version management module, the update strategy includes the canary release ratio, target node group, and update conditions.

3. The distributed cluster software real-time update system according to claim 2, characterized in that, The node proxy module includes a class loader management unit and a traffic control unit. The class loader management unit is used to create an independent class loader for the new version of the software, so as to realize the isolation and coexistence of multiple versions of code in a single process. The traffic control unit is used to interact with the traffic control center and the local traffic proxy module, report the status, and perform traffic reception and drainage operations for the new and old versions of the software code.

4. A method for real-time updating of distributed cluster software, applied to the system as described in claim 3, comprising the following steps: S1: Upload and Strategy Formulation Upload new version software packages and formulate update strategies through the version management module, and perform security scans and dependency verifications on the software packages; S2: Distribution and Loading According to the update strategy, the version management module distributes the new version software package to the node agent module of the target node group. The node agent module stores the software package in the local cache. The class loader management unit of the node agent module creates a new class loader for the new version software and loads the class files and related resources in it. S3: Traffic Coordination and Canary Release The node proxy module reports its readiness status to the traffic control center; the traffic control center issues initial canary traffic rules to the node proxy module according to the update strategy; the traffic control unit of the node proxy module notifies the local traffic proxy module to start directing new requests to the thread pool created by the new class loader; the existing thread pool that was processing the old version of the software code continues to process the received existing requests, and enters an idle state after processing is completed; the traffic control center makes decisions and repeatedly executes traffic ratio adjustments based on the global health data summarized by the status monitoring and coordination module, until all traffic is switched to the new version; S4: Emptying and Cleaning After all traffic is switched, wait for the old version to finish processing and emptying the existing requests, and then the traffic control center will instruct the node proxy module to unload the old version resources. S5: Monitoring and Rollback During gray-scale or global rollout, if the status monitoring and coordination module detects that the error rate or response delay of the new version request has doubled, it will immediately report the anomaly to the traffic control center. The traffic control center will trigger a rollback and issue emergency rollback rules to the traffic proxy modules across the entire network within seconds, switching all traffic back to the unupdated nodes or the old version threads, and notifying all updated node proxies to uninstall the new version and reload the old version software package.

5. The method for real-time updating of distributed cluster software according to claim 4, characterized in that, In step S2, during the distribution and loading process, the old version of the software code is still running and providing services for the traffic.

6. A distributed cluster software real-time update system according to claim 4, characterized in that, In step S3, the global health data includes CPU utilization, memory utilization, request error rate, and request response time.

7. A distributed cluster software real-time update system according to claim 4, characterized in that, In step S3, the monitoring and rollback are triggered by the status monitoring and coordination module detecting the anomaly in real time and immediately reporting it to the flow control center.

Images