A system for implementing blue-green deployments in a Kubernetes system
By introducing a webhook module for blue-green deployment of custom resources and workloads into the Kubernetes system, a bypass deployment without additional workload is achieved, solving the problems of high resource consumption and user learning costs in existing technologies, and providing an efficient and secure blue-green deployment solution.
Patent Information
- Application Number
- CN202411026930.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2025-11-11
- Estimated Expiration
- 2044-07-30
AI Technical Summary
When implementing blue-green deployments in Kubernetes systems, existing technologies require the creation of additional workloads, increasing resource burden and operational overhead. Furthermore, the coupling of custom tools with native workloads leads to high learning costs for users and an opaque deployment process.
It adopts a blue-green deployment of custom resources, a webhook module for workloads, and a custom resource controller. It monitors new version releases through custom resource definitions and webhook mechanisms, pauses workloads and controls blue-green deployments, supports batch deployments, traffic switching, and metric analysis, and achieves bypass deployments without additional workloads.
It simplifies the release process, supports second-level traffic switching and rapid version rollback, reduces business risks, ensures workload information consistency, reduces resource consumption, and improves release efficiency and security.
Smart Images

Figure CN118963817B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and more particularly to a system for implementing blue-green deployment in a Kubernetes system. Background Technology
[0002] With the increasing prevalence of microservices and cloud-native applications, securely and efficiently releasing new versions has become a significant challenge. Blue-green deployment, as a mature deployment strategy, can reduce user downtime during system upgrades and improve system availability and stability. However, implementing blue-green deployment in Kubernetes typically requires creating additional workloads (such as tools like Flager), which not only increases system resource burden but also complicates deployment processes and increases operational overhead. On the other hand, tools like Argo Rollouts support blue-green deployment using custom workloads, but because the deployment strategy is tightly coupled with the workload, users must migrate workloads when they cannot use native Kubernetes workloads (such as Deployments), leading to increased learning costs and a lack of transparency in the deployment process. Summary of the Invention
[0003] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide a system for implementing blue-green deployment in the Kubernetes system.
[0004] The technical solution adopted in this invention is:
[0005] A system for implementing blue-green deployment in Kubernetes includes:
[0006] Blue-green publishing of custom resources;
[0007] The workload's webhook module is used to monitor new version releases and perform pre-processing for blue-green deployments of workloads;
[0008] A custom resource controller is used to define custom resources based on blue-green deployments and execute deployment control logic on the corresponding workloads to achieve the desired blue-green deployment behavior.
[0009] Specifically, when the webhook module detects a new version release, it checks whether the release should be allowed. If allowed, the release of the new version is temporarily paused so that the custom resource controller can manage the release later.
[0010] Furthermore, the blue-green publishing of custom resources includes:
[0011] Custom resource definitions (Rollout CRDs) are designed to register new resource types, namely Rollout resources, through the Kubernetes system's API server, thereby extending the functionality of Kubernetes and implementing the blue-green deployment strategy.
[0012] Custom resources (Rollout CR) are based on custom resource definitions. Users can create instances of custom resources named Rollout, which allow users to configure the blue-green deployment process in detail.
[0013] Custom webhooks for resources are used to perform configuration validity checks and prevent unauthorized modifications when resources are created or updated, thereby ensuring the accuracy of the publishing strategy and the stability of the system.
[0014] Furthermore, the webhook module of the workload operates as follows:
[0015] A1. Check if there is a new version release for the workload. If a new version release is found, proceed to the next step.
[0016] A2. Check if the Rollout resource exists and determine whether a blue-green deployment should be performed. If yes, proceed to the next step. If no, the blue-green strategy will not be used for this deployment. Return to step A1.
[0017] A3. Save some workload information to comments; The purpose of this step is that some fields of the workload will be updated during the blue-green deployment process. Saving the original information will allow the original configuration to be restored after the deployment is completed, ensuring the consistency of the workload before and after the blue-green deployment.
[0018] A4. Pause the workload; the purpose of this step is to prevent the logic of the workload's native controller from interfering with blue-green deployments; the workload will resume from the paused state after the Rollout controller starts executing the blue-green deployment logic.
[0019] After A5 and webhook work is completed, the subsequent blue-green deployment process is handled by a custom resource controller.
[0020] Furthermore, the custom resource controller includes a batch release module, a traffic switching module, a measurement and analysis module, and a rollback module;
[0021] The batch release module is used to release new versions of Pods in batches;
[0022] The traffic switching module is used to realize the batch switching of traffic during the blue-green deployment process, and to ensure the lossless traffic during the switching process;
[0023] The measurement and analysis module is used to verify whether the released new version has met the preset performance and stability standards, thereby determining whether the release process can continue or should be terminated.
[0024] The rollback module is used to roll back the application state to the previous stable version when it detects that the version update does not meet expectations, so as to ensure business continuity and service stability.
[0025] Furthermore, the specific working method of the batch release module is as follows:
[0026] B1. Configure the workload's minReadySeconds to infinity, configure maxUnavailability to 0, and resume the workload from a paused state;
[0027] B2. Configure the maxSurge for the current batch based on the Rollout resources;
[0028] B3. Determine if all the new version Pods are ready. If so, continue to determine if the metrics analysis module allows entry into the next stage. If not, determine that the creation of the new version Pods has failed, and this release is marked as a failure. Then, call the rollback module to reset this release.
[0029] B4. Based on the Rollout resources, repeat steps B2-B3 until maxSurge is configured to 100%;
[0030] B5. Release complete. Restore the original configuration from the workload's comments, including minReadySeconds, maxUnavailable, and maxSurge; the stable version will be scaled down, and the new version will be used as the new stable version.
[0031] Furthermore, the operation of the traffic switching module is as follows:
[0032] C1. The module performs initialization tasks;
[0033] C2. Traffic is switched gradually. The routing policy of the network configuration object is dynamically updated according to the routing policy configured in the current batch. The routing policy includes weight-based, request header and cookie-based policies.
[0034] C3. Determine whether the metrics analysis module allows entry into the next stage. If yes, continue to step C4; otherwise, this release is marked as a failure, and the rollback module will be called to reset this release.
[0035] C4. Release complete. The module is now undergoing finalization.
[0036] Further, step C1 includes:
[0037] 1) Check that the required Service resources and network configuration resources exist and are configured correctly;
[0038] 2) Update network configuration resources to ensure all traffic is routed to the stable version;
[0039] 3) Depending on the different network configuration resources, configure the Service object to select only the old version Pods, or to select only the new version Pods.
[0040] Further, step C4 includes:
[0041] 1) Update network configuration resources to ensure all traffic is routed to the new version;
[0042] 2) Reduced-size stable version;
[0043] 3) Restore Service resources;
[0044] 4) Restore network configuration resources.
[0045] Furthermore, the operation of the measurement and analysis module is as follows:
[0046] D1. Initialize the measurement and analysis environment. Before the workload enters the measurement and analysis phase, it is necessary to ensure that all relevant monitoring and measurement tools are active and can capture and report data normally.
[0047] D2. Real-time monitoring and measurement: The system will collect data in real time based on preset metrics, and this data will become the basis for decision-making in subsequent steps.
[0048] D3. Evaluate the collected metric data based on the success and failure conditions defined in the Rollout resource;
[0049] D4. Determine whether the conditions for entering the next release stage are met. If the analysis results of the current step meet the user-configured strategy, the metrics analysis module will allow the update to enter the next release batch; otherwise, the rollback mechanism will be triggered.
[0050] D5. Record and visualize the results of metric analysis.
[0051] Furthermore, the rollback module operates as follows:
[0052] E1. Determine rollback trigger conditions: When the metrics analysis module determines that the current new version does not meet the release standards, or when the user actively terminates the release process, the rollback module is activated and ready to execute the rollback task.
[0053] E2. Implement a phased rollback strategy: According to the batches configured in the Rollout resources, the system will gradually redirect traffic from the new version back to the stable version. This process does not involve adjusting the number of Pods between the old and new versions, but only operates on the traffic in order to achieve rapid service quality recovery and reduce network fluctuations caused by sudden traffic switching.
[0054] E3. Complete traffic switching and status check: After all traffic has been successfully switched back to the stable version, users can choose to continue to observe the application status to troubleshoot the problem, or immediately perform a global rollback to scale down the problematic new version.
[0055] E4. Perform a global rollback: Users trigger a global rollback by redeploying a stable version of the workload. After the system detects this operation, it will start the global rollback process. The global rollback will not be performed according to the batch configured in the Rollout resource, but will roll back all changes related to the new version, including traffic routing and workload scaling down.
[0056] E5. Perform other cleanup and recovery: After the rollback operation is completed, the system will clean up all resources and configurations related to the release of the new version, including the corresponding routing rules and related monitoring settings, to ensure the accuracy and cleanliness of application resources and configurations, and avoid potential configuration chaos and resource waste.
[0057] The beneficial effects of this invention are: it performs blue-green deployment of the target workload in a bypass manner, without changing the workload's metadata before and after deployment; it eliminates the need to create additional workloads, simplifying deployment management, and supports deployment of existing workloads. Furthermore, this invention supports second-level traffic switching, rapid version rollback, and optional multi-batch deployment and continuous deployment strategies, reducing the business risks associated with deploying new versions. Attached Figure Description
[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0059] Figure 1 This is a schematic diagram illustrating the operation of a system implementing blue-green deployment in a Kubernetes system according to an embodiment of the present invention;
[0060] Figure 2 This is a schematic diagram of the structure for blue-green publishing of custom resources in an embodiment of the present invention;
[0061] Figure 3 This is a flowchart of the workflow of the webhook module of the workload in this embodiment of the invention;
[0062] Figure 4 This is a flowchart of the workflow of the custom resource controller in an embodiment of the present invention;
[0063] Figure 5 This is a flowchart of the batch release module in this embodiment of the invention;
[0064] Figure 6 This is a flowchart of the flow switching module in an embodiment of the present invention;
[0065] Figure 7 This is a flowchart of the measurement and analysis module in an embodiment of the present invention;
[0066] Figure 8 This is a flowchart of the rollback module in an embodiment of the present invention. Detailed Implementation
[0067] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.
[0068] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.
[0069] In the description of this invention, "several" means one or more, "multiple" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. If "first" or "second" is used, it is only for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features. Furthermore, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0070] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0071] Terminology Explanation:
[0072] Kubernetes (K8s): A widely used open-source container orchestration system for automating the deployment, scaling, and management of application containers.
[0073] Blue-Green Deployment: A software deployment model where "blue" represents the production version and "green" represents the new version. This method allows two versions to run simultaneously on the system and enables rapid switching between them.
[0074] minReadySeconds: Minimum number of seconds to be ready. A Kubernetes Deployment property that defines the minimum number of seconds a Pod must wait before being considered ready after it has been created and started.
[0075] Currently, Argo Rollouts is a relatively mature solution in the community that provides complete application deployment capabilities, and it offers a blue-green deployment solution. However, it has the following drawbacks: (1) Using custom workloads, the cost for users to learn the tool is high, and existing workloads cannot be used directly with the tool, requiring workload migration (from the original Deployment to the custom workload); it cannot utilize the controller of the native workload, thus making the implementation more complex, and the expected behavior may be inconsistent with the controller of the native workload. (2) It only supports one type of stateless workload, resulting in poor scalability. (3) The rollback strategy for blue-green deployments is singular. After rolling back to the production version, the new version of Pods will be scaled down, so it is impossible to switch traffic to the new version again.
[0076] Flagger is also a relatively mature solution in the community that provides complete application deployment capabilities, and it also provides a blue-green deployment solution. However, it has the following drawbacks: (1) Its functions are too simple, such as not supporting traffic rollback and not supporting batch deployment. (2) It creates additional Deployments and modifies the Deployments deployed by the business, which not only increases the consumption of system resources, but also causes confusion for users. (3) During the deployment process, it modifies the name of the Deployment and the label of the Pod, so all resources selected based on the name of the load or the label of the Pod will not take effect.
[0077] To address existing technical issues, this invention aims to simplify the blue-green deployment process in Kubernetes environments. By configuring existing workloads with custom resources, it achieves a smooth transition and rapid rollback without additional expansion or reconstruction. Based on this, this invention provides a bypass-based, hot-swappable blue-green deployment solution that eliminates the need to create additional workloads or migrate existing ones. Before and after a blue-green deployment, the workload's identification information, such as the Deployment name, remains unchanged. Therefore, other configurations for the workload application (such as horizontal autoscaling (HPA), Pod interference budget (PDB), etc.) and components like logging and monitoring still function normally, ensuring business continuity and complete user control over the deployment process. The bypass-based nature is mainly reflected in: 1) Using the blue-green deployment method of this invention, users only need to create a Rollout resource without manually modifying existing workloads (unlike Argo Rollouts, which require users to use their workloads for deployment). 2) It can be created and deleted at any time without affecting the workload itself (after creation, the workload will be deployed using a blue-green deployment; after deletion, the workload will be deployed using the original Kubernetes deployment method).
[0078] like Figure 1 As shown, this embodiment provides a system for implementing blue-green deployment in a Kubernetes system, including:
[0079] Blue-green publishing of custom resources;
[0080] The workload's webhook module is used to monitor new version releases and perform pre-processing for blue-green deployments of workloads;
[0081] A custom resource controller is used to define custom resources based on blue-green deployments and execute deployment control logic on the corresponding workloads to achieve the desired blue-green deployment behavior.
[0082] Specifically, when the webhook module detects a new version release, it checks whether the release should be allowed. If allowed, the release of the new version is temporarily paused so that the custom resource controller can manage the release later.
[0083] In some embodiments, taking Deployment as an example, the release of a new version is detected by the Deploymentwebhook module of this invention. The webhook checks whether this release should be allowed. If allowed, the Deployment is placed in a brief pause state so that the controller can perform more granular release management later. Then, under the coordination of the Deployment native controller and the custom resource controller, the Deployment will perform this blue-green release according to the rollout strategy submitted by the user until the release is complete.
[0084] The following section provides a detailed explanation of each of the above modules in conjunction with the accompanying drawings.
[0085] (1) Blue-green publishing of custom resources
[0086] As an optional implementation method, such as Figure 2 As shown, the custom resource for blue-green deployment includes a Custom Resource Definition (CRD), a Custom Resource (CR), and a webhook for the custom resource. This embodiment defines a CRD named "Rollout." This CRD aims to register new resource types, namely Rollout resources, through the Kubernetes API server, thereby extending Kubernetes functionality and implementing the blue-green deployment strategy. Based on the Rollout CRD, users can create CR instances named Rollout. This instance allows users to configure the blue-green deployment process in detail, including but not limited to deployment phase configurations, traffic allocation strategies, and analysis strategies based on real-time monitoring metrics. Furthermore, in addition to binding to supported workload resources, Rollout resources can also bind to other resources in Kubernetes, such as Service resources, Ingress, and other gateway resources, and dynamically update them, further enhancing the traffic management capabilities of blue-green deployments. This invention also sets up a webhook mechanism for Rollout resources. The Rollout webhook is mainly responsible for verifying the legality of configurations and preventing improper modifications when resource creation, updates, and other operations occur, ensuring the accuracy of the deployment strategy and the stability of the system.
[0087] (2) Webhook module for workload
[0088] As an optional implementation method, such as Figure 3As shown, the webhook module for this workload is designed to monitor new version releases and perform pre-processing for blue-green deployments of the workload, including the following steps:
[0089] Step 110: Check if a new version has been released for the workload. If a new version is found, proceed to the next step.
[0090] Step 120: Check if the Rollout resource exists and determine whether a blue-green deployment should be performed. If yes, proceed to the next step; otherwise, the blue-green strategy will not be used for this deployment, and return to step 110.
[0091] Step 130: Save some workload information to comments. The purpose of this step is to save the original information so that the original configuration can be restored after the blue-green deployment, ensuring the consistency of the workload before and after the blue-green deployment, as some fields of the workload will be updated during the blue-green deployment process.
[0092] Step 140: Pause the workload. For example, for a Deployment resource, you can configure the pause field to true. The purpose of this step is to prevent the logic of the workload's native controller from interfering with blue-green deployments. The workload will resume from the paused state after the Rollout controller starts executing the blue-green deployment logic.
[0093] Step 150: The webhook operation is complete. The subsequent blue-green deployment process is handled by the controller of the custom resource. Return to step 110.
[0094] For ease of understanding, it is necessary to emphasize that the workload webhook module mentioned here is not the native workload webhook of Kubernetes, and is even more different from the webhook module for custom resources.
[0095] (3) Custom resource controller
[0096] The function of this custom resource controller is to execute release control logic on the corresponding workload based on the custom resources submitted by the user, so as to achieve the desired blue-green release behavior. For example... Figure 4 As shown, the custom resource controller consists of four parts: a batch release module, a traffic switching module, a measurement and analysis module, and a rollback module.
[0097] To explain the concept of batches more clearly, the following example illustrates a possible blue-green deployment strategy:
[0098] Batch 1: 50% replicas, 0% traffic: Create a new version with 50% of the number of stable version Pods (this is usually for resource saving purposes), and do not route traffic to the new version.
[0099] Batch 2: 100% replicas, 0% traffic: Increase the number of Pods from the new version to the stable version, and do not route traffic to the new version.
[0100] Batch 3: 100% replicas, 50% traffic: Gradually switch traffic to the new version Pod, where the new version and the stable version each receive 50% of the traffic.
[0101] Batch 4: 100% replicas, 100% traffic: All traffic is switched to the new version Pod. After final analysis and confirmation, the release is completed, and the stable version is scaled down.
[0102] It is important to emphasize that this example is used only to more intuitively explain the concept of batches. In fact, in the custom resource rollout of this invention, users can flexibly adjust the publishing parameters, including but not limited to: the number of publishing batches, the expected number of new version Pods in a specific batch, the expected traffic strategy for a specific batch, and the conditions for entering the next batch.
[0103] (a) Phased Release Module
[0104] As an optional implementation method, such as Figure 5 As shown, its function is to release new versions of Pods in batches, including the following steps:
[0105] Step 210: Configure the workload's minReadySeconds to infinity, configure maxUnavailability to 0, and resume the workload from the paused state.
[0106] Step 220: Optionally, execute the release strategy in batch order. The batch release module configures the maxSurge for the current batch based on the submitted Rollout resources. For example, if the expected number of new version Pods in the first batch is 50%, the batch release module will configure the maxSurge for the workload to 50%.
[0107] Step 230: Determine if all new version Pods are ready. If so, proceed to determine if the metrics analysis module allows entry into the next stage. If not, it indicates that the creation of new version Pods failed. This could be due to issues with the image or configuration, or insufficient system resources. Consequently, this release is marked as failed, and the rollback module can be called to reset it.
[0108] Step 240: Configure the workload's maxSurge to 100%. Now the number of Pods in the new version is the same as the number of Pods in the stable version.
[0109] Step 250: Repeat step 230.
[0110] Step 260: Release complete. Restore the original configuration from the workload's comments, including minReadySeconds, maxUnavailability, and maxSurge. The stable version will be scaled down, and the new version will become the new stable version.
[0111] It is necessary to explain some concepts here:
[0112] `maxSurge`: Specifies the maximum number of additional Pods that can be created during a rolling update. It can be an absolute value, such as 5, or a percentage of the desired number of Pods, such as 10%. This means that during the update process, the number of Pods can temporarily exceed the expected total to more smoothly replace older Pods. The default value is 25%.
[0113] `maxUnavailable`: Specifies the maximum number of Pods that can be unavailable during a rolling update. It can be an absolute value or a percentage of the desired number of Pods; the default value is 25%. This ensures that a certain percentage of services are available even during the update process.
[0114] `minReadySeconds`: Specifies the minimum time (in seconds) a newly created Pod should take to reach a ready state before it is considered available, provided there are no container crashes. The default value is 0, meaning the Pod is considered available immediately once it is ready.
[0115] This embodiment sets `minReadySeconds` to infinity, so newly created Pods will never enter an available state, but will still be in a ready state, thus allowing them to receive traffic. Furthermore, this method sets `maxUnavailability` to 0, so Kubernetes' native workload control logic will not scale down stable versions, achieving a "blue-green coexistence" state within a single workload. Therefore, this embodiment does not require an additional workload. Moreover, this method achieves the aforementioned batch release effect by dynamically adjusting `MaxSurge`.
[0116] (b) Traffic switching module
[0117] In some embodiments, the traffic switching module functions to enable batch switching of traffic during blue-green deployment and ensure lossless traffic flow during the switching process. For example... Figure 6 As shown, it includes the following steps:
[0118] Step 310: The module performs initialization tasks, including:
[0119] 1) Check that the required Service resources and network configuration resources exist and are configured correctly.
[0120] 2) Update network configuration resources to ensure that all traffic is routed to the stable version.
[0121] 3) Depending on the network configuration resources, optionally, configure the Service object to select only older version Pods.
[0122] 4) Depending on the different network configuration resources, optionally, create a Service object that only selects the new version of Pods.
[0123] Step 320: Traffic is switched gradually. The routing policy of the network configuration object is dynamically updated according to the routing policy configured in the current batch. The routing policy includes weight-based, request header and cookie-based policies.
[0124] Step 330: Determine whether the metrics analysis module allows entry into the next stage. If yes, continue to step 340. If no, this release is marked as a failure, and the rollback module will be called to reset this release.
[0125] Step 340: Release complete. The module undergoes final finishing touches, including:
[0126] 1) Update network configuration resources to ensure that all traffic is routed to the new version.
[0127] 2) The scaled-down stable version described in step 260.
[0128] 3) Restore Service resources.
[0129] 4) Restore network configuration resources.
[0130] The network configuration resources mentioned typically refer to resources that can act as traffic gateways in the Kubernetes system, including but not limited to Kubernetes ingress, Gateway API, and service mesh gateway.
[0131] It is important to emphasize here that batch traffic switching according to the method described in this invention can achieve lossless switching regardless of whether the routing strategy is based on weight, request headers, or cookies. This means that during the switching process, there should be no instances of correct access resulting in failed responses. Furthermore, the Service or network configuration resources remain unchanged before and after the deployment, ensuring a "clean" state after the blue-green deployment.
[0132] (c) Measurement and Analysis Module
[0133] In some embodiments, such as Figure 7As shown, the metrics analysis module's function is to verify whether the released version meets preset performance and stability standards, thereby determining whether the release process can continue or should be terminated. The core of the metrics analysis module lies in the introduction of a metrics analysis strategy, which allows the system to decide on the release's progress based on real-time metrics, or, if necessary, perform a rollback. It includes the following steps:
[0134] Step 410: Initialize the metrics analysis environment. Before the workload enters the metrics analysis phase, it needs to ensure that all relevant monitoring and metrics tools are active and can capture and report data normally.
[0135] Step 420: Perform real-time monitoring and measurement. The system will collect data in real time based on preset metrics, such as error rate, response time, and load capacity. This data will serve as the basis for decision-making in subsequent steps.
[0136] Step 430: Evaluate the collected metrics data based on the success and failure criteria defined in the Rollout resources. For example, if the success criterion is an error rate below 1%, and metrics analysis confirms that the error rate is indeed below this standard, then this stage of the release is considered successful.
[0137] Step 440: Determine if the conditions for proceeding to the next release phase are met. If the analysis results of the current step meet the user-configured strategy (e.g., multiple consecutive metric results meet expectations), the metric analysis module will allow the update to proceed to the next release batch; otherwise, a rollback mechanism may be triggered.
[0138] Step 450: Record and visualize the metrics analysis results. To enhance the auditability of the system, every step and result of the metrics analysis is recorded. Furthermore, the metrics analysis process is visualized to users, which helps them understand the release process and make informed decisions.
[0139] Furthermore, the metrics and analysis module of this invention supports data collection via external monitoring systems, such as Prometheus, and can query specified time-series databases through defined analysis runs. Through this metrics and analysis module, this invention significantly improves the efficiency and security of blue-green deployments in Kubernetes, while minimizing manual intervention and achieving automated risk control.
[0140] (d) Rollback module
[0141] In some embodiments, such as Figure 8As shown, the rollback module's function is to safely and efficiently revert the application state to the previous stable version when a version update is detected as not meeting expectations, thus ensuring business continuity and service stability. The specific steps are as follows:
[0142] Step 510: Determine the rollback trigger conditions. When the metrics analysis module determines that the current new version does not meet the release standards, or when the user actively terminates the release process, the rollback module is activated and ready to execute the rollback task.
[0143] Step 520: Execute the phased rollback strategy. According to the batches configured in the Rollout resources, the system will gradually redirect traffic from the new version back to the stable version. This process does not involve adjusting the number of Pods between the old and new versions; it only operates on traffic to achieve rapid service quality restoration while reducing network fluctuations caused by sudden traffic switching.
[0144] Step 530: Complete traffic switching and status check. After all traffic has successfully switched back to the stable version, users can choose to continue monitoring the application status to troubleshoot the problem, or immediately perform a global rollback to scale down the problematic new version.
[0145] Step 540: Perform a global rollback. The user triggers a global rollback by redeploying a stable version of the workload. Upon detecting this operation, the system will initiate the global rollback process. The global rollback will not be performed according to the batches configured in the Rollout resource, but will roll back all changes related to the new version, including traffic routing and workload scaling down.
[0146] Step 550: Other Cleanup and Recovery. After completing the rollback operation, the system will clean up all resources and configurations related to the release of the new version, including corresponding routing rules and related monitoring settings, to ensure the accuracy and cleanliness of application resources and configurations, and avoid potential configuration chaos and resource waste.
[0147] The described rollback module emphasizes risk control and service availability assurance during the release of new versions. Its phased rollback strategy focuses on quickly mitigating the potential impact of new versions without affecting service continuity. Combined with a global rollback strategy, this method balances rollback efficiency and system stability, providing a safe and reliable fallback for blue-green deployments on the Kubernetes platform.
[0148] In summary, the present invention has at least the following advantages and beneficial effects compared to the prior art:
[0149] (1) The present invention adopts a bypass-type, hot-swappable blue-green deployment scheme, which does not require migration of existing workloads and supports deployment of existing workloads.
[0150] (2) By properly configuring the native fields in the workload, this method does not require the creation of additional workloads during blue-green deployment, thus avoiding the consumption of additional resources and simplifying the management of the deployment process.
[0151] (3) This invention provides richer functionality, supports second-level traffic switching, rapid version rollback, and optional multi-batch release and continuous release strategies, reducing the business risks associated with releasing new versions. It can also work in conjunction with various mainstream gateways and service meshes, supporting multiple traffic switching and matching strategies.
[0152] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0153] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
[0154] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A system for implementing blue-green deployment in Kubernetes, characterized in that, include: Blue-green publishing of custom resources; The workload's webhook module is used to monitor new version releases and perform pre-processing for blue-green deployments of workloads; A custom resource controller is used to define custom resources based on blue-green deployments and execute deployment control logic on the corresponding workloads to achieve the desired blue-green deployment behavior. Specifically, when the webhook module detects a new version release, it checks whether the release should be allowed. If allowed, the release of the new version is paused so that the custom resource controller can manage the release. The webhook module of the workload works as follows: A1. Check if there is a new version release for the workload. If a new version release is found, proceed to the next step. A2. Check if the Rollout resource exists and determine whether a blue-green deployment should be performed. If yes, proceed to the next step; otherwise, this deployment will not use the blue-green strategy, and return to step A1. A3. Save some workload information to comments; The purpose of this step is that some fields of the workload will be updated during the blue-green deployment process. Saving the original information will allow the original configuration to be restored after the deployment is completed, ensuring the consistency of the workload before and after the blue-green deployment. A4. Pause the workload; the purpose of this step is to prevent the logic of the workload's native controller from interfering with blue-green deployments; the workload will resume from the paused state after the Rollout controller starts executing the blue-green deployment logic. After A5 and webhook work is completed, the subsequent blue-green deployment process is handled by a custom resource controller.
2. The system for implementing blue-green deployment in a Kubernetes system according to claim 1, characterized in that, The blue-green publishing of custom resources includes: Custom resource definitions are used to register new resource types, such as Rollout resources, through the Kubernetes system's API server, thereby extending the functionality of Kubernetes and implementing the blue-green deployment strategy. Custom resources, based on custom resource definitions, allow users to create instances of custom resources named Rollout, which allow users to configure the blue-green deployment process in detail; Custom webhooks for resources are used to perform configuration validity checks and prevent unauthorized modifications when resources are created or updated, thereby ensuring the accuracy of the publishing strategy and the stability of the system.
3. A system for implementing blue-green deployment in a Kubernetes system according to claim 1, characterized in that, The custom resource controller includes a batch release module, a traffic switching module, a measurement and analysis module, and a rollback module; The batch release module is used to release new versions of Pods in batches; The traffic switching module is used to realize the batch switching of traffic during the blue-green deployment process, and to ensure the lossless traffic during the switching process; The measurement and analysis module is used to verify whether the released new version has met the preset performance and stability standards, thereby determining whether the release process should continue or should be terminated. The rollback module is used to roll back the application state to the previous stable version when it detects that the version update does not meet expectations, so as to ensure business continuity and service stability.
4. A system for implementing blue-green deployment in a Kubernetes system according to claim 3, characterized in that, The specific working method of the batch release module is as follows: B1. Configure the workload's minReadySeconds to infinity, configure maxUnavailable to 0, and resume the workload from the paused state; minReadySeconds specifies the minimum time a newly created Pod will take to reach a ready state without container crashes before it is considered available; maxUnavailable specifies the maximum number of Pods that are unavailable during a rolling update. B2. Configure the maxSurge for the current batch based on the Rollout resources; the maxSurge specifies the maximum number of additional Pods created during the rolling update. B3. Determine if all Pods of the new version are ready. If so, continue to determine if the metrics analysis module allows entry into the next stage. If not, determine that the creation of the new version Pods failed, this release is marked as a failure, and then call the rollback module to reset this release. B4. Based on the Rollout resources, repeat steps B2-B3 until maxSurge is configured to 100%; B5. Release complete. Restore the original configuration from the workload's comments, including minReadySeconds, maxUnavailable, and maxSurge; the stable version will be scaled down, and the new version will be used as the new stable version.
5. A system for implementing blue-green deployment in a Kubernetes system according to claim 3, characterized in that, The specific working principle of the traffic switching module is as follows: C1. The module performs initialization tasks; C2. Traffic is switched gradually, and the routing policy of the network configuration object is dynamically updated according to the routing policy configured in the current batch. C3. Determine whether the metrics analysis module allows entry into the next stage. If yes, continue to step C4; otherwise, this release is marked as a failure, and the rollback module will be called to reset this release. C4. Release complete. The module is now undergoing finalization.
6. A system for implementing blue-green deployment in a Kubernetes system according to claim 5, characterized in that, Step C1 includes: 1) Check that the required Service resources and network configuration resources exist and are configured correctly; 2) Update network configuration resources to ensure all traffic is routed to the stable version; 3) Depending on the different network configuration resources, configure the Service object to select only the old version Pods, or to select only the new version Pods.
7. A system for implementing blue-green deployment in a Kubernetes system according to claim 5, characterized in that, Step C4 includes: 1) Update network configuration resources to ensure all traffic is routed to the new version; 2) Reduced-size stable version; 3) Restore Service resources; 4) Restore network configuration resources.
8. A system for implementing blue-green deployment in a Kubernetes system according to claim 3, characterized in that, The rollback module operates as follows: E1. Determine rollback trigger conditions: When the metrics analysis module determines that the current new version does not meet the release standards, or when the user actively terminates the release process, the rollback module is activated and ready to execute the rollback task. E2. Implement a phased rollback strategy: According to the batches configured in the Rollout resources, the system will gradually redirect traffic from the new version back to the stable version. This process does not involve adjusting the number of Pods between the old and new versions, but only operates on the traffic in order to achieve rapid service quality recovery and reduce network fluctuations caused by sudden traffic switching. E3. Complete traffic switching and status check: After all traffic has been successfully switched back to the stable version, users can choose to continue to observe the application status to troubleshoot the problem, or immediately perform a global rollback to scale down the problematic new version. E4. Perform a global rollback: Users trigger a global rollback by redeploying a stable version of the workload. After the system detects this operation, it will start the global rollback process. The global rollback will not be performed according to the batch configured in the Rollout resource, but will roll back all changes related to the new version, including traffic routing and workload scaling down. E5. Perform other cleanup and recovery: After the rollback operation is completed, the system will clean up all resources and configurations related to the release of the new version, including the corresponding routing rules and related monitoring settings, to ensure the accuracy and cleanliness of application resources and configurations, and avoid potential configuration chaos and resource waste.
Citation Information
Patent Citations
Automatic blue green deployment method based on container technology
CN111142920A
Blue-green automatic publishing method based on automatic publishing platform
CN113986334A