Micro-service adaptive gray release method and system

Abstract

The present application relates to a kind of microservice adaptive gray release method and system.The method includes: building new version microservice program for release, submitting gray task and setting gray task parameter;Real-time collection of the running condition index of each node in cluster, node running state score and node traffic saturation are calculated;According to node running state score and node traffic saturation, periodically select a proper amount of microservice nodes for gray replacement deployment, after each gray replacement is completed, according to the running state score of gray node, state abnormal node is rolled back according to rollback strategy, and according to the expected gray release end time and the number of remaining nodes that have not been gray replaced, the state of current gray task is determined.The present application dynamically adjusts the deployment scale of gray service by continuously monitoring the running state of microservice cluster, can quickly respond to service exception, and improves user experience in the process of gray release.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a method and system for adaptive canary deployment of microservices. Background Technology

[0002] In recent years, with the rise of cloud-native technologies such as microservices and containerization, and the continuous development of highly agile software development methods such as model-driven, low-code, and no-code approaches, the software development iteration cycle has been greatly shortened, posing significant challenges to the software release and testing of various cutting-edge, software-intensive large-scale technology enterprises. Typically, enterprises mainly adopt methods such as blue-green deployment, canary release, and canary release to integrate continuous deployment into the process, supporting user experience and A / B testing during the launch of new version services.

[0003] Currently, enterprises widely adopt canary release methods for continuous deployment. This involves incrementally replacing older microservice nodes in the production cluster with newer ones during the deployment of a new version, and rolling back if a microservice fails. The canary release process requires manual decisions by operations or deployment personnel based on user feedback, the collection and analysis of extensive monitoring logs, and operational data. However, this method is inefficient for adjusting microservice clusters. If a failure occurs during canary deployment, the deployment ratio cannot be adjusted promptly, impacting user experience and increasing the risk of user churn. Summary of the Invention

[0004] To address the issues of significant manual intervention, low efficiency in cluster deployment adjustments, and lack of adaptability in existing canary deployment methods, this invention proposes a microservice adaptive canary deployment method and system. During the continuous deployment of a microservice cluster, the deployment scale of the canary service is dynamically adjusted by continuously monitoring the operational status of the microservice cluster, thereby quickly responding to service anomalies and improving the user experience during the canary deployment process.

[0005] The technical solution adopted in this invention is as follows:

[0006] Firstly, this invention provides a microservice adaptive canary deployment method, the main steps of which are as follows:

[0007] Build a new version of the microservice application for release, submit the canary release task, and set the canary release task parameters;

[0008] Real-time collection of operational metrics for each microservice node in the microservice cluster; calculation of node operational status score based on node status scoring model; calculation of node traffic saturation based on node traffic saturation model.

[0009] Based on the node running status score and node traffic saturation, a suitable number of microservice nodes are periodically selected for canary deployment. After each canary deployment, nodes with abnormal status are rolled back according to the rolling back strategy based on the running status score of the canary nodes (nodes after canary deployment). The status of the current canary task is determined based on the expected end time of the canary release and the number of remaining nodes that have not undergone canary deployment.

[0010] Preferably, the grayscale task parameters mainly include: expected service response time R. e Maximum expected request volume F per service unit time e Number of grayscale release nodes, step size n, grayscale replacement period T r The expected duration of the gray-scale task is T.

[0011] Preferably, the operational metrics of each microservice node include:

[0012] T r The time consumed by 90% of the microservice quantiles is denoted as R. 90％ ;

[0013] T r Internal microservice call error rate, denoted as: Where, N error For T r The number of abnormal responses within, N total For T r Total number of requests within.

[0014] Preferably, the method for calculating the node operating status score based on the node status scoring model is as follows:

[0015] The operational status score of a microservice node is recorded as follows: Wherein, α is usually 0 to 1 (preferably 0.4), and β is usually 0 to 1 (preferably 0.6).

[0016] Preferably, the method for calculating node traffic saturation based on the node traffic saturation model is as follows:

[0017] T r The average number of requests per unit time for an internal microservice is denoted as: Where f is T r Total number of requests within the microservice;

[0018] The traffic saturation measure for microservice nodes is:

[0019] Preferably, the method of periodically selecting an appropriate number of nodes for grayscale replacement deployment, and after each grayscale replacement, rolling back nodes with abnormal status according to the rollback strategy based on the grayscale node running status score, is as follows:

[0020] Step S1: Select nodes with a flow saturation D less than 1 from the set of nodes to be replaced Z, and sort the selected nodes in ascending order of flow saturation to obtain the candidate node set S;

[0021] Step S2: If If so, proceed to step S3; otherwise, proceed to step S6.

[0022] Step S3: If |S|≥n, then for the node set S consisting of the n nodes with the smallest flow saturation in S... n Perform a replacement deployment (the replacement action means deleting the microservice on that node and deploying the new version of the microservice), Z = ZS n Then proceed to step S5; otherwise, proceed to step S4.

[0023] Step S4: Perform grayscale replacement on all nodes in S, Z = ZS;

[0024] Step S5: At interval T r Then, for the node set after grayscale replacement, check T. r The system scores the running status of nodes within the system. If any node is found to be in an abnormal state, it rolls back the node and executes step S5 (i.e., it repeatedly executes step S5, which itself is an iterative function that continuously increments by a time interval T). r Execute until there are no abnormal nodes in the state, then exit the loop; otherwise, execute step S6.

[0025] Step S6: If If so, proceed to step S7; otherwise, proceed to step S9.

[0026] Step S7: If the current time is less than the expected duration T of the grayscale task, execute step S1; otherwise, execute step S8.

[0027] Step S8: If the current time is greater than or equal to the expected duration T of the canary deployment task, end the current canary deployment task, roll back all replaced nodes, and submit the canary deployment failure result.

[0028] Step S9: If the current time is less than the expected duration T of the grayscale task, proceed to step S5; otherwise, proceed to step S10.

[0029] Step S10: If the current time is greater than or equal to the expected duration T of the grayscale task, end the current grayscale task and submit the successful grayscale release result.

[0030] Preferably, in step S5, the method for selecting abnormal nodes is as follows:

[0031] Two-stage thresholds p (preferably 0.6) and q (preferably 1) are preset. If the maximum node running state score is greater than p and less than q, the node with the maximum running state score is selected as the abnormal node. If the running state score of the node with the maximum score is greater than q, the node with the running state score greater than q is selected as the abnormal node.

[0032] Optionally, in step S5, the number of rollback triggers for the grayscale task can be recorded. After the grayscale task reaches a certain number of rollback triggers, the current grayscale task can be terminated early and the grayscale failure result can be submitted.

[0033] Optionally, in step S8, the end time of the grayscale task can be automatically extended based on the recorded number of rollback triggers.

[0034] Secondly, this invention provides a microservice adaptive canary deployment system, the system comprising: a node monitoring module, a traffic monitoring module, a canary task management module, and a node management module, wherein:

[0035] The node monitoring module is used to collect the operation index information of each microservice node, and is also responsible for calculating the node operation status score based on the node status scoring model.

[0036] The traffic monitoring module is used to collect the number of requests from each microservice node and is also responsible for calculating the node traffic saturation based on the node traffic saturation model.

[0037] The gray-scale task management module is used to publish gray-scale tasks, control the termination of gray-scale tasks, maintain the execution progress information of gray-scale tasks, and control the rollback of gray-scale tasks.

[0038] The node management module is responsible for performing specific canary replacements on each microservice node and for rolling back canary tasks.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] 1) Adaptability: The node status scoring model enables this method to effectively handle abnormal situations in the replacement service during the gray-scale process. The gradual gray-scale process and the task rollback process do not require manual handling, and it has strong adaptability.

[0041] 2) Real-time performance: Since the gray-scale process collects and monitors the operation of each microservice node in real time, there is no need for manual testing and feedback. When a node fails, the service cluster can be adjusted in a timely manner to reduce the impact on users.

[0042] 3) Stability: This method selects appropriate nodes for replacement or rollback based on a node traffic scoring model, which can minimize the impact on cluster load. In addition, this method adopts a gradual replacement and rollback approach, which can smooth service replacement operations during the canary release process and avoid the impact of a large number of service node adjustments on cluster operation. Attached Figure Description

[0043] Figure 1 This is a flowchart of the steps of a microservice adaptive canary release method according to the present invention.

[0044] Figure 2 This is a module structure diagram of a microservice adaptive canary release system according to the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0046] This invention provides a microservice adaptive canary deployment method, such as... Figure 1 As shown, it includes the following steps:

[0047] 1) Build a new version of the microservice application for release, submit the canary release task, and set the canary release task parameters.

[0048] In one embodiment of the present invention, the grayscale task parameters mainly include: expected service response time R. e Maximum expected request volume F per service unit time e Number of grayscale release nodes, step size n, grayscale replacement period T r The expected duration of the gray-scale task is T.

[0049] 2) Collect real-time operational metrics of each microservice node in the microservice cluster, calculate node operational status scores based on the node status scoring model, and calculate node traffic saturation based on the node traffic saturation model.

[0050] In one embodiment of the present invention, the operational metrics of each microservice node include:

[0051] T r The time consumed by 90% of the microservice quantiles is denoted as R. 90％ ;

[0052] T r Internal microservice call error rate, denoted as: Where, N error For T r The number of abnormal responses within, N total For T r Total number of requests within.

[0053] In one embodiment of the present invention, the method for calculating the node operating status score based on the node status scoring model is as follows:

[0054] The operational status score of a microservice node is recorded as follows: Wherein, α is usually 0 to 1, and in this embodiment it is 0.4; β is usually 0 to 1, and in this embodiment it is 0.6.

[0055] In one embodiment of the present invention, the method for calculating node traffic saturation based on the node traffic saturation model is as follows:

[0056] T r The average number of requests per unit time for an internal microservice is denoted as: Where f is T r Total number of requests within the microservice;

[0057] The traffic saturation measure for microservice nodes is:

[0058] 3) Based on the node running status score and node traffic saturation, periodically select an appropriate number of microservice nodes for canary replacement deployment. After each canary replacement is completed, roll back nodes with abnormal status according to the rolling back strategy based on the running status score of the canary nodes. Determine the status of the current canary task based on the expected end time of the canary release and the number of remaining nodes that have not undergone canary replacement.

[0059] In one embodiment of the present invention, the method of periodically selecting an appropriate number of nodes for grayscale replacement deployment, and after each grayscale replacement, rolling back nodes with abnormal status according to a rollback strategy based on the grayscale node running status score, is as follows:

[0060] Step S1: Select nodes with a flow saturation D less than 1 from the set of nodes to be replaced Z, and sort the selected nodes in ascending order of flow saturation to obtain the candidate node set S;

[0061] Step S2: If If so, proceed to step S3; otherwise, proceed to step S6.

[0062] Step S3: If |S|≥n, then for the node set S consisting of the n nodes with the smallest flow saturation in S... n Perform a replacement deployment (the replacement action means deleting the microservice on that node and deploying the new version of the microservice), Z = ZS n Then proceed to step S5; otherwise, proceed to step S4.

[0063] Step S4: Perform grayscale replacement on all nodes in S, Z = ZS;

[0064] Step S5: At interval T r Then, for the node set after grayscale replacement, check T. r The system scores the running status of nodes within the system. If any node is in an abnormal state, it rolls back the node and executes step S5 (i.e., S5 itself is an iterative function that continuously increments by a time interval T). r Execute until there are no abnormal nodes in the state, then exit the loop; otherwise, execute step S6.

[0065] Step S6: If If so, proceed to step S7; otherwise, proceed to step S9.

[0066] Step S7: If the current time is less than the expected duration T of the grayscale task, execute step S1; otherwise, execute step S8.

[0067] Step S8: If the current time is greater than or equal to the expected duration T of the canary deployment task, end the current canary deployment task, roll back all replaced nodes, and submit the canary deployment failure result.

[0068] Step S9: If the current time is less than the expected duration T of the grayscale task, proceed to step S5; otherwise, proceed to step S10.

[0069] Step S10: If the current time is greater than or equal to the expected duration T of the grayscale task, end the current grayscale task and submit the successful grayscale release result.

[0070] In one embodiment of the present invention, the method for selecting abnormal nodes in step S5 is as follows:

[0071] Two-stage thresholds p (0.6 in this embodiment) and q (1 in this embodiment) are preset. If the maximum node running status score is greater than p and less than q, the node with the largest running status score is selected as the abnormal node. If the maximum node running status score is greater than q, the node with the running status score greater than q is selected as the abnormal node.

[0072] In one embodiment of the present invention, in step S5, the number of rollback triggers of the grayscale task can be recorded. After the grayscale task reaches a certain number of rollback triggers, the current grayscale task can be terminated in advance and the grayscale failure result can be submitted.

[0073] In one embodiment of the present invention, in step S8, the end time of the grayscale task can be automatically extended according to the recorded number of rollback triggers.

[0074] In one embodiment of the present invention, a microservice adaptive canary release system is provided, such as... Figure 2As shown, the system includes: a node monitoring module, a traffic monitoring module, a gray-scale task management module, and a node management module, wherein:

[0075] The node monitoring module is used to collect the operation index information of each microservice node, and is also responsible for calculating the node operation status score based on the node status scoring model.

[0076] The traffic monitoring module is used to collect the number of requests from each microservice node and is also responsible for calculating the node traffic saturation based on the node traffic saturation model.

[0077] The gray-scale task management module is used to publish gray-scale tasks, control the termination of gray-scale tasks, maintain the execution progress information of gray-scale tasks, and control the rollback of gray-scale tasks.

[0078] The node management module is responsible for performing specific canary replacements on each microservice node and for rolling back canary tasks.

[0079] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0080] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0081] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0082] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A microservice adaptive canary deployment method, characterized in that, Includes the following steps: Build a new version of the microservice application for release, submit the canary release task, and set the canary release task parameters; Real-time collection of operational metrics for each microservice node in the microservice cluster; calculation of node operational status score based on node status scoring model; calculation of node traffic saturation based on node traffic saturation model. Based on the node running status score and node traffic saturation, a suitable number of microservice nodes are periodically selected for canary deployment. After each canary deployment, nodes with abnormal status are rolled back according to the rolling back strategy based on the running status score of the canary nodes. The status of the current canary task is determined based on the expected end time of the canary release and the number of remaining nodes that have not been canary deployed. The parameters for the grayscale task include: expected service response time. R e Maximum expected number of requests per service unit time F e Gray-scale release node step size n Grayscale replacement cycle T r Expected duration of gray-scale task T ; The operational metrics for each microservice node include: T r The time consumed by 90% of the microservice quantiles is denoted as R 90% ; T r Internal microservice call error rate, denoted as ,in, for T r The number of abnormal responses within, for T r Total number of requests within; The node status scoring model uses the following formula to calculate the running status score of a microservice node: ， in, It is 0~1. It is 0~1; The calculation of node traffic saturation based on the node traffic saturation model includes: First calculate T r Average number of requests per unit time for an internal microservice: ,in, for T r Total number of requests within the microservice; Then calculate the traffic saturation of the microservice nodes: ; The process involves periodically selecting an appropriate number of microservice nodes for canary deployment. After each canary deployment, based on the operational status score of the canary nodes, nodes with abnormal status are rolled back according to a rollback strategy, including: Step S1: From the set of nodes to be replaced Z Select flow saturation D For nodes with a saturation level less than 1, sort the selected nodes in ascending order of saturation level to obtain a candidate node set. S ; Step S2: If If yes, proceed to step S3; otherwise, proceed to step S6. Step S3: If Then for S The front with the lowest flow saturation n A node set consisting of nodes S n Perform a replacement deployment. Then proceed to step S5; otherwise, proceed to step S4. Step S4: For S All nodes in the array are replaced with grayscale values. ; Step S5: At the interval time T r Then, examine the node set after grayscale replacement. T r The node running status is scored. If there are nodes with abnormal status, the nodes with abnormal status are rolled back; otherwise, step S6 is executed. Step S6: If If yes, proceed to step S7; otherwise, proceed to step S9. Step S7: If the current time is less than the expected duration of the gray-scale task T If the above steps are not executed, proceed to step S1; otherwise, proceed to step S8. Step S8: If the current time is greater than or equal to the expected duration of the gray-scale task T End the current canary deployment task, roll back all replaced nodes, and submit the canary deployment failure result; Step S9: If the current time is less than the expected duration of the gray-scale task T If the above steps are not executed, proceed to step S5; otherwise, proceed to step S10. Step S10: If the current time is greater than or equal to the expected duration of the grayscale task T End the current grayscale task and submit the successful grayscale release result.

2. The method according to claim 1, characterized in that, The method for selecting the abnormal state nodes is as follows: Pre-set two-stage thresholds p and q If the maximum node running status score is greater than p and less than q Then the node with the highest running status score is selected as the node with the abnormal status. If the running status score of the node with the highest score is greater than 1, then the node with the highest score is selected as the node with the abnormal status. q Then select the running status score greater than q The node is considered an abnormal node.

3. The method according to claim 1, characterized in that, In step S5, the number of rollback triggers for the grayscale task is recorded. After the grayscale task reaches a certain number of rollback triggers, the current grayscale task is terminated early and the grayscale failure result is submitted. In step S8, the end time of the grayscale task is automatically extended according to the recorded number of rollback triggers.

4. A microservice adaptive canary release system, characterized in that, The system for performing the method according to any one of claims 1 to 3 comprises: Node monitoring module: Used to collect operational metrics information of each microservice node, and is also responsible for calculating the node operational status score based on the node status scoring model; Traffic monitoring module: used to collect the number of requests from each microservice node, and also responsible for calculating the node traffic saturation based on the node traffic saturation model; The gray-scale task management module is used to publish gray-scale tasks, control the termination of gray-scale tasks, maintain the execution progress information of gray-scale tasks, and control the rollback of gray-scale tasks. Node Management Module: Used to perform specific canary deployment replacements on each microservice node, and also responsible for the rollback operation of canary deployment tasks.

5. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 3.

Images