Full-link gray release method based on deep reinforcement learning

Abstract

The invention discloses a full-link gray release method based on deep reinforcement learning, and the method comprises the following steps: inputting the current operation state mt of each service into a decision model of a corresponding service, and making a decision of a gray adjustment action at of the maximum income of the corresponding service through the decision model; and the evaluation model monitors the service operation state mt + 1 after the gray adjustment action is executed, and neural network parameters of the evaluation model and the decision model are updated based on the income corresponding to the service operation state mt + 1. The technical problem that in the prior art, the requirement for differential batch processing of the full-link gray release task cannot be met is solved, manual participation and professional knowledge are not needed, the full-link gray release action can be adaptively and dynamically adjusted according to automatic evaluation of the gray release task running state in the running process, and the full-link gray release task operation efficiency is improved. Therefore, the execution efficiency and reliability of the batch processing full-link gray level release task are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and more specifically, the present invention relates to a full-link grayscale publishing method based on deep reinforcement learning. Background Art

[0002] In the era of Internet finance, financial products and service models are constantly innovating, and the demand for financial service application software is growing rapidly, with changes accelerating and versions iterating rapidly. This has led to the increasing complexity of the entire system. On the other hand, the system development time has been repeatedly compressed, and product quality and user experience are becoming increasingly difficult to guarantee. Building grayscale release capabilities is one way to resolve this increasingly intensified contradiction.

[0003] Grayscale release is a technology used in software development and deployment. It can gradually push new software versions to a small number of users for testing and evaluation without affecting normal users. Grayscale release can be used to discover and solve problems before the new version of software is launched on a large scale, so as to reduce online failures and improve software quality and stability. First, unknown errors, performance problems or incompatibilities may occur when a new version of the software system is released. Grayscale release only affects a small number of users, reducing the impact of the problem on the entire system. Secondly, the gradual introduction of new versions in grayscale release allows the development team to have time to observe the behavior of the system at each stage and make adjustments quickly when problems arise, thereby improving overall stability and reliability. Thirdly, the grayscale release capability can help the development team better understand the user's reaction to new features by gradually rolling out new features or changes, giving users time to adapt and provide feedback, so as to make improvements based on feedback. Finally, if serious problems are found during the grayscale release process, it is relatively easy to roll back to the old version to avoid affecting most users, and emergency fixes can be implemented during the release process without interrupting the entire system.

[0004] As the scale and complexity of software systems continue to expand, full-link grayscale releases have gradually attracted the attention of the industry. Full-link grayscale releases not only focus on the grayscale release of a single application, but also take all components and services in the entire system link into consideration, thereby ensuring that the entire system link of the new version of the application, including multiple levels of services, is gradually verified and deployed. The batch application grayscale release task is to automatically execute the grayscale release task of multiple applications by software without human intervention. However, since the application software system that executes batch application releases is composed of many services, and the calling relationships between these services are complex and automatically executed, full-link grayscale releases face new challenges.

[0005] Full-link grayscale release requires the deployment of multiple versions of services, real-time monitoring and evaluation of each service, and the decision of whether to adjust its grayscale ratio accurately and timely based on the analysis results of the monitoring data. Currently, the full-link grayscale release technology mainly uses the method of manually adjusting the grayscale ratio. When the operator finds that the new version of the service performs well, the grayscale ratio can be gradually increased to eventually achieve the new version of the service completely replacing the existing version of the service. On the contrary, the grayscale ratio can be reduced or the grayscale release can be suspended through a rollback operation. For batch application release tasks, the full-link grayscale release method that uses manual adjustment of the grayscale ratio has the following problems:

[0006] Operators need to analyze the operating status of many services that perform different tasks in real time, which requires in-depth and extensive professional knowledge to detect and diagnose root causes of faults. This places high demands on operators and makes it easy to miss faults. Summary of the invention

[0007] The present invention provides a full-link grayscale publishing method based on deep reinforcement learning, aiming to improve the above problems.

[0008] The present invention is implemented as follows: a full-link grayscale publishing method based on deep reinforcement learning, characterized in that the method comprises the following steps:

[0009] The current running status of each service t Input the decision model of the corresponding service, and the grid decision model determines the gray adjustment action a with the maximum benefit of the corresponding service t ;

[0010] The evaluation model monitors the service operation status m after the grayscale adjustment action is executed t+1 , based on the service running status m t+1 The corresponding benefits update the neural network parameters of the evaluation model and decision model.

[0011] Furthermore, the i-th service performs the grayscale adjustment action in the t-th grayscale adjustment cycle The benefits obtained The calculation formula is as follows:

[0012]

[0013] in, Cost benefits of reduced resource usage, For performance gains, is the benefit of improved reliability, w per 、w rsc and w reb Respectively represent income income and benefits The weight of .

[0014] Furthermore, resource indicators include computing resources required for request processing rc1, cache resources required for request processing rc2, persistence resources required for request processing rc3, and network resource usage required for request processing rc4. The calculation formula is as follows:

[0015]

[0016] Among them, p j represents the unit benefit obtained by the unit monitoring indicator j, Indicates that the i-th service performs grayscale adjustment action in the t-th grayscale adjustment cycle The change rate of the latter monitoring indicator j,

[0017] Furthermore, performance indicators include the time required to process a request rc5 and the number of requests processed per unit time rc6. The benefits of performance improvement

[0018] in, Indicates the time required to process the request and the revenue generated by rc5. Represents the revenue generated by the service unit time processing request number rc6,

[0019] Furthermore, reliability indicators include task failure rate rc7 and service availability rate rc8. The benefits of reliability improvement

[0020] in, Indicates the revenue generated by the failure rate rc7 of task execution, represents the revenue generated by the service availability rate rc8,

[0021] Furthermore, the calculation formula for the change rate of the monitoring index j is as follows:

[0022]

[0023] in, The monitoring indicator j of the i-th service is respectively in the grayscale adjustment action Before and after indicator values.

[0024] Furthermore, each service corresponds to a decision model, which is:

[0025]

[0026] Among them, μ(rc j ) The jth monitoring indicator rc jThe mean value, σ(rc j ) is the jth monitoring index rc j is the standard deviation of , and β is the model coefficient.

[0027] Set the current running state of the service m = {rc1, rc2, ..., rc j ,...,rc8} input the corresponding decision model, and the decision model decides the grayscale adjustment action a of the corresponding service.

[0028] Furthermore, a corresponding evaluation model is created for each service's decision model to evaluate the pros and cons of the corresponding decision model;

[0029] The running status of the new and old instances of n services that will constitute the grayscale release {m 1 ,m 2 ,...,m j ,...,m n}, corresponding grayscale adjustment action {a 1 ,a 2 ,...,a i ,...,a n}, and the benefit value generated by the corresponding action {r 1 ,r 2 ,...,r i ,...,r n} are constructed into samples and used to train evaluation models for n services.

[0030] Furthermore, the deep neural network parameters of the decision model are optimized and adjusted based on the output of the evaluation model. The update process of the decision model and the evaluation model is as follows:

[0031] 1) Monitor the current operating status m t ;

[0032] 2) Randomly select grayscale adjustment action a according to the decision model t , grayscale adjustment action a t Execute and switch to new running state m t+1 ;

[0033] 3) Calculate the running state m t+1 The profit under t+1 , according to the probability model a t ~π(·|m t θ t ) Randomly generate action a by sampling t , and obtain the state transition sequence (m t ,a t ,r t+1 ,m t+1 );

[0034] 4) According to the state transition sequence (m t ,a t ,r t+1 ,m t+1 ), use the deep deterministic policy gradient algorithm to update the decision model and evaluation model.

[0035] Aiming at publishing tasks to diverse and complex batch processing applications, the present invention proposes a dynamic adaptive batch processing full-link grayscale publishing method based on deep reinforcement learning, which improves the technical problem that the existing technology cannot meet the needs of differentiated batch processing full-link grayscale publishing tasks, and achieves the goal of not requiring human participation and professional knowledge. It can adaptively and dynamically adjust the full-link grayscale publishing action according to the automatic evaluation of the running status of the grayscale publishing task at runtime, thereby improving the execution efficiency and reliability of the batch processing full-link grayscale publishing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of a full-link grayscale publishing framework based on deep reinforcement learning provided by an embodiment of the present invention;

[0037] Figure 2 A flowchart of a full-link grayscale publishing method based on deep reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The specific implementation modes of the present invention are further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0039] Figure 1 A schematic diagram of the structure of a full-link grayscale publishing framework based on deep reinforcement learning provided by an embodiment of the present invention, Figure 2 A flowchart of a full-link grayscale publishing method based on deep reinforcement learning provided by an embodiment of the present invention, combined with Figure 1 and Figure 2 To illustrate the full-link grayscale release process based on deep reinforcement learning, the framework includes:

[0040] Service request grayscale transparent transmitter: Store the grayscale label in the context object that needs to process the service request, mark the grayscale label when sending the request, and parse the grayscale label when receiving the request.

[0041] Service request router: (1) Configure routing rules during initialization to associate grayscale service request traffic with grayscale version services; (2) During operation, when receiving a user's service request, check the specific X-Gray-Tag field in the request header, detect and mark it as grayscale traffic.

[0042] Service instance controller: When creating an instance for a service, it identifies the service instance version to which the grayscale release task is applied, so that when receiving a request, it can identify whether the current instance is a grayscale version or a formal version.

[0043] Grayscale release monitor: Use monitoring tools Prometheus and Istio to monitor the system's monitoring indicators.

[0044] When updating an application from the current version (also called the old version) to the new version, it is necessary to determine the grayscale release ratio of the new version of the service instance with the best benefits. The service is the basic unit of application composition and release. Each service has multiple instances. During the application release process, the grayscale release ratio of each service new version instance changes dynamically. The running status of the service instances during the grayscale release process is collected at a fixed time period. The grayscale adjustment decision model incrementally fine-tunes the grayscale release ratio of each service online according to the current running status. If the service performance of the application under the grayscale release ratio after fine-tuning is better than that before fine-tuning, the grayscale release ratio after fine-tuning is used as the release ratio of the application. Otherwise, the grayscale service instance ratio is reduced or the grayscale release process is suspended through a rollback operation.

[0045] Apply the grayscale status s of each service in the tth grayscale adjustment cycle t It is expressed as: Where n is the number of services in the application, Indicates the grayscale release ratio of the new version instance of the i-th service in the application in the t-th grayscale adjustment cycle.

[0046] The running status of the i-th service in the t-th grayscale adjustment cycle of the application It is expressed as: Among them, d is the number of monitoring indicators of the service, rc j is the index value of the j-th monitoring index of the service. The monitoring indicators are divided into three categories: resource indicators, performance indicators and reliability indicators. Among them, the resource indicators include the computing resources required for request processing rc1, the cache resources required for request processing rc2, the persistent resources required for request processing rc3, and the network resource usage required for request processing rc4; the performance indicators include: the time required to process the request rc5, and the number of requests processed per service unit time rc6; the reliability indicators include: the task failure rate rc7, and the service availability rate rc8, so d=8.

[0047] The running status of the i-th service monitored in the t-th grayscale adjustment period Running status When performing grayscale adjustment After that, the grayscale state of the i-th service Switch to grayscale The running status of the i-th service Switch to running state Grayscale Adjusted Returns Depends on the dynamically changing service operation status, taking into account service performance indicators, service resource indicators and service reliability indicators. In an application consisting of n services, each service performs grayscale adjustment actions in grayscale adjustment cycle t, and the i-th service performs grayscale adjustment actions in the t-th grayscale adjustment cycle The resulting revenue is expressed as The benefit r obtained by applying the grayscale adjustment action in the tth grayscale adjustment cycle t Expressed as

[0048] The i-th service performs the grayscale adjustment action in the t-th grayscale adjustment cycle The benefits obtained Cost benefits from reduced resource usage Performance benefits Benefits of improved reliability It consists of three parts, namely Among them, w per 、w rsc and w reb Respectively represent income income and benefits The weight of , while satisfying: w per +w rsc +w reb =1 and 0≤w per , w rsc , w reb ≤1. The change of each monitoring indicator after the grayscale adjustment action is performed in the tth grayscale adjustment cycle is applied. The benefit value is calculated based on the change of each monitoring indicator. The change rate of monitoring indicator j is:

[0049] in,

[0050] The monitoring indicator j of the i-th service is respectively in the grayscale adjustment action Before and after indicator values.

[0051] In terms of resource usage, if the grayscale release of a new version of the service instance uses less computing resources, cache resources, persistent storage resources, and network bandwidth resources than the original version of the service instance, the new version of the service instance released reduces the cost of resource investment, and thus executing the corresponding grayscale adjustment action can increase the benefits. The cost benefits of reduced resource usage obtained by the grayscale adjustment action of the service It is expressed as: Among them, p jRepresents the unit benefit obtained by unit monitoring indicator j, p1=0.95, p2=0.80, p3=0.75, p4=0.70.

[0052] In terms of performance, the lower the time required to process a request rc5 and the higher the number of requests processed per unit time rc6, the greater the benefit of releasing a new version of the service instance. The benefit brought by the time required to process a request rc5 is Indicates the number of requests processed per unit time by the service rc6. The performance improvement benefits

[0053] In terms of reliability, the lower the task execution failure rate rc7 and the higher the service availability rate rc8, the greater the benefit of releasing a new version of the service instance. Indicates that the benefits brought by service availability rc8 are The benefits of improved reliability

[0054] The i-th service performs the grayscale adjustment action in the t-th grayscale adjustment cycle The benefits obtained are: income ~Benefits The calculation formula is: Among them, w per =0.25, w rsc =0.35,w reb =0.40, p5=0.65, p6=0.70, p7=0.90, p8=0.85.

[0055] Monitor the running status of the grayscale release process service, and the decision model decides to execute the grayscale adjustment action of the service, so as to maximize the benefits of the application executing the grayscale adjustment action. The application is composed of many distributed services, and each service uses its own service instance controller to execute the grayscale adjustment action, that is, to change the number of new version service instances. The full-link grayscale release strategy adjustment is completed by two neural network models, namely the decision model and the evaluation model. Among them, the decision model is used to make decisions on grayscale adjustment actions, and the evaluation model is used to evaluate the advantages and disadvantages of the grayscale adjustment strategy, and dynamically update the parameters of the decision model according to the evaluation results.

[0056] Since the service types in the application are different, the present invention constructs a decision model for each service, which is expressed as a function form: π(a|m; θ), where θ is the network parameter of the decision model, a is the decision action, and the current local observation (operating state) of the corresponding service m={rc1, rc2, ..., rc j,...,rc8} input the corresponding decision model, and the decision model decides the grayscale adjustment action of the corresponding service.

[0057] The decision model is used to make decisions on the grayscale adjustment of services, and is modeled using a deep neural network. Assuming that the grayscale adjustment ratio values ​​of all services in the application conform to the normal distribution, with a mean of μ and a standard deviation of σ, the grayscale adjustment decision model is: β=6.28; Different services build the same convolutional layer structure, and obtain the multi-dimensional vector of the output through forward propagation. The last layer uses a fully connected layer to connect and process multiple dimensions of the vector. |m is the service operation status of the input decision model. The training data of the grayscale adjustment decision model is generated based on the Monte Carlo method, and random sampling is used to generate actions. The action observation monitoring data is then used to train the grayscale adjustment decision model.

[0058] Each service decision model creates a corresponding evaluation model to evaluate the pros and cons of the corresponding decision model. Based on the output of the evaluation model, the deep neural network parameters of the decision model are optimized and adjusted to improve the decision effect of each decision model. The grayscale release monitor monitors the running status of the new and old instances of the n services that constitute the grayscale release. 1 ,m 2 ,...,m j ,...,m n}, corresponding grayscale adjustment action {a 1 ,a 2 ,...,a i ,...,a n}, and the benefit value generated by the corresponding action {r 1 ,r 2 ,...,r i ,...,r n}, which are used to train evaluation models for n services respectively. The network structures of the evaluation models are the same, but the parameters w vary due to differences in service characteristics. The evaluation model of the i-th service is represented by r i =v(m i ;w i ), w i Parameters of the evaluation model corresponding to the nth service.

[0059] In the embodiment of the present invention, the following steps 1) to 4) are repeatedly performed to update the decision model and the evaluation model. The update process is as follows:

[0060] 1) Monitor the current operating status m t ;

[0061] 2) Randomly select grayscale adjustment action a according to the decision model t , grayscale adjustment action at Execute and switch to new running state m t+1 ;

[0062] 3) Calculate the running state m t+1 The profit under t+1 , according to the probability model a t ~π(·|m t θ t ) Randomly generate action a by sampling t , and obtain the state transition sequence (m t ,a t ,r t+1 ,m t+1 );

[0063] 4) According to the above state transition sequence, the Deep Deterministic Policy Gradient (DDPG) algorithm is used to update the decision model and the evaluation model. The decision model corresponds to the Actor network in DDPG, and the evaluation model corresponds to the Critic network in DDPG.

[0064] When a specific application grayscale release task is running, the grayscale adjustment behavior of service i is expressed as: Where n is the number of services in the application. After the decision model and evaluation model are trained, each service instance controller uses the decision model to randomly sample the execution grayscale adjustment action a according to the probability distribution. i ~π(·|m i θ i ), The grayscale adjustment action of service i in the tth grayscale adjustment cycle The number of instances participating in the grayscale release is increased (positive number) or decreased (negative number) accordingly. While increasing the number of grayscale version service instances, the same number of original service version instances are deleted until all original service version instances are replaced, and the grayscale release process of the service is completed.

[0065] The present invention has been described exemplarily. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A full-link grayscale release method based on deep reinforcement learning, characterized in that: The method comprises the following steps: The current running status of each service t Input the decision model of the corresponding service, and the grid decision model determines the gray adjustment action a with the maximum benefit of the corresponding service t ; The evaluation model monitors the service operation status m after the grayscale adjustment action is executed t+1 , based on the service running status m t+1 The corresponding benefits update the neural network parameters of the evaluation model and decision model.

2. The full-link grayscale publishing method based on deep reinforcement learning as claimed in claim 1, characterized in that: The i-th service performs the grayscale adjustment action in the t-th grayscale adjustment cycle The income obtained t i The calculation formula is as follows: in, Cost benefits of reduced resource usage, For performance gains, is the benefit of improved reliability, w per 、w rsc and w reb Respectively represent income income and benefits The weight of .

3. The full-link grayscale publishing method based on deep reinforcement learning as claimed in claim 2, characterized in that: Resource indicators include the computing resources required for request processing (rc1), the cache resources required for request processing (rc2), the persistence resources required for request processing (rc3), and the network resource usage required for request processing (rc4). The calculation formula is as follows: Among them, p j represents the unit benefit obtained by the unit monitoring indicator j, Indicates that the i-th service performs grayscale adjustment action in the t-th grayscale adjustment cycle Then monitor the rate of change of indicator j.

4. The full-link grayscale publishing method based on deep reinforcement learning as claimed in claim 2, characterized in that: Performance indicators include the time required to process a request (rc5) and the number of requests processed per unit of service time (rc6). The benefits of performance improvement in, Indicates the time required to process the request and the revenue generated by rc5. Represents the revenue generated by the service unit time processing request number rc6, 5. The full-link grayscale publishing method based on deep reinforcement learning as claimed in claim 2, characterized in that: Reliability indicators include task failure rate rc7 and service availability rate rc8. The benefits of reliability improvement in, Indicates the revenue generated by the failure rate rc7 of task execution, Represents the revenue generated by the service availability rate rc8, 6. The full-link grayscale release method based on deep reinforcement learning as described in any one of claims 3 to 5, characterized in that: The calculation formula for the change rate of monitoring index j is as follows: in, The monitoring indicator j of the i-th service is respectively in the grayscale adjustment action Before and after indicator values.

7. The full-link grayscale publishing method based on deep reinforcement learning as claimed in claim 1, characterized in that: Each service corresponds to a decision model, which is: Among them, μ(rc j ) The jth monitoring indicator rc j The mean value, σ(rc j ) is the jth monitoring index rc j is the standard deviation of , and β is the model coefficient. Set the current running state of the service m = {rc1, rc2, ..., rc j ,...,rc8} input the corresponding decision model, and the decision model decides the grayscale adjustment action a of the corresponding service.

8. The full-link grayscale publishing method based on deep reinforcement learning as claimed in claim 1, characterized in that: Create a corresponding evaluation model for each service's decision model to evaluate the pros and cons of the corresponding decision model; The running status of the new and old instances of n services that will constitute the grayscale release {m 1 ,m 2 ,...,m j ,...,m n }, corresponding grayscale adjustment action {a 1 ,a 2 ,...,a i ,...,a n }, and the benefit value generated by the corresponding action {r 1 ,r 2 ,...,r i ,...,r n } are constructed into samples and used to train evaluation models for n services respectively.

9. The full-link grayscale publishing method based on deep reinforcement learning as claimed in claim 1, characterized in that: The update process of the decision model and the evaluation model is as follows: 1) Monitor the current operating status m t ; 2) Randomly select grayscale adjustment action a according to the decision model t , grayscale adjustment action a t Execute and switch to new running state m t+1 ; 3) Calculate the running state m t+1 The profit under t+1 , according to the probability model a t ~π(·|m t θ t ) Randomly generate action a by sampling t , and obtain the state transition sequence (m t ,a t ,r t+1 ,m t+1 ); 4) According to the state transition sequence (m t ,a t ,r t+1 ,m t+1 ), use the deep deterministic policy gradient algorithm to update the decision model and evaluation model.

Images