Devops-based pipeline optimization method, device and medium
By acquiring historical data from the pipeline, Markov and Hidden Markov models are established to optimize the workload of pipeline nodes. This solves the problem that traditional pipelines cannot accurately assess the workload of nodes, and achieves pipeline path optimization and project management efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GHOSTCLOUD
- Filing Date
- 2022-08-09
- Publication Date
- 2026-05-15
AI Technical Summary
Traditional pipeline construction methods cannot accurately determine the workload of specific nodes in the pipeline, making it impossible to select the optimal path and thus slowing down the development process.
By acquiring historical data from the pipeline, determining node information and influencing factors, establishing a Markov model, and combining greedy algorithms and dynamic programming, a hidden Markov model is constructed to obtain the state set and time distribution of each node, thereby optimizing the workload of pipeline nodes.
It enables accurate assessment of workload at nodes in the pipeline, optimizes pipeline paths, reduces waste of human resources and project costs, and improves the efficiency and accuracy of project management.
Smart Images
Figure CN115357250B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of DevOps pipeline technology, specifically to a DevOps-based pipeline optimization method, device, and medium. Background Technology
[0002] DevOps is a set of processes, methodologies, and systems used to facilitate communication, collaboration, and integration among development (application / software engineering), technical operations, and quality assurance departments. It aims to shorten the time from system initiation to release to production while ensuring high software quality. Continuous integration and continuous delivery are the most crucial aspects of DevOps, and pipelines are the core of automated continuous delivery. In DevOps software, a pipeline is a set of automated processes that enable DevOps professionals and developers to reliably and efficiently compile, build, and deploy their code to production computing platforms.
[0003] The development process of a project is divided into stages such as software planning, requirements analysis, overall software design, detailed software design, software code development, software testing, and software delivery. Currently, traditional pipelines can only execute step-by-step. As projects grow larger and the connections between projects deepen, multiple instances of the same node exist within the pipeline, the number of state types between nodes increases, and the overall operation time requirements of the pipeline for each project further increase. Traditional pipeline construction methods cannot meet the needs of large-scale project scenarios, cannot accurately determine the workload of specific nodes in the pipeline, and cannot select the optimal path, leading to a slowdown in the development process. Summary of the Invention
[0004] The technical problem this application aims to solve is the inability to accurately determine the workload of specific nodes in a pipeline. The purpose is to provide a DevOps-based pipeline optimization method, device, and medium. By determining the pipeline complexity through pipeline node information and influencing factors, a Markov model is established to obtain the state set generated by each node and determine the time distribution of the target node, thereby obtaining the workload of specific nodes in the pipeline.
[0005] This application is achieved through the following technical solution:
[0006] The first aspect of this application provides a DevOps-based pipeline optimization method, including the following specific steps:
[0007] S1. Obtain historical data of the pipeline, get node information and impact factors, and determine the total weight of the nodes and the weight of the impact factors;
[0008] S2. Determine the pipeline complexity based on the total node weight and the influence factor weight;
[0009] S3. Based on the pipeline complexity and the greedy algorithm, establish a Markov model and obtain the state set generated by each node.
[0010] S4. Based on the state set generated by each node, a hidden Markov model is established using dynamic programming of the Markov model to determine the time distribution of the target node and obtain the workload of the target node.
[0011] This application identifies node information and influencing factors by acquiring historical data, determines pipeline complexity based on total node weights and influencing factor weights, establishes a Markov model based on pipeline complexity and a greedy algorithm, establishes a Hidden Markov Model based on dynamic programming of the Markov model, obtains the state set generated by each node, and determines the time distribution of the target node, thereby realizing the construction of an optimal pipeline based on a combination of dynamic programming and a greedy algorithm.
[0012] Furthermore, determining the total weight of the nodes includes:
[0013] Obtain node information and node weights;
[0014] Based on the obtained node information, segmented data is obtained, and segment weights are calculated based on the segmented data and the weights of the nodes corresponding to each segment.
[0015] The total weight of a node is determined based on its node weight and segment weight.
[0016] Furthermore, the impact factor weights include a first impact factor weight and a second impact factor weight, and the steps for obtaining the first impact factor weight and the second impact factor weight include:
[0017] Obtain the impact factor, divide the impact factor into the first impact factor and the second impact factor, determine the weight and level of the first impact factor and the second impact factor in each node, and obtain the weight value of the first impact factor and the weight value of the second impact factor based on the weight and level of the first impact factor and the second impact factor in each node.
[0018] Furthermore, the formula for calculating the weight of the first influence factor is as follows:
[0019]
[0020] The formula for calculating the weight of the second impact factor is:
[0021]
[0022] Among them, Grade represents the level, TCF represents the weight of the first impact factor, and ECF represents the weight of the second impact factor.
[0023] Furthermore, the pipeline complexity determination step specifically includes:
[0024]
[0025] Where PC represents pipeline complexity, TCF represents the weight of the first influence factor, ECF represents the weight of the second influence factor, UUCP represents the total weight of the nodes, and PF represents the productivity of the pipeline.
[0026] Furthermore, the specific steps of establishing the Markov model based on pipeline complexity combined with a greedy algorithm include:
[0027] A greedy algorithm is introduced to obtain the transition probability of the Markov chain based on the pipeline complexity;
[0028] Obtain node information and the current node time, and based on the transition probabilities of the Markov chain, obtain the n-step transition probability matrix of the Markov chain.
[0029] Furthermore, the establishment of a hidden Markov model based on dynamic programming of the Markov model specifically includes:
[0030] A hidden Markov model is established based on the n-step transition probability matrix of the Markov chain;
[0031] Obtain the initial state chain, state transition probability matrix, observation probability matrix, and initial state probability vector;
[0032] Based on the initial state chain, the state transition probability matrix, the observation probability matrix, and the initial state probability vector, the observation sequence and the transition probabilities of the nodes are determined.
[0033] Furthermore, the determination of the time distribution of the target node's target state specifically includes:
[0034] The latent variables and conditional probability distributions of the nodes are determined based on the transition probabilities of the nodes, and the observed variables are obtained based on the observation sequence.
[0035] Obtain the joint probability distribution of the observed variables and the latent variables to obtain the time distribution of the target state of the target node.
[0036] A second aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a DevOps-based pipeline optimization method.
[0037] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement a DevOps-based pipeline optimization method.
[0038] Compared with the prior art, this application has the following advantages and beneficial effects:
[0039] 1. This application determines the pipeline complexity by using pipeline node information and influencing factors, establishes a Markov model, obtains the state set generated by each node, determines the time distribution of the target node, and thus obtains the workload of specific nodes in the pipeline.
[0040] 2. This application constructs a Markov model using a greedy algorithm, then analyzes the Markov model to obtain the state set generated by each node, and uses dynamic programming to establish a hidden Markov model. By adjusting different input conditions, a small section of the pipeline can be treated as a whole, and the time estimate required for all paths can be derived in reverse. Attached Figure Description
[0041] To more clearly illustrate the technical solutions of the exemplary embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:
[0042] Figure 1 This is a flowchart illustrating the pipeline construction process in the embodiments of this application;
[0043] Figure 2 This is a diagram illustrating the representation of potential variable transfer in the embodiments of this application. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this application are only for explaining this application and are not intended to limit this application.
[0045] Example 1
[0046] like Figure 1 As shown, the first aspect of this application provides a DevOps-based pipeline optimization method, including the following specific steps:
[0047] S1. Obtain historical data of the pipeline, get node information and impact factors, and determine the total weight of the nodes and the weight of the impact factors;
[0048] S2. Determine the pipeline complexity based on the total node weight and the influence factor weight;
[0049] S3. Based on the pipeline complexity and the greedy algorithm, establish a Markov model and obtain the state set generated by each node.
[0050] S4. Based on the state set generated by each node, a hidden Markov model is established using dynamic programming of the Markov model to determine the time distribution of the target node and obtain the workload of the target node.
[0051] By acquiring historical data, node information and influencing factors are identified. The pipeline complexity is determined based on the total node weight and the influencing factor weight. A Markov model is established based on the pipeline complexity and a greedy algorithm. A Hidden Markov Model is established based on the dynamic programming of the Markov model. The state set generated by each node is obtained, and the time distribution of the target node is determined. This achieves the construction of the optimal pipeline based on the combination of dynamic programming and greedy algorithm.
[0052] In some possible implementations, nodes and related segments are identified by combining existing pipelines with the current pipeline. The pipeline complexity is determined based on the total node weights and influencing factor weights. By calculating the node and productivity values, the workload required for each path in the pipeline, expressed in person-hours, is obtained. For example, the final workload estimation for a pipeline path mainly depends on influencing factors and total weights. Influencing factors are further divided into first and second influencing factors. To change the final estimated workload of a pipeline path, the nodes and segments, along with their weights, can be altered. Influencing factors can also be increased to change the workload assessment. A Markov model is constructed using a greedy algorithm. Based on the state set generated at each stage of the Markov model, a Hidden Markov Model (HMM) is established using dynamic programming. By adjusting different input conditions, a small segment of the pipeline is treated as a whole, and the time estimates required for all paths are derived in reverse, achieving optimal pipeline construction based on a combination of dynamic programming and greedy algorithms.
[0053] In some possible embodiments, determining the total weight of a node includes:
[0054] Obtain node information and node weights, as shown in Table 1.
[0055] Table 1 Node Weight Definition Table
[0056] Serial Number Complexity level Complexity Standard weight participate 1 primary Evenly received data volume 1 x 2 intermediate The amount of received data is relatively uniform. 4 y 3 advanced The amount of data received is not uniform 9 z
[0057] The complexity level represents the level of data received by the node. The complexity standard depends on the uniformity of the received file data volume. The weight represents the impact of the complexity on the pipeline. The participation is the number of concurrent connections associated with the node in actual development.
[0058] Based on the obtained node information, segmented data is obtained. The segment weights are calculated based on the segmented data and the weights of the nodes corresponding to each segment, as shown in Table 2.
[0059] Table 2 Use Case Weight Definition Table
[0060] Serial Number Complexity level Number of things / scenes weight Number 1 primary 1-3 5 x 2 intermediate 4-6 10 y 3 advanced 7-9 15 z
[0061] The complexity level represents the depth of the segment, the number of things / scenes represents the environment in which the pipeline is located, the weight represents the impact of the complexity on the pipeline, and the number belongs to the number of pipeline steps in the actual scenario.
[0062] The total weight of a node is determined based on its node weight and segment weight, i.e., the total weight of a node = the sum of its node weight and segment weight.
[0063] In some possible embodiments, the impact factor weights include a first impact factor weight and a second impact factor weight, and the steps for obtaining the first impact factor weight and the second impact factor weight include:
[0064] Obtain the impact factor, divide the impact factor into the first impact factor and the second impact factor, determine the weight and level of the first impact factor and the second impact factor in each node, and obtain the weight value of the first impact factor and the weight value of the second impact factor based on the weight and level of the first impact factor and the second impact factor in each node.
[0065] Each factor in the pipeline becomes an influencing factor in building the model. The first and second influencing factors can be specified into 23 influencing factors. These 23 influencing factors can be continuously adjusted in their final evaluation data by changing their weights and values.
[0066] In some possible embodiments,
[0067] The calculation of the weights of the first impact factor is shown in Table 3. The formula for calculating the weights of the first impact factor is as follows:
[0068]
[0069] Table 3 Weighting of Technology Impact Factors
[0070] Technical factors illustrate Weight Grade T1 System Scale 2 5 T2 System performance 1 4 T3 Terminal efficiency 1 2 T4 code complexity 1 4 T5 Reusability 1 2 T6 Throughput 2 2 T7 Ease of use 0.5 2 T8 acceleration ratio 2 3 T9 Maintainability 1 3 T10 Concurrency requirements 1 2 T11 Security requirements 1 2 T12 Provide third-party interfaces 1 5
[0071] The weight represents the impact of this complexity on the entire pipeline; Grade represents the impact level, with values ranging from 0 to 5, where 0 indicates that the impact factor is irrelevant to the pipeline, 1-3 indicates that the impact factor has a moderate impact on the pipeline, and 4-5 indicates that the impact factor has a significant impact on the pipeline.
[0072] In some possible embodiments, the weights of the second impact factor are calculated as shown in Table 4, and the formula for calculating the weights of the second impact factor is:
[0073]
[0074] Table 4 Weighting of Non-Technical Impact Factors
[0075] Technical factors illustrate Weight Grade T1 UML skills 1.5 3 T2 Application layer skills 0.5 3 T3 Development experience 1 3 T4 Project Leader Competence 0.5 5 T5 Company-related incentives 1 3 T6 Project Requirements Completeness 2 3 T7 Unstable personnel -1 0.5 T8 Choosing a programming language -1 0 T9 joint development -1.5 2 T10 Special factors -2 4 T11 Company Developer Structure 0.5 1
[0076] The weight represents the impact of the complexity on the pipeline; Grade represents the impact level, with values ranging from 0 to 5, where 0 indicates that the impact factor is irrelevant to the pipeline, 1-3 indicates that the impact factor has a moderate impact on the pipeline, and 4-5 indicates that the impact factor has a significant impact on the pipeline.
[0077] In some possible embodiments, the pipeline complexity determination step specifically includes:
[0078]
[0079] Where PC is the pipeline complexity, TCF is the weight of the first influence factor, ECF is the weight of the second influence factor, UUCP is the total weight of the nodes, and PF is the productivity of the pipeline, which is mainly obtained by evaluating the overall performance of the pipeline.
[0080] In some possible embodiments, S3 specifically includes:
[0081] The transition probabilities of a Markov chain can be obtained by combining data input with a modified greedy algorithm. For any n∈T and i,j∈T, p ij (n)=p(PC n+1 =j|PC n =i) where i,j represent nodes in the pipeline, and this formula is the transition probability of the Markov chain.
[0082] In general, the transition probability depends not only on the states i and j and the pipeline complexity PC, but also on the node time n. However, if the transition probability is independent of time n, the Markov chain has a stationary transition probability, and {PC} n A chain with n ∈ T is called a homogeneous Markov chain.
[0083] To distinguish between homogeneous Markov chains and Markov chain transition probabilities, let p ij (n) is marked as p ij Obtain the system state transition probability matrix:
[0084]
[0085] The n-step transition probability matrix of the Markov chain can be obtained from this matrix, namely:
[0086]
[0087] In some possible embodiments, S4 specifically includes:
[0088] Based on the n-step transition probability matrix of the Markov chain, the entire pipeline steps are divided. The Hidden Markov Model (HMM) is a doubly stochastic process; not only are the state transitions random events, but the relationship between states and outputs is also a stochastic process. It consists of an initial state chain Π, a state transition probability matrix A, and an observation probability matrix. The decision is made, where Π and A determine the associated state sequence matrix. The state transition probability matrix A and the initial state probability vector π determine the hidden Markov chain, generating an unobservable state sequence; the observation probability matrix... Having determined how to generate observations from states, and combining this with the state sequence, we determine how to generate the probability sequence. Therefore, we can use the p obtained above... ij The transition probabilities between different states of each node are obtained by working backward from the last state of each node:
[0089]
[0090] Since A is shared in the transition probabilities, it is equivalent to sharing the state transition probability matrix. Therefore, the representation diagram of the latent variable transition can be obtained from the transition probability formula above, as shown in the figure. Figure 2 As shown;
[0091] This allows us to obtain the conditional probability distribution of the variable:
[0092]
[0093] Ultimately, the joint probability distribution over the observed and latent variables can be obtained, i.e.:
[0094]
[0095] Where x = [x1, x2, x3, ..., x] k ], z = [z1, z2, z3, ... z k ], where θ={π,A,φ} represents the set of model control parameters.
[0096] Finally, by analyzing the joint distribution above, we obtain the time distribution of the target state in the target stage.
[0097] A second aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a DevOps-based pipeline optimization method.
[0098] A third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is used to implement a DevOps-based pipeline optimization method.
[0099] This application uses a greedy algorithm to analyze data and establish a basic Markov model, then uses dynamic programming to backward derive a hidden Markov model, providing optimal pipeline evaluation. This application aims to provide the following effects:
[0100] 1. The overall project scale can be accurately measured, reducing the waste of human resources and avoiding the harm caused by inaccurate calculations due to human error in complex projects, thereby reducing project costs.
[0101] 2. Project details can be controlled. By processing data, the optimal construction method of the pipeline can be obtained, allowing project managers to better control the entire project process.
[0102] 3. Project managers can identify the shortcomings in their project development process design from all the data they acquire, thereby improving their capabilities and enabling them to work better for the company.
[0103] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A DevOps-based pipeline optimization method, characterized in that, The specific steps include the following: S1. Obtain historical data of the pipeline, get node information and impact factors, and determine the total weight of the nodes and the weight of the impact factors; The determination of the total weight of the node includes: Obtain node information and node weights; Based on the obtained node information, segmented data is obtained, and segment weights are calculated based on the segmented data and the weights of the nodes corresponding to each segment. The total weight of a node is determined based on its node weight and segment weight. S2. Determine the pipeline complexity based on the total node weight and the impact factor weight; the impact factor weight includes a first impact factor weight and a second impact factor weight, and the steps for obtaining the first impact factor weight and the second impact factor weight include: The impact factors are obtained and divided into a first impact factor and a second impact factor. The first impact factor is a technology impact factor, and the second impact factor is a non-technology impact factor. The formula for calculating the weight of the first impact factor is: ； The formula for calculating the weight of the second impact factor is: ； in, As a level, The weight of the first impact factor. The weight of the second impact factor; S3. Based on the pipeline complexity and the greedy algorithm, establish a Markov model and obtain the state set generated by each node. S4. Based on the state set generated by each node, a hidden Markov model is established using dynamic programming of the Markov model to determine the time distribution of the target node and obtain the workload of the target node.
2. The DevOps-based pipeline optimization method according to claim 1, characterized in that, The pipeline complexity determination steps specifically include: ； in, For pipeline complexity, The weight of the first impact factor. The weight of the second impact factor. The total weight of the nodes. The productivity of the assembly line.
3. The DevOps-based pipeline optimization method according to claim 1, characterized in that, The specific steps of establishing a Markov model based on pipeline complexity and a greedy algorithm include: A greedy algorithm is introduced to obtain the transition probability of the Markov chain based on the pipeline complexity; Obtain node information and the current node time, and based on the transition probabilities of the Markov chain, obtain the n-step transition probability matrix of the Markov chain.
4. The DevOps-based pipeline optimization method according to claim 3, characterized in that, The specific steps of establishing a hidden Markov model based on dynamic programming of a Markov model include: A hidden Markov model is established based on the n-step transition probability matrix of the Markov chain; Obtain the initial state chain, state transition probability matrix, observation probability matrix, and initial state probability vector; Based on the initial state chain, the state transition probability matrix, the observation probability matrix, and the initial state probability vector, the observation sequence and the transition probabilities of the nodes are determined.
5. The DevOps-based pipeline optimization method according to claim 4, characterized in that, The determination of the time distribution of the target node's target state specifically includes: The latent variables and conditional probability distributions of the nodes are determined based on the transition probabilities of the nodes, and the observed variables are obtained based on the observation sequence. Obtain the joint probability distribution of the observed variables and the latent variables to obtain the time distribution of the target state of the target node.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a DevOps-based pipeline optimization method as described in any one of claims 1 to 5.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a DevOps-based pipeline optimization method as described in any one of claims 1 to 5.