Extensible devops pipeline automation method and system

Abstract

The application discloses an extensible DevOps pipeline automatic arrangement method and system, and the method comprises the following steps: step one, pipeline initialization; step two, double graph construction, after the task type list of the current stage is de-duplicated and verified, a directed graph and an undirected graph are simultaneously constructed, the directed graph adds directed edges according to a predefined task dependency rule table; the undirected graph adds undirected edges between the same nodes; step three, loop detection; step four, connected component analysis and topological sorting; step five, hierarchical parallel execution; step six, stage iteration, after all batches in the current stage are successfully executed, if there is a subsequent stage, the task list of the subsequent stage is loaded, steps two to five are repeated until all stages are executed. The application does not need to manually declare the dependency relationship and parallel grouping, and significantly reduces the pipeline configuration complexity and error probability.

Description

Technical Field

[0001] This invention relates to an automatic pipeline orchestration method and system, and more particularly to a scalable DevOps pipeline automatic orchestration method and system. Background Technology

[0002] Modern software development commonly employs CI / CD pipelines to automate the entire process of code building, testing, scanning, deployment, load testing, and artifact release. Traditional CI / CD tools (such as Jenkins Pipeline and GitLab CI) define pipelines using declarative or script-based configuration files. Users need to manually declare the execution order, phase divisions, and parallel relationships of tasks in YAML or Groovy scripts. When the pipeline involves multiple task types (such as code building, unit testing, code scanning, security scanning, deployment, and automated testing), the complexity of manual orchestration increases dramatically, easily leading to dependency omissions or redundant sequential execution.

[0003] In existing technologies, topological sorting of directed acyclic graphs (DAGs) has been widely used in the field of task scheduling. Currently, the industry generally uses DAGs to model task dependencies to achieve pipeline scheduling. However, in CI / CD pipeline scenarios, there is still no complete technical solution that organically combines "connected component analysis of dual-graph architecture (directed graph + undirected graph)," "connected component analysis of dual-graph architecture (directed graph + undirected graph)," and "layered parallel execution framework."

[0004] Chinese invention patent publication number "CN111104102A" discloses a method for constructing automated pipelines for multi-business scenarios. This solution uses a single directed graph to complete task scheduling calculations and supports multi-business process orchestration. However, this solution has the following drawbacks: it only uses a single directed graph for modeling, does not introduce undirected graph connectivity analysis, and cannot automatically decompose independent parallel task links; adding new pipeline steps requires modifying the engine's core Action definition, resulting in high coupling and a lack of plug-in extension mechanisms. Chinese invention patent publication number "CN108399082B" discloses a method and system for generating continuous integration pipelines. This method generates pipelines based on fixed-level testing, with dependencies fixed by a preset level. However, it has the following drawbacks: it does not support dynamic dependency topology analysis, cannot calculate the optimal parallel sequence based on the task list in real time, and lacks dynamic grouping capabilities.

[0005] The existing technology has the following drawbacks: First, manual orchestration is inefficient and error-prone: Existing CI / CD tools require users to manually declare task dependencies and parallel groupings in configuration files. When the pipeline involves dozens of heterogeneous task types (such as build, test, scan, deployment, elastic scaling, domain management, etc.), manually sorting out dependencies is extremely complex and can easily lead to dependency omissions (causing execution failure) or excessive serialization (wasting execution time).

[0006] Second, it cannot automatically identify independent parallel links: Existing DAG-based scheduling schemes mostly use a single directed graph for topology sorting and advance execution layer by layer with an in-degree of zero. However, when there are multiple completely independent task links in the pipeline (such as the "code building → deployment → automated testing" link and the "third-party library release → tagging → cache clearing" link, which are independent of each other), existing schemes cannot automatically identify them as independent groups for parallel processing and still schedule them serially according to a unified topology level.

[0007] Third, pipeline step extension requires modification of the framework code: When adding language or build tool support to the existing system, the core scheduling logic or configuration mapping table of the pipeline engine needs to be modified. The high coupling between the framework and the step implementation makes it impossible to achieve "zero-intrusion" step extension.

[0008] Fourth, lack of step-level degradation control capability: When a certain type of task executor fails, the existing solution can usually only pause the entire pipeline or skip the entire stage, and cannot perform fine-grained degradation control on individual steps (such as marking them as offline and continuing to execute subsequent dependent tasks). Summary of the Invention

[0009] In view of the above situation and in order to solve the technical problems of the prior art, the present invention provides a scalable DevOps pipeline automatic orchestration method and system.

[0010] This invention solves the above-mentioned technical problems through the following technical solution: a scalable DevOps pipeline automatic orchestration method, characterized by comprising the following steps: Step 1: Pipeline initialization. After receiving the pipeline execution request, the system loads the corresponding pipeline definition, obtains the task type list for each stage, and the annotation scanner has completed step registration during the system startup phase, and the executor factory is ready. Step 2: Dual-graph construction. After deduplicating and validating the task type list for the current stage, both a directed graph and an undirected graph are constructed simultaneously. Directed edges are added to the directed graph according to the predefined task dependency rule table; undirected edges are added between the same nodes to the undirected graph. Step 3: Cycle detection. Perform a cycle detection algorithm on the directed graph. If a circular dependency exists, throw an exception to terminate the orchestration and prevent deadlock. Step 4: Connectivity component analysis and topological sorting. First, identify isolated nodes (both in-degree and out-degree are zero) and execute them as independent batches. Then, identify the starting node (in-degree is zero, out-degree is greater than zero). Perform BFS (Breadth-First Search) on the undirected graph to group the starting nodes. For each group, perform DFS (Depth-First Search) postorder traversal on the directed graph and reverse it to obtain the topological sorting sequence of the group. Step 5: Layered parallel execution. The topology analysis results are converted into execution batches and submitted to the concurrent scheduler. Batches from different groups are executed in parallel. Tasks within the same group are executed in the topology sequence. The scheduler monitors the execution results and terminates quickly when there is a failure. When paused, it waits for approval callback. Step Six: Stage Iteration. After all batches in the current stage are executed successfully, if there are subsequent stages, load their task list and repeat steps two through five until all stages are completed.

[0011] Preferably, the annotation scanner functions as follows: when the system starts, it uses reflection to scan all implementation classes marked with build step annotations under the specified package path. The annotations declare the combination of programming languages ​​and build tools supported by the step. After extracting the annotation metadata, the annotation scanner registers it in the step registry.

[0012] Preferably, the step registry is a mapping table that maintains "language: build tool" to step implementation class. This mapping table is built once by a static code block during the system initialization phase. During subsequent runtime, the corresponding step implementation class is obtained by looking up the table and its constructor is instantiated by invoking reflection.

[0013] Preferably, the executor factory dynamically selects and returns the corresponding task executor instance based on the task type code and deployment environment. The executor factory internally maintains the mapping relationship between task types and executors, and implements lazy loading of all executors through a dependency injection framework.

[0014] Preferably, the directed graph in step two is constructed by a directed graph builder, which takes the list of task types selected by the user in the pipeline as input and constructs the directed graph according to a predefined task dependency rule table.

[0015] Preferably, the undirected graph in step two is constructed by an undirected graph builder, which synchronously constructs an undirected graph using the same node and edge information. This undirected graph is used for subsequent connectivity analysis to determine which task nodes have direct or indirect associated paths.

[0016] Preferably, the directed graph is subjected to loop detection by a connected component analysis loop detector. If a loop is detected, an exception is immediately thrown and the orchestration process is terminated to prevent the pipeline from entering a deadlock state.

[0017] Preferably, the connected component analysis in step four is performed by at least one connected component analyzer. First, all “starting nodes” are identified in the directed graph. Then, in the undirected graph, a breadth-first search is performed with each unvisited starting node as a seed. All starting nodes reachable through undirected edges are grouped into the same group. During the BFS process, the visited set is maintained to prevent repeated traversal. Finally, multiple starting node groups are output, and each group corresponds to an independent task chain that can be scheduled in parallel.

[0018] Preferably, the topological sorting in step four is performed by at least one topological sorter. Each starting node output by the connected component analyzer is grouped, and a depth-first postorder traversal is performed on the directed graph. The traversal result is reversed to obtain the correct topological sorting sequence. For isolated nodes that have no dependencies and no successors, they are executed as a separate batch. Finally, a two-dimensional list is output, with the outer layer being the parallel execution group and the inner layer being the ordered task sequence within each group.

[0019] This invention also provides a scalable DevOps pipeline automatic orchestration system, characterized in that it comprises: The step registration layer includes an annotation scanner, a step registry, and an executor factory connected in sequence. The annotation scanner scans all implementation classes of build step annotations under a specified package path using reflection. Annotations have built-in metadata supporting programming languages ​​and build tool combinations. Annotation information is automatically extracted and registered to the step registry, eliminating the need for manual mapping. A built-in static mapping table stores the correspondence between language build tools and step implementation classes; the system initialization completes the mapping construction once. At runtime, the corresponding step implementation class is matched by looking up the table, and step objects are instantiated using reflection. Adding a new build process only requires adding an annotated implementation class, without modifying the engine's core code, adhering to the open / closed design principle. The executor factory has built-in mapping relationships between task types, deployment environments, and executors, and uses a dependency injection framework to lazily load all executors. It dynamically matches and returns a dedicated executor instance based on the task code and deployment environment, allowing the same task to adapt to multiple environment execution strategies. The topology analysis layer includes a directed graph builder, a loop detector, a topology sorter, an undirected graph builder, and a connected component analyzer. The directed graph builder takes the current stage's task list as input and generates directed edges based on a predefined task dependency rule table. Nodes represent task types, and directed edges represent predecessor-to-successor dependencies, retaining only valid dependencies in the current pipeline. The undirected graph builder synchronously reuses nodes and associated edges of the same task to construct an undirected graph for connectivity analysis, identifying direct / indirect links between tasks. The loop detector executes a loop detection algorithm based on the directed graph. If a circular dependency is detected, an exception is immediately thrown, terminating the orchestration process and avoiding pipeline deadlock. The topology sorter performs topology sorting, groups each starting node output by the connected component analyzer, performs a depth-first postorder traversal on the directed graph, and reverses the traversal result to obtain the correct topology sorted sequence. Isolated nodes without any dependencies or successors are executed as a separate batch. The final output is a two-dimensional list, with the outer layer representing parallel execution groups and the inner layer representing the ordered task sequences within each group. The parallel execution layer comprises a batch generator, a concurrent task scheduler, a degradation controller, and an execution result collector, all connected in sequence. The batch generator receives the topology sorting results, binds them to the corresponding task executor instances, encapsulates them into standardized schedulable batches, and distributes them to the concurrent task scheduler. The concurrent task scheduler implements concurrent scheduling of thread pools based on the Completion Service pattern. Different batches are executed in complete parallel, while tasks within the same batch are executed serially according to the topology sequence. It supports rapid termination upon execution failure and a manual approval pause callback mechanism. The degradation controller allows each task executor to inherit from a degradation base class. Before execution, it queries the switch status of the step. If the switch is off, the task is marked as "offline" and a success message is returned directly without actually executing the task logic, allowing subsequent dependent tasks to continue running. The execution result collector summarizes the execution results of all batches and updates the overall execution status of the pipeline. The infrastructure layer includes interconnected thread pool managers and database routers, as well as a state persistence module. The thread pool manager configures independent thread pools for different types of tasks, supporting configurable core thread counts and maximum thread counts. During pipeline task execution, the database router forces routing to the master database through the prompt manager of the sharding middleware to prevent data inconsistency issues caused by master-slave replication delays. The state persistence module persists the topology analysis results, the execution status and details of each task to the database, supporting pipeline breakpoint recovery and execution auditing.

[0020] The positive and progressive effects of this invention are as follows: (1) High degree of automation, eliminating the burden of manual orchestration: This invention automatically analyzes task dependencies and calculates the optimal execution topology through graph algorithms. Users only need to select the required task type, without having to manually declare dependencies and parallel grouping, which significantly reduces pipeline configuration complexity and error probability.

[0021] (2) Automatic parallel execution of independent links with optimal efficiency: Through the innovative design of a dual-graph architecture (directed graph + undirected graph), the BFS connectivity analysis of the undirected graph is used to automatically identify completely independent task links and divide them into independent execution groups that can be scheduled in parallel. Compared with the existing schemes that advance layer by layer based on a single directed graph, this invention can explore more parallel opportunities and shorten the overall pipeline execution time.

[0022] (3) Zero-intrusive and scalable plug-in architecture: Through annotation-driven reflection scanning mechanism, adding new pipeline steps only requires adding an annotated implementation class, without modifying the core framework code. The step registry is automatically built when the system starts, realizing the "open-closed principle"—open to extension, closed to modification.

[0023] (4) Fine-grained degradation control: Through the switching mechanism of the degradable base class, the start and stop control of a single step actuator can be performed independently. In case of failure, the step is marked as offline without affecting the execution of the remaining steps, thus improving the fault tolerance and availability of the system.

[0024] (5) Robust exception handling: Loop detection detects and blocks circular dependencies during the orchestration stage to avoid runtime deadlock; the fast termination strategy prevents meaningless subsequent task execution; the breakpoint recovery mechanism supports re-execution from the point of failure.

[0025] (6) Multi-environment adaptability: The actuator factory dynamically selects the matching actuator implementation according to the deployment environment (physical machine, containerized platform, etc.). The same task type can use different execution strategies in different environments without the need to configure pipelines separately for each environment. Attached Figure Description

[0026] Figure 1 This is a flowchart of the scalable DevOps pipeline automatic orchestration method of the present invention.

[0027] Figure 2 This is a schematic diagram of the principle of the scalable DevOps pipeline automatic orchestration system of the present invention. Detailed Implementation

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0029] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0030] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0031] like Figure 1 As shown, the scalable DevOps pipeline automatic orchestration method of the present invention includes the following steps: Step 1: Pipeline initialization. After receiving the pipeline execution request, the system loads the corresponding pipeline definition, obtains the task type list for each stage, and the annotation scanner has completed step registration during the system startup phase, and the executor factory is ready. Step 2: Dual-graph construction. After deduplicating and validating the task type list for the current stage, both a directed graph and an undirected graph are constructed simultaneously. Directed edges are added to the directed graph according to the predefined task dependency rule table; undirected edges are added between the same nodes to the undirected graph. Step 3: Cycle detection. Perform a cycle detection algorithm on the directed graph. If a circular dependency exists, throw an exception to terminate the orchestration and prevent deadlock. Step 4: Connectivity component analysis and topological sorting. First, identify isolated nodes (both in-degree and out-degree are zero) and execute them as independent batches. Then, identify the starting node (in-degree is zero, out-degree is greater than zero). Perform BFS (Breadth-First Search) on the undirected graph to group the starting nodes. For each group, perform DFS (Depth-First Search) postorder traversal on the directed graph and reverse it to obtain the topological sorting sequence of the group. Step 5: Layered parallel execution. The topology analysis results are converted into execution batches and submitted to the concurrent scheduler. Batches from different groups are executed in parallel. Tasks within the same group are executed in the topology sequence. The scheduler monitors the execution results and terminates quickly when there is a failure. When paused, it waits for approval callback. Step Six: Stage Iteration. After all batches in the current stage are executed successfully, if there are subsequent stages, load their task list and repeat steps two through five until all stages are completed.

[0032] The annotation scanner functions as follows: At system startup, it uses reflection to scan all implementation classes annotated with build step annotations under a specified package path. These annotations declare the supported programming languages ​​(e.g., Java, Go, Python) and build tools (e.g., Maven, Gradle, NPM) for that step. The annotation scanner extracts the annotation metadata and registers it in the step registry. The step annotation attributes include `languages` (an array of supported programming languages) and `buildTypes` (an array of supported build tools). The step registry maintains a mapping table from "language:build tool" to the step implementation class. This mapping table is built once during system initialization by a static code block. Subsequent runtime checks the table to retrieve the corresponding step implementation class and instantiates it by calling its constructor via reflection.

[0033] The executor factory dynamically selects and returns the corresponding task executor instance based on the task type code and deployment environment (such as daily, pre-production, production, containerized environment, etc.). The executor factory internally maintains the mapping relationship between task types and executors, and implements lazy loading of all executors through a dependency injection framework.

[0034] Step two, the directed graph, is constructed by a directed graph builder. It takes a list of user-selected task types in the pipeline as input and builds the graph according to a predefined task dependency rule table (e.g., "Build" depends on "Unit Testing" and "Code Scanning"). Each node in the graph represents a task type, and each directed edge represents a dependency relationship of "predecessor task → successor task". Edges are added only if the dependent task is also in the current pipeline task list, ensuring the graph contains only valid dependencies. Step two, the undirected graph, is constructed by an undirected graph builder. An undirected graph is built synchronously using the same node and edge information. This undirected graph is used for subsequent connectivity analysis to determine which task nodes have direct or indirect paths. The directed graph is then inspected for loops by a connectivity component analysis loop detector. If a loop (i.e., a circular dependency) is detected, an exception is immediately thrown and the orchestration process is terminated to prevent the pipeline from entering a deadlock state.

[0035] The connected component analysis in step four is performed by at least one connected component analyzer. First, all "starting nodes" (nodes with zero in-degree and greater than zero out-degree) are identified in the directed graph. Then, in the undirected graph, breadth-first search (BFS) is performed using each unvisited starting node as a seed, grouping all starting nodes reachable via undirected edges into the same group. During BFS, a visited set is maintained to prevent repeated traversals. Finally, multiple starting node groups are output, each corresponding to an independent, parallel-scheduling task chain. The topology sorting in step four is performed by at least one topology sorter. For each starting node group output by the connected component analyzer, a depth-first post-order traversal (DFS post-order) is performed in the directed graph. The traversal result is reversed to obtain the correct topology sorting sequence. Isolated nodes without any dependencies or successors are treated as separate execution batches. Finally, a two-dimensional list is output, with the outer layer representing parallel execution groups and the inner layer representing the ordered task sequences within each group. The execution batches in step four are completed by at least one execution batch generator, which receives the hierarchical topology sequence output by the topology sorter and transforms it into execution batches that can be submitted to a scheduler. Each batch contains a set of tasks and their corresponding executor instances.

[0036] Implemented based on the Completion Service pattern, each execution batch is encapsulated into concurrent task commands by at least one concurrent task scheduler and submitted to the thread pool for parallel execution. Tasks within the same batch are executed in topological sequence (maintaining dependency order within the same link). Different batches are completely parallel. The scheduler listens for the completion signal of each batch. If any batch fails to execute, is aborted, or enters a manual approval pause state, the scheduling of subsequent batches is immediately terminated.

[0037] like Figure 2 As shown, the scalable DevOps pipeline automatic orchestration system of the present invention includes: The step registration layer includes an annotation scanner, a step registry, and an executor factory connected in sequence. The annotation scanner scans all implementation classes of build step annotations under a specified package path using reflection. The annotations have built-in support for metadata of programming languages ​​and build tools (Java / Go / Python, Maven / NPM / Gradle, etc.). Annotation information is automatically extracted and registered to the step registry, eliminating the need for manual maintenance of mapping relationships. A built-in static mapping table stores the correspondence between language build tools and step implementation classes, and the system initializes the mapping and build process once. At runtime, the corresponding step implementation class is matched by looking up the table, and the step object is instantiated by reflection. Adding a new build process only requires adding an annotated implementation class, without modifying the core engine code, following the open / closed design principle. The system establishes a mapping relationship between storage language build tools and step implementation classes, completing the mapping and construction in one go during system initialization. At runtime, it matches the corresponding step implementation class by looking up a table and instantiates the step object using reflection. Adding a new build process only requires adding an annotated implementation class, without modifying the core engine code, adhering to the open / closed design principle. The executor factory has a built-in mapping relationship between task types, deployment environments, and executors, and relies on a dependency injection framework to lazily load all executors. Based on the task code and deployment environment (physical machine / container / pre-production / production), it dynamically matches and returns a dedicated executor instance, allowing the same task to adapt to multiple environment execution strategies. The topology analysis layer includes a directed graph builder, a loop detector, a topology sorter, an undirected graph builder, and a connected component analyzer. The directed graph builder takes the current stage's task list as input and generates directed edges based on a predefined task dependency rule table. Nodes represent task types, and directed edges represent predecessor-to-successor dependencies, retaining only valid dependencies in the current pipeline. The undirected graph builder synchronously reuses nodes and associated edges of the same task to construct an undirected graph for connectivity analysis, identifying direct / indirect links between tasks. The loop detector executes a loop detection algorithm based on the directed graph. If a circular dependency is detected, an exception is immediately thrown to terminate the orchestration process, avoiding pipeline deadlock. The topology sorter performs topology sorting, groups each starting node output by the connected component analyzer, performs a depth-first post-order traversal (DFS post-order) on the directed graph, and reverses the traversal result to obtain the correct topology sorted sequence. Isolated nodes without any dependencies or successors are executed as a separate batch. The final output is a two-dimensional list: the outer layer represents parallel execution groups, and the inner layer represents the ordered task sequences within each group. The parallel execution layer comprises a batch generator, a concurrent task scheduler, a degradation controller, and an execution result collector, all connected in sequence. The batch generator receives the topology sorting results, binds them to the corresponding task executor instances, encapsulates them into standardized schedulable batches, and distributes them to the concurrent task scheduler. The concurrent task scheduler implements concurrent scheduling of thread pools based on the Completion Service pattern. Different batches are executed in complete parallel, while tasks within the same batch are executed serially according to the topology sequence. It supports rapid termination upon execution failure and a manual approval pause callback mechanism. The degradation controller allows each task executor to inherit from a degradation base class. Before execution, it queries the switch status of the step. If the switch is off, the task is marked as "offline" and a success message is returned directly without actually executing the task logic, allowing subsequent dependent tasks to continue running. The execution result collector summarizes the execution results of all batches (success, failure, skipped, paused, etc.) and updates the overall execution status of the pipeline. The infrastructure layer includes interconnected thread pool managers and database routers, as well as a state persistence module. The thread pool manager configures independent thread pools for different types of tasks, supporting configurable core thread counts and maximum thread counts. During pipeline task execution, the database router forces routing to the master database through the prompt manager of the sharding middleware to prevent data inconsistency issues caused by master-slave replication delays. The state persistence module persists the topology analysis results, the execution status and details of each task to the database, supporting pipeline breakpoint recovery and execution auditing.

[0038] The essential difference of this invention lies in the fact that it achieves automatic identification and grouping of independent task links through dual-graph (directed graph and undirected graph) modeling, and achieves zero-intrusion extension of pipeline steps through annotation-driven modeling, forming an end-to-end automated orchestration closed loop from "step registration → dependency modeling → topology analysis → parallel execution". Example

[0039] Application scenario: An internal DevOps platform of a large Internet company needs to support continuous integration and continuous deployment of hundreds of applications, involving multiple programming languages ​​(Java, Go, Python, Node.js, Scala, etc.) and multiple deployment environments (physical machines, containerized platforms).

[0040] Implementation example: An application's release pipeline includes the following 8 task types: unit testing, code quality scanning, code building, artifact deployment, performance stress testing, automated testing, third-party library release, and manual review.

[0041] The automatic arrangement is performed using the method of this invention as follows: Dual-graph construction: According to the dependency rule table, "Build" depends on the completion of "Unit Test" and "Code Quality Scan"; "Artifact Deployment" and "Second-Party Library Release" both depend on "Build"; "Performance Load Testing" and "Automated Testing" depend on "Artifact Deployment"; "Manual Review" has no dependencies and is an isolated node.

[0042] Connectivity analysis: In the directed graph, the starting nodes with zero in-degree and greater than zero out-degree are "Unit Test" and "Code Quality Scan". Performing BFS on the undirected graph reveals that these two starting nodes are indirectly connected through the "Build" node and belong to the same group G1. Although "Secondary Library Release" is also connected to "Build", its in-degree is not zero (depending on "Build"), it is not a starting node, and is already within the traversal range of G1.

[0043] Topological sorting output: For each starting node group output by the connected component analyzer, perform a depth-first post-order traversal on the directed graph, and reverse the traversal result to obtain the correct topological sorting sequence.

[0044] Isolated node batches: Manual review.

[0045] Grouped G1 sequences (DFS postorder reversal): unit testing, code quality scanning, code building, artifact deployment, third-party library release, performance stress testing, and automated testing.

[0046] Parallel execution: The "Manual Review" batch and the "Group G1" batch can be executed in parallel. Within Group G1, execution proceeds in topological order, with "Unit Testing" and "Code Quality Scanning" starting as parallel start nodes and able to be launched in parallel; "Artifact Deployment" and "Third-Party Library Release" both rely solely on "Build" to complete and can also be launched in parallel.

[0047] This invention achieves the following functions: Automation of orchestration: After the user selects the task type, the system automatically completes dependency analysis and parallel grouping, eliminating the need for manual dependency configuration. Increased parallelism: Independent task chains are automatically identified and executed in parallel, significantly reducing execution time compared to purely serial solutions. Ease of expansion: Adding build tool support requires only adding an annotated implementation class and restarting the service; the core framework code remains unchanged. Parameter setting example (not intended to limit the scope of the claims): The core thread count of the thread pool is configured to 50, and the maximum thread count is configured to 500. The annotation scanning package path is configured to the implementation package of the project's steps. The dependency rule table is maintained in the form of an enumeration class, supporting runtime queries.

[0048] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A scalable DevOps pipeline automatic orchestration method, characterized in that, It includes the following steps: Step 1: Pipeline initialization. After receiving the pipeline execution request, the system loads the corresponding pipeline definition, obtains the task type list for each stage, and the annotation scanner has completed step registration during the system startup phase, and the executor factory is ready. Step 2: Dual-graph construction. After deduplicating and validating the task type list for the current stage, both a directed graph and an undirected graph are constructed simultaneously. Directed edges are added to the directed graph according to the predefined task dependency rule table; undirected edges are added between the same nodes to the undirected graph. Step 3: Cycle detection. Perform a cycle detection algorithm on the directed graph. If a circular dependency exists, throw an exception to terminate the orchestration and prevent deadlock. Step 4: Connected component analysis and topological sorting. First, identify isolated nodes and execute them as independent batches. Then, identify the starting node and perform BFS on the undirected graph to group the starting nodes. For each group, perform DFS postorder traversal on the directed graph and reverse it to obtain the topological sort sequence of the group. Step 5: Layered parallel execution. The topology analysis results are converted into execution batches and submitted to the concurrent scheduler. Batches from different groups are executed in parallel. Tasks within the same group are executed in the topology sequence. The scheduler monitors the execution results and terminates quickly when there is a failure. When paused, it waits for approval callback. Step Six: Stage Iteration. After all batches in the current stage are executed successfully, if there are subsequent stages, load their task list and repeat steps two through five until all stages are completed.

2. The scalable DevOps pipeline automatic orchestration method as described in claim 1, characterized in that, The annotation scanner functions as follows: when the system starts, it uses reflection to scan all implementation classes marked with build step annotations under the specified package path. The annotations declare the combination of programming languages ​​and build tools supported by the step. After extracting the annotation metadata, the annotation scanner registers it in the step registry.

3. The scalable DevOps pipeline automatic orchestration method as described in claim 1, characterized in that, The step registry maintains a mapping table from "language: build tool" to step implementation classes. This mapping table is built once by a static code block during system initialization. During subsequent runtime, the corresponding step implementation class is obtained by looking up the table and its constructor is instantiated by invoking reflection.

4. The scalable DevOps pipeline automatic orchestration method as described in claim 1, characterized in that, The executor factory dynamically selects and returns the corresponding task executor instance based on the task type code and deployment environment. The executor factory internally maintains the mapping relationship between task types and executors, and implements lazy loading of all executors through a dependency injection framework.

5. The scalable DevOps pipeline automatic orchestration method as described in claim 4, characterized in that, The directed graph in step two is constructed by a directed graph builder, which takes the list of task types selected by the user in the pipeline as input and constructs the directed graph according to a predefined task dependency rule table.

6. The scalable DevOps pipeline automatic orchestration method as described in claim 1, characterized in that, The undirected graph in step two is constructed by an undirected graph builder, which synchronously constructs an undirected graph using the same node and edge information. This undirected graph is used for subsequent connectivity analysis to determine which task nodes have direct or indirect related paths.

7. The scalable DevOps pipeline automatic orchestration method as described in claim 1, characterized in that, The directed graph is subjected to loop detection by a loop detector in connected component analysis. If a loop is detected, an exception is immediately thrown and the orchestration process is terminated to prevent the pipeline from entering a deadlock state.

8. The scalable DevOps pipeline automatic orchestration method as described in claim 1, characterized in that, The connected component analysis in step four is performed by at least one connected component analyzer. First, all "starting nodes" are identified in the directed graph. Then, in the undirected graph, a breadth-first search is performed with each unvisited starting node as a seed. All starting nodes reachable through undirected edges are grouped into the same group. During the BFS process, the visited set is maintained to prevent repeated traversal. Finally, multiple starting node groups are output, and each group corresponds to an independent task chain that can be scheduled in parallel.

9. The scalable DevOps pipeline automatic orchestration method as described in claim 1, characterized in that, The topological sorting in step four is completed by at least one topological sorter. Each starting node output by the connected component analyzer is grouped, and a depth-first postorder traversal is performed on the directed graph. The traversal result is reversed to obtain the correct topological sorting sequence. For isolated nodes that have no dependencies and no successors, they are executed as a separate batch. Finally, a two-dimensional list is output, with the outer layer being the parallel execution group and the inner layer being the ordered task sequence within each group.

10. A scalable DevOps pipeline automatic orchestration system, characterized in that, It includes: The step registration layer includes an annotation scanner, a step registry, and an executor factory connected in sequence. The annotation scanner scans all implementation classes of build step annotations under a specified package path using reflection. Annotations have built-in metadata supporting programming languages ​​and build tool combinations. Annotation information is automatically extracted and registered to the step registry, eliminating the need for manual mapping. A built-in static mapping table stores the correspondence between language build tools and step implementation classes; the system initialization completes the mapping construction once. At runtime, the corresponding step implementation class is matched by looking up the table, and step objects are instantiated using reflection. Adding a new build process only requires adding an annotated implementation class, without modifying the engine's core code, adhering to the open / closed design principle. The executor factory has built-in mapping relationships between task types, deployment environments, and executors, and uses a dependency injection framework to lazily load all executors. It dynamically matches and returns a dedicated executor instance based on the task code and deployment environment, allowing the same task to adapt to multiple environment execution strategies. The topology analysis layer includes a directed graph builder, a loop detector, a topology sorter, an undirected graph builder, and a connected component analyzer. The directed graph builder takes the current stage's task list as input and generates directed edges according to a predefined task dependency rule table. Nodes represent task types, and directed edges represent predecessor-to-follower dependencies, retaining only valid dependencies in the current pipeline. The undirected graph builder synchronously reuses nodes and associated edges of the same tasks to construct an undirected graph for connectivity analysis, identifying direct / indirect links between tasks. The loop detector executes a loop detection algorithm based on the directed graph. If a circular dependency is detected, it immediately throws an exception to terminate the orchestration process, avoiding pipeline deadlock issues. The topology sorter completes the topology sorting, groups each starting node output by the connected component analyzer, performs a depth-first postorder traversal on the directed graph, reverses the traversal result to obtain the correct topology sorting sequence, and treats isolated nodes without any dependencies and without successors as separate execution batches. Finally, it outputs a two-dimensional list, with the outer layer being the parallel execution group and the inner layer being the ordered task sequence within each group. The parallel execution layer consists of a batch generator, a concurrent task scheduler, a fallback controller, and an execution result collector, all connected in sequence. The batch generator receives the topology sorting results, binds them to the corresponding task executor instances, encapsulates them into standardized schedulable batches, and sends them to the concurrent task scheduler. The concurrent task scheduler implements concurrent scheduling of thread pools based on the Completion Service pattern. Different batches are executed in complete parallelism, while tasks within the same batch are executed serially, strictly following the topology sequence. Supports rapid termination upon execution failure and a pause callback mechanism for manual approval; the degradation controller controls each task executor to inherit from and be degraded to a base class. Before execution, it queries the switch status of the step. If the switch is off, the task is marked as "offline" and a success is returned directly without actually executing the task logic, allowing subsequent dependent tasks to continue running; the execution result collector summarizes the execution results of all batches and updates the overall execution status of the pipeline. The infrastructure layer includes interconnected thread pool managers and database routers, as well as a state persistence module. The thread pool manager configures independent thread pools for different types of tasks, supporting configurable core thread counts and maximum thread counts. During pipeline task execution, the database router forces routing to the master database through the prompt manager of the sharding middleware to prevent data inconsistency issues caused by master-slave replication delays. The state persistence module persists the topology analysis results, the execution status and details of each task to the database, supporting pipeline breakpoint recovery and execution auditing.

Images