Operation and maintenance task workflow arrangement system in distributed graph database

Abstract

The invention relates to an operation and maintenance task workflow arrangement system in a distributed graph database, and the system comprises a workflow arrangement layer which is used for carrying out the recursive analysis of a workflow specification inputted by a user, so as to generate an executable DAG operation and maintenance task graph; the task management layer is used for adopting a two-stage submission protocol to ensure the atomic consistency of each operation and maintenance task in the DAG operation and maintenance task graph in a volatile memory and a persistent storage, and performing fault recovery on the uncompleted operation and maintenance task from the persistent storage based on the atomic consistency under the condition of memory data loss; and the conditional task execution layer is used for creating a security isolation sandbox for execution of each operation and maintenance task in the DAG operation and maintenance task graph, and evaluating a task execution result based on a dynamically updated conditional function in the execution process of the current operation and maintenance task so as to determine an execution path of a subsequent operation and maintenance task. Through the method and the device, the problem of how to improve the workflow efficiency of the operation and maintenance task in the graph database is solved.

Description

Technical Field

[0001] The present application relates to the field of graph database technology, and in particular to a workflow orchestration system for operation and maintenance tasks in a distributed graph database. Background Art

[0002] In modern enterprise applications, distributed graph database systems often need to manage hundreds of service nodes, involving complex service dependencies and operational procedures. As enterprises grow and develop, the demand for large-scale graph data processing also increases. The operational management of distributed graph database systems faces increasing challenges. Traditional task scheduling solutions have the following technical issues: The difficulty in dynamically selecting execution paths based on real-time operating status leads to inefficient task execution. Complex dependencies between tasks make it difficult to effectively manage the mixed execution of serial and parallel tasks. There is also a lack of intelligent recovery and status management mechanisms for task failures or timeouts. Furthermore, there are difficulties in effectively managing task resource allocation and load balancing in a distributed environment, as well as a lack of real-time status tracking and visual management of task execution processes.

[0003] Currently, no effective solution has been proposed for how to improve the workflow efficiency of operation and maintenance tasks in graph databases in related technologies. Summary of the Invention

[0004] An embodiment of the present application provides a workflow orchestration system for operation and maintenance tasks in a distributed graph database, so as to at least solve the problem in the related art of how to improve the workflow efficiency of operation and maintenance tasks in a graph database.

[0005] In a first aspect, an embodiment of the present application provides a system for orchestrating operation and maintenance task workflows in a distributed graph database, the system comprising a workflow orchestration layer, a task management layer, and a conditional task execution layer; The workflow orchestration layer is used to receive the operation and maintenance task workflow specification input by the user and recursively parse the workflow specification to generate an executable DAG operation and maintenance task graph; The task management layer is used to ensure the atomic consistency of each operation and maintenance task in the DAG operation and maintenance task graph in volatile memory and persistent storage using a two-phase commit protocol. In the event of a failure resulting in memory data loss, unfinished operation and maintenance tasks are recovered from persistent storage based on the atomic consistency. The conditional task execution layer is used to create a secure isolation sandbox for the execution of various operation and maintenance tasks in the DAG operation and maintenance task graph. During the execution of the current operation and maintenance task, the task execution result is evaluated based on the dynamically updated conditional function to determine the execution path of subsequent operation and maintenance tasks.

[0006] In some embodiments, the workflow orchestration layer is used to receive the operation and maintenance task workflow specification input by the user, and uniformly abstract the serial branches, parallel branches and conditional branches in the workflow specification into a directed acyclic graph, and then parse the dependencies and execution order of the operation and maintenance tasks through the Kahn algorithm to topologically sort the directed acyclic graph to generate an executable DAG operation and maintenance task graph.

[0007] In some embodiments, the workflow orchestration layer is used to introduce multi-version concurrency control through the MVCC version control mechanism to support workflow specification updates during operation and maintenance task execution: During the execution of an operation and maintenance task, if the workflow specification is updated, a new version of the DAG operation and maintenance task graph is copied and recursively parsed to generate the new version. Then, atomic.SwapPointer is used to perform an atomic update of the version, so that the operation and maintenance task being executed continues to use the old version of the DAG operation and maintenance task graph, and subsequent unexecuted operation and maintenance tasks use the new version of the DAG operation and maintenance task graph.

[0008] In some embodiments, the task management layer is configured to, when a task status update occurs for an operation and maintenance task in the DAG operation and maintenance task graph, first write a preparation update log to the distributed storage during the preparation phase of the two-phase commit protocol, and then atomically update the task status of the operation and maintenance task in the volatile memory and the persistent storage during the commit phase of the two-phase commit protocol, and delete the preparation update log after the update is successful; In the event of a failure resulting in memory data loss, unfinished operation and maintenance tasks are recovered from persistent storage based on the atomic update feature.

[0009] In some embodiments, the task management layer is used to support high-concurrency execution of various operation and maintenance tasks in the DAG operation and maintenance task graph through a concurrent task execution mechanism based on Go coroutines.

[0010] In some embodiments, the conditional task execution layer is used to create a Wasm security isolation sandbox through a dynamically loaded Wasm plug-in, or to create a Cgroups security isolation sandbox through a dynamically loaded Go plug-in, for the execution of various operation and maintenance tasks in the DAG operation and maintenance task graph.

[0011] In some embodiments, the conditional task execution layer is used to evaluate the task execution results based on the dynamically updated conditional function during the execution of the current operation and maintenance task to determine the execution path of the subsequent operation and maintenance task, wherein the conditional function supports complex logic combinations and high-order function programming, the update method of the conditional function is hot update, and the programming languages ​​of the conditional function include Rust, C++ and TinyGo.

[0012] In some embodiments, the system includes a user interface layer; The user interface layer is used to provide multiple interaction methods for users to input operation and maintenance task workflow specifications, wherein the interaction methods include a web console, REST API and command line tools.

[0013] In some embodiments, the system includes a storage layer and a resource management layer, wherein the storage layer includes a persistent distributed storage unit; The storage layer is used to persistently store data related to operation and maintenance tasks and supports transaction logs and checkpoint mechanisms; The resource management layer is used to intelligently schedule hardware resources for the execution of operation and maintenance tasks to perform load balancing.

[0014] In some embodiments, the system includes a graph database cluster, which is a cluster of NebulaGraph distributed graph database.

[0015] Compared with related technologies, the embodiment of the present application provides a system for orchestrating operation and maintenance task workflows in a distributed graph database, which includes: a workflow orchestration layer, which is used to recursively parse the workflow specifications input by the user to generate an executable DAG operation and maintenance task graph; a task management layer, which is used to adopt a two-phase commit protocol to ensure the atomic consistency of each operation and maintenance task in the DAG operation and maintenance task graph in volatile memory and persistent storage. In the event of memory data loss, the unfinished operation and maintenance tasks are fault recovered from the persistent storage based on atomic consistency; a conditional task execution layer, which is used to create a secure isolation sandbox for the execution of each operation and maintenance task in the DAG operation and maintenance task graph. During the execution of the current operation and maintenance task, the task execution result is evaluated based on the dynamically updated conditional function to determine the execution path of the subsequent operation and maintenance task, thereby realizing automatic state recovery of the operation and maintenance task after fault restart and avoiding global rollback. The sandbox execution environment effectively prevents damage from malicious code, and uses the conditional function to evaluate the task running status in real time to dynamically select the execution path, thereby improving the task execution efficiency, and solving the problem of how to improve the workflow efficiency of operation and maintenance tasks in the graph database. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 This is a structural block diagram of an operation and maintenance task workflow orchestration system according to an embodiment of the present application; Figure 2 is a workflow diagram of the workflow orchestration layer according to an embodiment of the application; Figure 3 is a schematic diagram of a workflow of a task management layer according to an embodiment of the application; Figure 4 is a schematic diagram of the workflow of the conditional task execution layer according to an embodiment of the application; Figure 5 This is a structural block diagram of an operation and maintenance task workflow orchestration system according to a specific embodiment of the present application; Figure 6 Schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0018] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0019] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0020] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "an," "the," and similar expressions used herein do not denote quantitative limitations and may refer to either the singular or the plural. The terms "comprise," "include," "have," and any variations thereof, used herein, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules (units) is not limited to the listed steps or units but may also include steps or units not listed, or may include other steps or units inherent to the process, method, product, or apparatus. The terms "connected," "connected," "coupled," and similar expressions used herein are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used herein, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" may mean: A exists alone; A and B exist simultaneously; or B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0021] The present invention provides a distributed graph database operation and maintenance task workflow arrangement system. Figure 1 This is a structural diagram of the operation and maintenance task workflow orchestration system according to an embodiment of the present application. Figure 1 As shown in the figure, the system includes a workflow orchestration layer, a task management layer (TaskManager) and a conditional task execution layer (ConditionalTask); The workflow orchestration layer is used to receive the operation and maintenance task workflow specifications input by the user and recursively parse the workflow specifications to generate an executable DAG operation and maintenance task graph; Specifically, the workflow orchestration layer is used to receive the operation and maintenance task workflow specification (WorkflowSpec) input by the user, and uniformly abstract the serial branches, parallel branches, and conditional branches in the workflow specification into a directed acyclic graph. The Kahn algorithm is then used to parse the dependencies and execution order of the operation and maintenance tasks, and the directed acyclic graph is topologically sorted to generate an executable DAG operation and maintenance task graph.

[0022] It is also used to introduce multi-version concurrency control through the MVCC version control mechanism to support workflow specification updates during operation and maintenance task execution: during the execution of an operation and maintenance task, if the workflow specification is updated, it is copied and recursively parsed to generate a new version of the DAG operation and maintenance task graph, and then atomically updated through atomic.SwapPointer, so that the operation and maintenance task being executed continues to use the old version of the DAG operation and maintenance task graph, and subsequent unexecuted operation and maintenance tasks use the new version of the DAG operation and maintenance task graph.

[0023] It should be noted that existing workflow engines mostly rely on statically defined task graphs when handling dependencies. While some systems support dynamic workflow adjustments at runtime, such as by returning a nested WorkflowDAG or using a "rewrite" mechanism to add new dependencies to running tasks, these operations are often heavyweight and lack a set of lightweight dynamic dependency parsing and execution algorithms that are tightly integrated with resource scheduling and conditional judgment. The workflow orchestration layer in the embodiments of this application, however, implements efficient topological sorting and scheduling combined with heterogeneous resources for scenarios such as mixed serial, parallel, and conditional branching. Figure 2 This is a workflow diagram of the workflow orchestration layer according to an embodiment of the application, such as Figure 2 As shown, the workflow orchestrator in the workflow orchestration layer is the brain of the system, responsible for parsing complex workflow definitions into executable task graphs and intelligently scheduling them. Its technical innovations include: DAG-based unified modeling—abstracting serial, parallel, and conditional branches into a directed acyclic graph; Kahn algorithm-based topological sorting—efficiently resolving task dependencies and execution order; MVCC versioning—supporting dynamic updates of workflow definitions at runtime using a copy-on-write mechanism; HEFT heuristic scheduling—a heterogeneous resource scheduling algorithm that incorporates data locality awareness; and incremental dependency resolution—supporting the dynamic addition or modification of task dependencies at runtime.

[0024] It should be further explained that if Figure 2 As shown, workflow parsing: recursively parse WorkflowSpec to construct a complete DAG graph. Kahn algorithm topological sorting: calculate the in-degree of all nodes, add nodes with in-degree 0 to the ready queue, loop through the ready queue, execute executable tasks in parallel, and dynamically update the in-degree of subsequent nodes. MVCC version control mechanism: to support runtime workflow updates, introduce multi-version concurrency control: Copy-on-Write (Copy-on-Write) - creating a new version when modifying a workflow without affecting ongoing tasks; atomic version switching - atomic version updates are achieved through atomic.SwapPointer; snapshot read - ongoing tasks continue to use the old version of the DAG, while new tasks use the new version.

[0025] Preferably, the core architecture code of the workflow orchestration layer is as follows: type WorkflowEngine struct { dagManager *DAGManager / / DAG manager topologyManager *TopologyManager / / Topology sort manager versionManager *MVCCVersionManager / / Version Control Manager scheduler *HEFTScheduler / / HEFT scheduler taskManager *TaskManager / / Task Manager logger logx.Logger } type WorkflowSpec struct { ID string / / workflow unique identifier Version int64 / / Version number Type WorkflowType / / "serial" | "parallel" | "conditional" | "dag" Rollback bool / / Rollback support Tasks []*TaskSpec / / Subtask list Dependencies map[string][]string / / Task dependency graph Metadata map[string]interface{} / / metadata } type DAGNode struct { TaskID string / / Task ID TaskSpec *TaskSpec / / Task specification Dependencies []string / / Pre-dependencies Successors []string / / subsequent tasks Status DAGNodeStatus / / Node status Priority int / / Upward rank EstCost time.Duration / / Estimated execution time } type DAGManager struct { nodes map[string]*DAGNode / / all nodes adjList map[string][]string / / adjacency list inDegree map[string]int / / in-degree statistics version int64 / / current version isImmutable bool / / Is it an immutable graph? } Preferably, the implementation code of the topological sorting based on Kahn algorithm is as follows: type TopologyManager struct { dagManager *DAGManager readyQueue chan *DAGNode logger logx.Logger } / / Kahn algorithm implementation func (tm *TopologyManager) ExecuteTopologicalSort(dag *DAG) error { / / 1. Calculate the in-degree of all nodes inDegree := make(map[string]int) for nodeID, node := range dag.nodes { inDegree[nodeID] = len(node.Dependencies) } / / 2. Add the node with in-degree 0 to the ready queue readyNodes := make([]*DAGNode, 0) for nodeID, degree := range inDegree { if degree == 0 { readyNodes = append(readyNodes, dag.nodes[nodeID]) } } / / 3. Start the topological sorting process processedCount := 0 totalNodes := len(dag.nodes) for len(readyNodes) > 0 { / / Process all currently ready nodes in parallel currentBatch := readyNodes readyNodes = make([]*DAGNode, 0) / / Calculate the priority and sort the tasks for the current batch tm.sortByPriority(currentBatch) / / Execute the current batch of tasks in parallel err := tm.executeBatchTasks(currentBatch) if err != nil { return fmt.Errorf("batch execution failed: %w", err) } processedCount += len(currentBatch) / / Update the in-degree of subsequent nodes for _, completedNode := range currentBatch { for _, successorID := range completedNode.Successors { inDegree[successorID]-- if inDegree[successorID] == 0 { readyNodes = append(readyNodes, dag.nodes[successorID]) } }} } / / 4. Check if all nodes have been processed (detect circular dependencies) if processedCount != totalNodes { return fmt.Errorf("cycle detected in DAG: processed %d of %d nodes", processedCount, totalNodes) } tm.logger.Infof("Successfully executed DAG with %d nodes",totalNodes) return nil } / / Sort tasks by priority func (tm *TopologyManager) sortByPriority(nodes []*DAGNode) { sort.Slice(nodes, func(i, j int) bool { return nodes[i].Priority > nodes[j].Priority / / High priority first }) } / / Execute batch tasks in parallel func (tm *TopologyManager) executeBatchTasks(nodes []*DAGNode) error{ var wg sync.WaitGroup errChan := make(chan error, len(nodes)) for _, node := range nodes { wg.Add(1) go func(n *DAGNode) { defer wg.Done() tm.logger.Infof("Executing task: %s (priority: %d)", n.TaskID,n.Priority) if err := tm.executeTask(n); err != nil { errChan <- fmt.Errorf("task %s failed: %w", n.TaskID, err) return } tm.logger.Infof("Task completed: %s", n.TaskID) }(node) } wg.Wait() close(errChan) / / Check for errors for err := range errChan { if err != nil { return err } } return nil } Preferably, the implementation code of MVCC version control is as follows: typeMVCCVersionManager struct { currentVersion *atomic.Value / / Points to the current version of DAG versionHistory map[int64]*DAG / / version history mutex sync.RWMutex logger logx.Logger } / / Copy-on-write update workflow func (vm *MVCCVersionManager) UpdateWorkflowCopyOnWrite(modifications*WorkflowModification) error { vm.mutex.Lock() defer vm.mutex.Unlock() / / 1. Get the current version currentDAG := vm.currentVersion.Load().(*DAG) / / 2. Create a deep copy newDAG := vm.deepCopyDAG(currentDAG) newVersion := time.Now().UnixNano() newDAG.version = newVersion / / 3. Apply changes if err := vm.applyModifications(newDAG, modifications); err != nil { return fmt.Errorf("failed to modify applys: %w", err) } / / 4. Verify the legitimacy of the new version if err := vm.validateDAG(newDAG); err != nil { return fmt.Errorf("invalid DAG after modifications: %w", err) } / / 5. Atomically update the version pointer vm.currentVersion.Store(newDAG) vm.versionHistory[newVersion] = newDAG / / 6. Clean up old versions (keep the latest 3 versions) vm.cleanupOldVersions() vm.logger.Infof("Workflow updated to version %d with copy-on-write",newVersion) return nil } / / Deep copy DAG func (vm *MVCCVersionManager) deepCopyDAG(originalDAG *DAG) *DAG { newDAG := &DAG{ nodes: make(map[string]*DAGNode), adjList: make(map[string][]string), inDegree: make(map[string]int), } / / Copy all nodes for nodeID, originalNode := range originalDAG.nodes { newNode := &DAGNode{ TaskID: originalNode.TaskID, TaskSpec: originalNode.TaskSpec, / / The task specification can be shared Dependencies: make([]string, len(originalNode.Dependencies)), Successors: make([]string, len(originalNode.Successors)), Status: originalNode.Status, Priority: originalNode.Priority, EstCost: originalNode.EstCost, } copy(newNode.Dependencies, originalNode.Dependencies) copy(newNode.Successors, originalNode.Successors) newDAG.nodes[nodeID] = newNode } / / Copy the adjacency list and in-degree information for nodeID, neighbors := range originalDAG.adjList { newDAG.adjList[nodeID] = make([]string, len(neighbors)) copy(newDAG.adjList[nodeID], neighbors) } for nodeID, degree := range originalDAG.inDegree { newDAG.inDegree[nodeID] = degree } return newDAG } The task management layer uses a two-phase commit protocol to ensure atomic consistency between volatile memory and persistent storage for each operation and maintenance task in the DAG operation and maintenance task graph. In the event of a failure that causes memory data loss, unfinished operation and maintenance tasks are recovered from persistent storage based on atomic consistency. Specifically, the task management layer is used to write a prepare update log to distributed storage during the prepare phase of the two-phase commit protocol when the task status of an operation and maintenance task in the DAG operation and maintenance task graph is updated. Then, during the commit phase of the two-phase commit protocol, the task status of the operation and maintenance task is atomically updated in volatile memory and persistent storage. After the update is successful, the prepare update log is deleted. In the event of a failure that causes memory data loss, unfinished operation and maintenance tasks are recovered from persistent storage based on the atomic update feature.

[0026] It is also used to support high-concurrency execution of various operation and maintenance tasks in the DAG operation and maintenance task graph through the Go coroutine-based task concurrent execution mechanism.

[0027] It should be noted that while traditional graph computing systems, such as the early Pregel, employ checkpointing mechanisms, these are typically globally blocking. Recovery may require all nodes to roll back to the previous checkpoint, resulting in a significant waste of computing resources. While subsequent optimizations have been implemented, such as incremental checkpoints or lightweight checkpoints (LWCP) that decouple vertex updates, these optimizations are often tailored to specific computing models. The task management layer in the embodiments of this application, however, implements a general atomic state recovery solution that integrates with complex workflow orchestration and supports hybrid tasks. Figure 3 Schematic diagram of the workflow of the task management layer according to the embodiment of the application. Figure 3 As shown, the asynchronous task manager (TaskManager) in the task management layer is the cornerstone of the system's stable operation, responsible for task lifecycle management, state persistence, and fault recovery. Its technical innovations include: a concurrent task execution mechanism based on Goroutines that supports high-concurrency task processing; a two-phase commit (2PC) protocol that ensures atomic consistency between memory and persistent state; a coordinated checkpointing mechanism that supports distributed fault recovery; a HEFT heuristic scheduling algorithm combined with data locality awareness for intelligent resource allocation; and a confined recovery mechanism that avoids global rollbacks and improves recovery efficiency.

[0028] Preferably, the core architecture code of the task management layer is as follows: type WorkflowEngine struct { dagManager *DAGManager / / DAG manager topologyManager *TopologyManager / / Topology sort manager versionManager *MVCCVersionManager / / Version Control Manager scheduler *HEFTScheduler / / HEFT scheduler taskManager *TaskManager / / Task Manager logger logx.Logger } type WorkflowSpec struct { ID string / / workflow unique identifier Version int64 / / Version number Type WorkflowType / / "serial" | "parallel" | "conditional" | "dag" Rollback bool / / Rollback support Tasks []*TaskSpec / / Subtask list Dependencies map[string][]string / / Task dependency graph Metadata map[string]interface{} / / metadata } type DAGNode struct { TaskID string / / Task ID TaskSpec *TaskSpec / / Task specification Dependencies []string / / Pre-dependencies Successors []string / / subsequent tasks Status DAGNodeStatus / / Node status Priority int / / Upward rank EstCost time.Duration / / Estimated execution time } type DAGManager struct { nodes map[string]*DAGNode / / all nodes adjList map[string][]string / / adjacency list inDegree map[string]int / / in-degree statistics version int64 / / current version isImmutable bool / / Is it an immutable graph? } Preferably, the implementation code of the two-phase commit protocol is as follows: / / Two-phase commit implementation of status update func (tm *TaskManager) UpdateTaskStateAtomic(taskID string, newStatusTaskStatus, result *TaskResult) error { / / Phase 1: Preparation phase - writing to the transaction log commitLog := &CommitLog{ TaskID: taskID, FinalStatus: newStatus, Result: result, Timestamp: time.Now(), LogID: utils.GenerateUUID(), } if err := tm.taskRepo.WriteCommitLog(commitLog); err != nil { return fmt.Errorf("prepare phase failed: %w", err) } / / Phase 2: Commit phase - atomic updates to memory and persistent storage tm.mutex.Lock() defer tm.mutex.Unlock() / / Update memory status if taskState, exists := tm.runningTasks.Load(taskID); exists { state := taskState.(*TaskState) state.Status = newStatus state.UpdateTime = time.Now() state.Result = result tm.runningTasks.Store(taskID, state) } / / Persist the final state if err := tm.taskRepo.CommitFinalState(commitLog); err != nil { / / Submission failed, will rely on the recovery mechanism to handle tm.logger.Errorf("Commit phase failed, will be recovered: %v", err) return err } / / Clean up the transaction log tm.taskRepo.DeleteCommitLog(commitLog.LogID) return nil } / / Resume logic when system starts func (tm *TaskManager) RecoverFromCommitLogs() error { unfinishedLogs, err := tm.taskRepo.GetUnfinishedCommitLogs() if err != nil { return fmt.Errorf("failed to get unfinished commit logs: %w", err) } for _, log := range unfinishedLogs { tm.logger.Infof("Recovering task %s from commit log", log.TaskID) / / Re-execute the submission phase if err := tm.taskRepo.CommitFinalState(log); err != nil { tm.logger.Errorf("Failed to recover task %s: %v", log.TaskID, err) continue } / / Clear the recovery log tm.taskRepo.DeleteCommitLog(log.LogID) tm.logger.Infof("Successfully recovered task %s", log.TaskID) } return nil } Preferably, the implementation code of the coordination checkpoint mechanism is as follows: type CheckpointManager struct { nodeManager *NodeManager logger logx.Logger checkpointDir string mutex sync.RWMutex } / / Initiate a global consistency checkpoint func (cm *CheckpointManager) InitiateGlobalCheckpoint(workflowIDstring) error { cm.logger.Infof("Initiating global checkpoint for workflow %s",workflowID) / / 1. Send a checkpoint preparation signal to all participating nodes nodes := cm.nodeManager.GetActiveNodes() prepareChan := make(chan error, len(nodes)) for _, node := range nodes { go func(n *NodeInfo) { prepareChan <- cm.sendPrepareCheckpoint(n, workflowID) }(node) } / / 2. Wait for all nodes to be ready for i := 0; i < len(nodes); i++ { if err := <-prepareChan; err != nil { return fmt.Errorf("node prepare failed: %w", err) } } / / 3. All nodes are ready and send a commit checkpoint signal checkpointID := fmt.Sprintf("cp_%s_%d", workflowID, time.Now().Unix()) commitChan := make(chan error, len(nodes)) for _, node := range nodes { go func(n *NodeInfo) { commitChan <- cm.sendCommitCheckpoint(n, checkpointID) }(node) } / / 4. Wait for all nodes to submit for i := 0; i < len(nodes); i++ { if err := <-commitChan; err != nil { return fmt.Errorf("node commit failed: %w", err) } } / / 5. Record global checkpoint return cm.recordGlobalCheckpoint(checkpointID, workflowID) } / / Limited recovery implementation func (cm *CheckpointManager) PerformConfinedRecovery(failedNodeIDstring) error { cm.logger.Infof("Performing confined recovery for node %s",failedNodeID) / / 1. Get the latest global checkpoint latestCheckpoint, err := cm.getLatestGlobalCheckpoint() if err != nil { return fmt.Errorf("failed to get latest checkpoint: %w", err) } / / 2. Restore only the status of the failed node if err := cm.restoreNodeFromCheckpoint(failedNodeID,latestCheckpoint); err != nil { return fmt.Errorf("failed to restore node: %w", err) } / / 3. Reschedule the running tasks on the failed node runningTasks, err := cm.getRunningTasksOnNode(failedNodeID,latestCheckpoint) if err != nil { return fmt.Errorf("failed to get running tasks: %w", err) } for _, task := range runningTasks { if err := cm.rescheduleTask(task); err != nil { cm.logger.Errorf("Failed to reschedule task %s: %v", task.ID, err) } } cm.logger.Infof("Confined recovery completed for node %s",failedNodeID) return nil } Preferably, the implementation code of the HEFT scheduling algorithm is as follows: type HEFTScheduler struct { nodeManager *NodeManager graphDB GraphDBInterface logger logx.Logger localityWeight float64 / / data locality weight } / / Task sorting phase of HEFT algorithm func (hs *HEFTScheduler) CalculateUpwardRank(dag *DAG) map[string]int{ upwardRank := make(map[string]int) visited := make(map[string]bool) / / Depth-first traversal calculation upward ranking var dfs func(nodeID string) int dfs = func(nodeID string) int { if visited[nodeID] { return upwardRank[nodeID] } visited[nodeID] = true node := dag.GetNode(nodeID) maxSuccessorRank := 0 for _, successorID := range node.Successors { successorRank := dfs(successorID) if successorRank > maxSuccessorRank { maxSuccessorRank = successorRank } } / / Upward ranking = current task estimated time + maximum subsequent path weight upwardRank[nodeID] = int(node.EstCost.Seconds()) + maxSuccessorRank return upwardRank[nodeID] } / / Calculate the upward ranking of all nodes for nodeID := range dag.nodes { dfs(nodeID)} return upwardRank } / / Processor selection phase of the HEFT algorithm func (hs *HEFTScheduler) SelectOptimalNode(task *TaskSpec,availableNodes []*NodeInfo) (*NodeInfo, error) { var bestNode *NodeInfo var minFinishTime time.Duration = time.Duration(math.MaxInt64) for _, node := range availableNodes { / / Calculate the estimated execution time of the task on this node execTime := hs.estimateExecutionTime(task, node) / / Calculate data transmission time transferTime := hs.calculateDataTransferTime(task, node) / / Calculate data locality weight localityBonus := hs.calculateLocalityBonus(task, node) / / Total cost = execution time + transmission time - locality bonus totalCost := execTime + transferTime - time.Duration(float64(localityBonus)*hs.localityWeight) / / Calculate the earliest completion time earliestStartTime := hs.getEarliestStartTime(node) finishTime := earliestStartTime + totalCost if finishTime < minFinishTime { minFinishTime = finishTime bestNode = node } } if bestNode == nil { return nil, fmt.Errorf("no suitable node found for task %s", task.ID) } hs.logger.Infof("Selected node %s for task %s (finish time: %v)", bestNode.ID, task.ID, minFinishTime) return bestNode, nil } / / Data locality awareness func (hs *HEFTScheduler) calculateLocalityBonus(task *TaskSpec, node*NodeInfo) time.Duration { bonus := time.Duration(0) / / Query the graph database metadata to obtain data sharding information shardInfo, err := hs.graphDB.GetShardInfo(task.DataRequirements) if err != nil { hs.logger.Warnf("Failed to get shard info: %v", err) return bonus } / / Calculate the local data ratio localDataRatio := hs.calculateLocalDataRatio(shardInfo, node) / / Locality reward = local data ratio * maximum reward time maxBonus := time.Duration(10 * time.Second) / / Maximum 10-second bonus bonus = time.Duration(float64(maxBonus) * localDataRatio) hs.logger.Debugf("Node %s locality bonus for task %s: %v (ratio: %.2f)", node.ID, task.ID, bonus, localDataRatio) return bonus } The conditional task execution layer is used to create a secure isolation sandbox for the execution of various operation and maintenance tasks in the DAG operation and maintenance task graph. During the execution of the current operation and maintenance task, the task execution result is evaluated based on the dynamically updated conditional function to determine the execution path of subsequent operation and maintenance tasks.

[0029] Specifically, the conditional task execution layer is used to create a Wasm security isolation sandbox through a dynamically loaded Wasm plug-in, or to create a Cgroups security isolation sandbox through a dynamically loaded Go plug-in for the execution of various operation and maintenance tasks in the DAG operation and maintenance task graph.

[0030] It is also used to evaluate the task execution results based on the dynamically updated conditional function during the execution of the current operation and maintenance task to determine the execution path of subsequent operation and maintenance tasks. The conditional function supports complex logical combinations and high-order function programming. The update method of the conditional function is hot update. The programming languages ​​of the conditional function include Rust, C++ and TinyGo.

[0031] It should be noted that Traditional graph computing systems introduce conditional branches into workflows, many of which are implemented through simple if-else logic or dedicated branch operators (such as Airflow's BranchPythonOperator). However, this brings two deep-seated problems: if the conditional judgment logic is provided by the user, its code may contain malicious operations or defects, and executing it directly on the master node or worker node will bring serious security risks; the traditional method usually hard-codes the conditional logic in the workflow definition, lacking the ability to dynamically load, update, and inject new conditional logic at runtime. The conditional task execution layer in the embodiment of the present application realizes dynamic updating and flexible expansion of conditional logic through a plug-in architecture; sandbox execution effectively reduces the security risks of user-defined code, while supporting WebAssembly multi-language programming and high flexibility. Figure 4 is a schematic diagram of the workflow of the conditional task execution layer according to the embodiment of the application, such as Figure 4As shown in the figure, the conditional task executor (ConditionalTask) in the conditional task execution layer is the core component of the system's intelligent decision-making, supporting secure dynamic conditional judgment and branch selection. Its technical innovations lie in: plug-in architecture - supporting runtime dynamic loading of WebAssembly (Wasm) and Go native plug-ins; sandbox execution - a secure isolated execution environment based on Linux namespaces and Cgroups; functional programming paradigm - supporting complex logical combinations (AND, OR, NOT) and high-order functions; hot update mechanism - supporting runtime updates of conditional logic without system restart; multi-language support - conditional functions can be written in Rust, C++, TinyGo and other languages.

[0032] Furthermore, regarding the sandboxed execution mechanism, ConditionalTask ​​places user-defined conditional functions in a strictly restricted sandbox environment when executing them: namespace isolation—creating independent PID, Network, User, and Mount namespaces; resource restrictions—limiting CPU, memory, and I / O usage through Cgroups; system call filtering—using seccomp-bpf to restrict executable system calls; and file system isolation—mounting essential system files as read-only files, disabling access to sensitive directories. Regarding dynamic plugin loading: the WebAssembly runtime—using Wasmtime to provide cross-platform, high-performance sandboxed execution; and Go native plugins—using the plugin package to dynamically load .so shared library files.

[0033] Preferably, the core architecture code of the conditional task execution layer is as follows: typeConditionalParams struct { Host string / / target host Condition ConditionFunc / / Condition evaluation function CheckTask *TaskSpec / / Check task TrueTask *TaskSpec / / Execute when the condition is true FalseTask *TaskSpec / / Executed when the condition is false Description string / / Task description PluginPath string / / plugin path (Wasm or .so) SandboxConfig *SandboxConfig / / Sandbox configuration ResourceLimits *ResourceLimits / / Resource limits } / / Conditional function type definition type ConditionFunc func(*agent.ShellResult) bool / / Sandbox configuration type SandboxConfig struct { EnableNetworkIsolation bool / / Network isolation EnableFileSystemIsolation bool / / File system isolation AllowedSyscalls []string / / Allowed system calls TimeoutSeconds int / / Execution timeout } / / Sandbox executor interface type SandboxExecutor interface { Execute(ctx context.Context, pluginPath string, input []byte) ([]byte, error) CreateSandbox(config SandboxConfig) (*Sandbox, error) DestroySandbox(sandbox *Sandbox) error } Preferably, the implementation code of the sandbox execution mechanism of the conditional task is as follows: / / Sandbox implementation based on Cgroups and Namespace type CgroupSandbox struct { cgroupPath string sandboxDir string config SandboxConfig logger logx.Logger } func (s *CgroupSandbox) Execute(ctx context.Context, pluginPathstring, input []byte) ([]byte, error) { sandboxID := utils.GenerateUUID() / / 1. Create and configure Cgroup cgroupName := fmt.Sprintf("sandbox-%s", sandboxID) if err := s.createCgroup(cgroupName); err != nil { return nil, fmt.Errorf("failed to create cgroup: %w", err) } defer s.cleanupCgroup(cgroupName) / / 2. Set resource limits if err := s.configureCgroupLimits(cgroupName); err != nil { return nil, fmt.Errorf("failed to configure cgroup limits: %w", err) } / / 3. Create a sandbox process cmd := exec.CommandContext(ctx, pluginPath) / / Set up a separate namespace cmd.SysProcAttr = &syscall.SysProcAttr{ Cloneflags: syscall.CLONE_NEWPID | / / Process namespace syscall.CLONE_NEWNET | / / Network namespace syscall.CLONE_NEWUSER | / / User namespace syscall.CLONE_NEWNS | / / Mount namespace syscall.CLONE_NEWUTS | / / UTS namespace syscall.CLONE_NEWIPC, / / IPC namespace / / User namespace mapping (run as non-root user) UidMappings: []syscall.SysProcIDMap{ {ContainerID: 0, HostID: 65534, Size: 1}, / / Mapped to user nobody }, GidMappings: []syscall.SysProcIDMap{ {ContainerID: 0, HostID: 65534, Size: 1}, / / Mapped to the nobody group }, } / / 4. Configure seccomp system call filter if err := s.configureSeccomp(cmd); err != nil { return nil, fmt.Errorf("failed to configure seccomp: %w", err) } / / 5. Set up input and output cmd.Stdin = bytes.NewReader(input) var stdout, stderr bytes.Buffer cmd.Stdout = &stdout cmd.Stderr = &stderr / / 6. Start the process and add it to the Cgroup if err := cmd.Start(); err != nil { return nil, fmt.Errorf("failed to start sandbox process: %w", err) } if err := s.addProcessToCgroup(cgroupName, cmd.Process.Pid); err !=nil { cmd.Process.Kill() return nil, fmt.Errorf("failed to add process to cgroup: %w", err) } / / 7. Wait for execution to complete if err := cmd.Wait(); err != nil { s.logger.Errorf("Sandbox execution failed: %v, stderr: %s", err,stderr.String()) return nil, fmt.Errorf("sandbox execution failed: %w", err) } return stdout.Bytes(), nil } / / Create Cgroup func (s *CgroupSandbox) createCgroup(name string) error { cgroupPath := filepath.Join(" / sys / fs / cgroup", name) / / Create Cgroup directory if err := os.MkdirAll(cgroupPath, 0755); err != nil { return fmt.Errorf("failed to create cgroup directory: %w", err) } s.cgroupPath = cgroupPath return nil } / / Configure Cgroup resource limits func (s *CgroupSandbox) configureCgroupLimits(name string) error { cgroupPath := s.cgroupPath / / Set memory limit (default 64MB) memoryLimit := s.config.MemoryLimitMB if memoryLimit == 0 { memoryLimit = 64 } if err := s.writeCgroupFile(cgroupPath, "memory.limit_in_bytes", fmt.Sprintf("%d", memoryLimit*1024*1024)); err != nil { return err } / / Set CPU limit (default 0.1 core) cpuQuota := s.config.CPUQuotaPercent if cpuQuota == 0 { cpuQuota = 10 / / 10% } if err := s.writeCgroupFile(cgroupPath, "cpu.cfs_quota_us", fmt.Sprintf("%d", cpuQuota*1000)); err != nil {<( return err } if err := s.writeCgroupFile(cgroupPath, "cpu.cfs_period_us", "100000"); err != nil { return err } / / Set IO limit It should be noted that there is a misspelling in the original text where " " and "<( " are incorrect tags. They should be " " and " " respectively. The above translation has made corrections based on the correct understanding of the text.if err := s.writeCgroupFile(cgroupPath, "blkio.throttle.read_bps_device", "8:0 10485760"); err != nil { / / 10MB / s read limit return err } return nil } Preferably, the WebAssembly plug-in runtime provides more advanced sandbox protection, and its implementation code is as follows: type WasmSandbox struct { engine *wasmtime.Engine store *wasmtime.Store config WasmConfig logger logx.Logger } func (ws *WasmSandbox) Execute(ctx context.Context, wasmPath string,input []byte) ([]byte, error) { / / 1. Configure the Wasm engine config := wasmtime.NewConfig() config.SetCraneliftOptLevel(wasmtime.OptLevelNone) / / Disable optimization to improve security config.SetConsumeFuel(true) / / Enable fuel consumption mechanism engine := wasmtime.NewEngineWithConfig(config) store := wasmtime.NewStore(engine) / / 2. Set resource limits store.AddFuel(ws.config.MaxInstructions) / / Maximum number of instructions / / 3. Load the Wasm module wasmBytes, err := os.ReadFile(wasmPath) if err != nil { return nil, fmt.Errorf("failed to read wasm file: %w", err) } module, err := wasmtime.NewModule(engine, wasmBytes) if err != nil { return nil, fmt.Errorf("failed to compile wasm module: %w", err) } / / 4. Create restricted import functions imports := ws.createLimitedImports(store) / / 5. Instantiate the module instance, err := wasmtime.NewInstance(store, module, imports) if err != nil { return nil, fmt.Errorf("failed to instantiate wasm module: %w", err) } / / 6. Get the exported conditional function conditionFunc := instance.GetFunc(store, "evaluate_condition") if conditionFunc == nil { return nil, fmt.Errorf("condition function not found in wasm module") } / / 7. Set timeout execution resultChan := make(chan []byte, 1) errorChan := make(chan error, 1) go func() { defer func() { if r := recover(); r != nil { errorChan <- fmt.Errorf("wasm execution panic: %v", r) } }() / / Execute conditional function result, err := ws.executeConditionFunction(store, conditionFunc,input) if err != nil { errorChan <- err return } resultChan <- result }() / / 8. Wait for result or timeout select { case result := <-resultChan: return result, nil case err := <-errorChan: return nil, err case <-ctx.Done(): return nil, fmt.Errorf("wasm execution timeout") } } / / Create restricted import function func (ws *WasmSandbox) createLimitedImports(store *wasmtime.Store)[]*wasmtime.Extern { var imports []*wasmtime.Extern / / Only provide safe logging functions logFunc := wasmtime.NewFunc(store, wasmtime.NewFuncType( []*wasmtime.ValType{wasmtime.NewValType(wasmtime.KindI32)}, []*wasmtime.ValType{}, ), func(caller *wasmtime.Caller, args []wasmtime.Val) ([]wasmtime.Val, *wasmtime.Trap) { / / Secure logging to prevent malicious output ws.logger.Infof("Wasm log: %d", args[0].I32()) return []wasmtime.Val{}, nil }) imports = append(imports, logFunc.AsExtern()) / / You can add other safe import functions as needed return imports } Preferably, the implementation code of the dynamic plug-in hot update mechanism is as follows: type PluginManager struct { pluginStore map[string]*Plugin versionStore map[string]int64 updateChan chan PluginUpdateEvent mutex sync.RWMutex logger logx.Logger } / / Hot update plugin func (pm *PluginManager) UpdatePlugin(pluginID string, newVersion []byte) error { pm.mutex.Lock() defer pm.mutex.Unlock() / / 1. Verify the new plugin if err := pm.validatePlugin(newVersion); err != nil { return fmt.Errorf("plugin validation failed: %w", err) } / / 2. Create a new version newVersionID := time.Now().Unix() tempPath := fmt.Sprintf(" / tmp / plugin_%s_v%d", pluginID, newVersionID) if err := os.WriteFile(tempPath, newVersion, 0644); err != nil { return fmt.Errorf("failed to write plugin file: %w", err) } / / 3. Atomic update oldPlugin := pm.pluginStore[pluginID] newPlugin := &Plugin{ ID: pluginID, Version: newVersionID, Path: tempPath, LoadTime: time.Now(), } pm.pluginStore[pluginID] = newPlugin pm.versionStore[pluginID] = newVersionID / / 4. Clean up old versions if oldPlugin != nil { go func() { time.Sleep(30 * time.Second) / / Wait for the running task to complete os.Remove(oldPlugin.Path) pm.logger.Infof("Cleaned up old plugin version: %s", oldPlugin.Path) }() } pm.logger.Infof("Plugin %s updated to version %d", pluginID,newVersionID) return nil } In some specific embodiments, Figure 5 This is a structural diagram of the operation and maintenance task workflow orchestration system according to a specific embodiment of the present application. Figure 5 As shown, the system includes a user interface layer, a storage layer, a resource management layer and a graph database cluster; The user interface layer is used to provide multiple interaction methods for users to input operation and maintenance task workflow specifications. The interaction methods include web console, REST API and command line tools.

[0034] The storage layer contains persistent distributed storage units; the storage layer is used to persistently store data related to operation and maintenance tasks, and supports transaction logs and checkpoint mechanisms; the resource management layer is used to intelligently schedule hardware resources for the execution of operation and maintenance tasks to perform load balancing.

[0035] A graph database cluster is a cluster of the NebulaGraph distributed graph database, including Meta nodes, Graph nodes, and Storage nodes.

[0036] This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above system embodiments.

[0037] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0038] Optionally, the electronic device may also include a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an operation and maintenance task workflow orchestration system is implemented. The display screen of the electronic device may be a liquid crystal display or an electronic ink display screen, and the input device of the electronic device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse.

[0039] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0040] In addition, in conjunction with the operation and maintenance task workflow orchestration system in the above embodiments, embodiments of the present application may provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any of the operation and maintenance task workflow orchestration systems in the above embodiments.

[0041] In one embodiment, Figure 6 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 6 As shown, an electronic device is provided, which can be a server, and its internal structure can be as shown in FIG. Figure 6 As shown. The electronic device includes a processor, a network interface, an internal memory, and a non-volatile memory connected via an internal bus, wherein the non-volatile memory stores an operating system, a computer program, and a database. The processor is used to provide computing and control capabilities, the network interface is used to communicate with external terminals via a network connection, the internal memory is used to provide an environment for the operation of the operating system and computer program, and when the computer program is executed by the processor, it implements an operation and maintenance task workflow orchestration system. The database is used to store data.

[0042] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0043] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiment systems can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0044] Those skilled in the art should understand that the various technical features of the above-described embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0045] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A system for orchestrating operation and maintenance task workflows in a distributed graph database, characterized in that: The system includes a workflow orchestration layer, a task management layer and a conditional task execution layer; The workflow orchestration layer is used to receive the operation and maintenance task workflow specification input by the user and recursively parse the workflow specification to generate an executable DAG operation and maintenance task graph; The task management layer is used to ensure the atomic consistency of each operation and maintenance task in the DAG operation and maintenance task graph in volatile memory and persistent storage using a two-phase commit protocol. In the event of a failure resulting in memory data loss, unfinished operation and maintenance tasks are recovered from persistent storage based on the atomic consistency. The conditional task execution layer is used to create a secure isolation sandbox for the execution of various operation and maintenance tasks in the DAG operation and maintenance task graph. During the execution of the current operation and maintenance task, the task execution result is evaluated based on the dynamically updated conditional function to determine the execution path of subsequent operation and maintenance tasks.

2. The system according to claim 1, wherein: The workflow orchestration layer is used to receive the operation and maintenance task workflow specification input by the user, and uniformly abstract the serial branches, parallel branches and conditional branches in the workflow specification into a directed acyclic graph. The Kahn algorithm is then used to parse the dependency and execution order of the operation and maintenance tasks, so as to topologically sort the directed acyclic graph to generate an executable DAG operation and maintenance task graph.

3. The system according to claim 2, characterized in that The workflow orchestration layer is used to introduce multi-version concurrency control through the MVCC version control mechanism to support workflow specification updates during operation and maintenance task execution: During the execution of an operation and maintenance task, if the workflow specification is updated, a new version of the DAG operation and maintenance task graph is copied and recursively parsed to generate the new version. Then, atomic.SwapPointer is used to perform an atomic update of the version, so that the operation and maintenance task being executed continues to use the old version of the DAG operation and maintenance task graph, and subsequent unexecuted operation and maintenance tasks use the new version of the DAG operation and maintenance task graph.

4. The system according to claim 1, wherein: The task management layer is configured to, when a task status update occurs for an operation and maintenance task in the DAG operation and maintenance task graph, first write a preparation update log to the distributed storage during the preparation phase of the two-phase commit protocol, and then atomically update the task status of the operation and maintenance task in the volatile memory and the persistent storage during the commit phase of the two-phase commit protocol, and delete the preparation update log after the update is successful; In the event of a failure resulting in memory data loss, unfinished operation and maintenance tasks are recovered from persistent storage based on the atomic update feature.

5. The system according to claim 4, characterized in that The task management layer is used to support high-concurrency execution of various operation and maintenance tasks in the DAG operation and maintenance task graph through a concurrent task execution mechanism based on Go coroutines.

6. The system according to claim 1, wherein: The conditional task execution layer is used to create a Wasm security isolation sandbox through a dynamically loaded Wasm plug-in, or to create a Cgroups security isolation sandbox through a dynamically loaded Go plug-in, for the execution of various operation and maintenance tasks in the DAG operation and maintenance task graph.

7. The system according to claim 6, characterized in that The conditional task execution layer is used to evaluate the task execution results based on the dynamically updated conditional function during the execution of the current operation and maintenance task to determine the execution path of subsequent operation and maintenance tasks. The conditional function supports complex logical combinations and high-order function programming. The update method of the conditional function is hot update. The programming languages ​​of the conditional function include Rust, C++ and TinyGo.

8. The system according to claim 1, wherein: The system includes a user interface layer; The user interface layer is used to provide multiple interaction methods for users to input operation and maintenance task workflow specifications, wherein the interaction methods include a web console, REST API and command line tools.

9. The system according to claim 1, wherein: The system includes a storage layer and a resource management layer, wherein the storage layer includes a persistent distributed storage unit; The storage layer is used to persistently store data related to operation and maintenance tasks and supports transaction logs and checkpoint mechanisms; The resource management layer is used to intelligently schedule hardware resources for the execution of operation and maintenance tasks to perform load balancing.

10. The system according to claim 1, wherein: The system includes a graph database cluster, which is a cluster of NebulaGraph distributed graph database.

Images