An operation and maintenance task flow deduplication method and system based on execution trajectory

Abstract

The application discloses an operation and maintenance task flow deduplication method and system based on execution track, which comprises the following steps: S1, analyzing the target task flow definition, extracting the DAG structure feature, the operation object feature and the data flow feature, and excluding the natural language text; S2, normalizing the multi-dimensional features into execution track vectors and generating hash fingerprints; S3, obtaining a similar candidate set through the hash fingerprint coarse screening, calculating the DAG track edit distance of the candidate set and mapping it into a normalized similarity; S4, if the similarity is lower than a threshold value, storing it as a new master edition, if the similarity is not lower than the threshold value, locating the difference node; S5, integrating the difference node into the master edition task flow as an optional branch to complete the merging; the application also provides a system for realizing the above method; the application does not depend on the text semantics, takes the structured execution logic as the comparison basis, can effectively reduce the storage and retrieval cost, avoids the text ambiguity and model illusion risk, and realizes the continuous sedimentation and reuse of operation and maintenance knowledge.

Description

Technical Field

[0001] This invention relates to the fields of intelligent operations and maintenance (AIOps) and workflow management technology, specifically to a method and system for deduplicating and merging massive, heterogeneous task flows in an intelligent agent platform for operations and maintenance scenarios. This invention particularly focuses on efficient and accurate equivalence determination and merging based on the structured characteristics of the actual execution logic of the task flows, rather than relying on textual semantics. Background Technology

[0002] In intelligent agent platforms designed for operations and maintenance (O&M) scenarios, frontline O&M personnel typically describe their operational intentions in natural language. The platform backend then automatically parses and generates executable O&M task flows based on these instructions. These task flows consist of several execution nodes based on dependencies and are usually defined using a Directed Acyclic Graph (DAG) data structure. They are used to perform specific inspection or O&M operations such as "checking for MySQL deadlocks".

[0003] As the platform's user base grows and usage frequency increases, the system will accumulate tens of thousands of task flow definitions. However, operational data shows that a very high proportion of these task flows are redundant. Due to the different terminology and descriptive granularity used by different creators, many task flows that are completely equivalent or highly similar in execution logic have significantly different names and natural language descriptions. For example, the same operational operation of "checking database deadlocks" might be named "Check Deadlocks," "View Locked Tables," or "Troubleshoot Lock Waits" respectively.

[0004] To address the aforementioned issues, existing technical solutions primarily rely on textual semantic comparison of task flow names or their natural language descriptions to identify repetitive tasks. However, in large-scale task flow management scenarios geared towards operations and maintenance, this approach suffers from the following significant drawbacks and technical challenges: (1) Storage and retrieval overhead expands linearly with redundant data: Logically equivalent task flows are stored as independent entities, resulting in a waste of storage resources. When the upper-level automated process needs to call specific operation and maintenance tasks, its retrieval space will expand rapidly due to a large number of redundant candidates, leading to system response delays and causing the "selection difficulty" problem, that is, it is difficult to effectively distinguish and select the best among multiple equivalent candidate task flows.

[0005] (2) Insufficient reliability of text-based semantic comparison: The phenomenon of "synonymous" between task flow names and descriptions is common. Relying solely on text similarity algorithms can easily classify equivalent tasks as different entities. If a Large Language Model (LLM) is introduced for semantic understanding to improve robustness, it is susceptible to the influence of model illusion, which may incorrectly classify task flows with different execution logic as similar or identical, resulting in unstable comparison results that cannot be used as a reliable basis for deduplication operations.

[0006] (3) Implicit knowledge is difficult to accumulate and reuse efficiency is low: Due to the lack of an effective perception mechanism for existing equivalent task flows in the platform, newly joined operation and maintenance personnel cannot learn about the mature task logic that can be reused and tend to create it repeatedly. This not only wastes human resources, but also makes it impossible for operation and maintenance knowledge in the platform to be effectively accumulated and continuously optimized.

[0007] In summary, there is an urgent need for a method and system for deduplicating operation and maintenance task flows based on execution trajectories to overcome the aforementioned technical deficiencies. Summary of the Invention

[0008] To overcome the above problems, this invention provides a method and system for deduplicating operation and maintenance task flows based on execution trajectories. Its core concept lies in mapping the "task flow comparison" problem from the text space to the execution logic space, and achieving semantically independent equivalence determination and merging by extracting the structured execution trajectory features of the task flows.

[0009] I. Technical Solution: This invention provides a method for deduplicating operation and maintenance task flows based on execution trajectories, comprising the following steps: S1, Steps for trajectory analysis and multidimensional feature extraction: Obtain the definition of the target task flow to be processed, parse its internal execution logic, and extract multi-dimensional structured features. These features include at least: DAG structure features, i.e., a graph structure with execution nodes as vertices and execution dependencies between nodes as directed edges; operation object features, i.e., the database objects (such as databases and tables) operated on by each execution node; and data flow features, i.e., the data fields flowing between nodes. This step explicitly excludes the collection and parsing of task flow names and natural language description text.

[0010] S2, Perform trajectory vectorization and hash fingerprint generation steps: The multidimensional features extracted in step S1 are normalized to eliminate representation differences at the non-execution logic level, and the normalized features are encoded into structured execution trajectory vectors, denoted as... V At the same time, based on VA hash fingerprint of the task flow is generated by hashing a predefined subset of stable features, denoted as . H .

[0011] The stable feature subset includes at least: a normalized set of operation objects (containing a unified identifier for all involved database names and table names) and a sequence of key node types (i.e., a sequence of node operation types arranged in topological order).

[0012] Preferably, the topology order normalization process adopts a depth-first traversal topology sorting algorithm. During the traversal, multiple successor child nodes of the same node are arranged in lexicographical order according to the node type name and then traversed sequentially. For the case of the same node type, the order in which the nodes appear in the original task flow definition is used as the sorting basis to ensure the complete determinism of the topology order.

[0013] The hash calculation employs the SimHash algorithm (a locality-sensitive hashing algorithm), which encodes the stable feature subset into a fixed-length binary fingerprint. By leveraging the locality sensitivity of SimHash, the hash fingerprints generated by task flows with similar execution logic have a small Hamming distance, thus ensuring they fall into the same hash bucket during coarse screening. Alternatively, the MD5 (Message Digest Algorithm 5) algorithm can be used to precisely match task flows with completely equivalent execution logic.

[0014] When using the SimHash algorithm, the method for encoding the stable feature subset into a SimHash input feature vector is as follows: First, each node type in the key node type sequence is mapped to a preset unique numerical identifier, forming a sequence of length [number missing]. L , where L This represents the total number of nodes in the task flow; Secondly, each library name and table name in the set of operation objects is mapped to a preset unique numerical identifier, which is then concatenated with the beginning and end of the above node type sequence to form a unified one-dimensional numerical sequence. The library names and table names in the set of operation objects are sorted in lexicographical order before concatenation to ensure that the concatenation order generated from the same set at any given time is unique and deterministic. When library names and table names are sorted in a mixed manner, they are arranged in order of library name priority followed by table name priority. Then, the one-dimensional numerical sequence is input into the SimHash algorithm to generate a fixed-length (e.g., 64-bit or 128-bit) binary fingerprint using a preset deterministic random projection matrix or a weighted summation method; the dimension of the deterministic random projection matrix is... d × ,in The length of the one-dimensional numerical sequence is... d The number of bits (64 or 128) is used to output the hash fingerprint. The element values ​​of this matrix are generated by a pseudo-random number generator using a seed parameter during system initialization and remain fixed after generation. The seed parameter is set during system deployment and kept consistent throughout the system to ensure that the SimHash fingerprints generated by the same task flow are exactly the same at any time.

[0015] The weights of each feature dimension are equal, or different weight coefficients are preset according to the feature type (for example, the weight of node type sequence is 1, and the weight of operation object feature is 1.5, so as to enhance the distinguishability of database operation objects).

[0016] The weight coefficients are applied in the weighted summation stage of the SimHash algorithm: the value of each element in the one-dimensional numerical sequence is multiplied by the weight coefficient of the corresponding feature type, and then random projection or weighted summation is performed. The feature type with a larger weight coefficient contributes more to the final hash fingerprint, thereby making task flows with the same key operation object characteristics more likely to fall into the same hash bucket in the coarse screening stage.

[0017] S3, Steps for calculating two-level similarity: When it is necessary to perform deduplication on the target task flow, execute the following sub-steps: S31, Hash Coarse Screening: Under the default configuration, the SimHash algorithm is used to generate hash fingerprints and perform coarse screening; based on the hash fingerprint of the target task flow... For indexing, a search is performed in a pre-built historical trajectory feature database. The search criteria are identical fingerprints or falling into the same hash bucket. The hash bucket partitioning rule is as follows: when using the SimHash algorithm, the first few fingerprints are used as the index. K The position serves as a bucket identifier, before the fingerprint. K All task streams with the same bit value fall into the same hash bucket, where K It is a positive integer, and its value is determined by the following formula: ,in N This represents the total number of historical task flows in the trajectory feature library. The preset expected candidate set size, The value range is 1 to 100; when using the MD5 algorithm, the complete MD5 value is used as the bucket identifier, and only tasks with completely identical fingerprints flow into the same hash bucket; historical task flows with the same fingerprint as the target task flow or falling into the same hash bucket are retrieved from the trajectory feature library, and the retrieval results constitute a similarity candidate set, denoted as . C .

[0018] Preferably, the expected candidate set size The value is determined based on the size of the trajectory feature database.N Determined by the following piecewise function: ; The above values ​​were determined by performing a grid search on a test set containing known equal / non-equivalent task flow samples to ensure that the recall rate in the coarse screening stage is not less than 95% and the average size of the candidate set does not exceed the preset limit. value.

[0019] S32, Trajectory Edit Distance Calculation: If candidate set C If not empty, then... C For each historical task flow, calculate its DAG trajectory edit distance with the target task flow, denoted as . D The distance calculation process includes: expanding the DAG structure of the two task flows into a node sequence according to a normalized topological order; performing local alignment of the sequence using a sliding window; defining the node replacement cost based on the feature consistency of nodes in three dimensions: structure, operation object, and data flow; accumulating the minimum cost of insertion, deletion, and replacement operations required to transform one trajectory into another; and finally calculating the edit distance. D The mapping is the normalized similarity, denoted as S .

[0020] S4, Threshold determination and differential node localization steps: Compare the similarity S With preset threshold : like S < If the target task flow does not overlap with the historical task flow, it is stored as a new master task flow in the trajectory feature library. like S ≥ If the target task flow is determined to be a duplicate of the historical master task flow with the highest similarity, and the set of difference nodes in the two trajectories is located and output according to the node correspondence of the sliding window alignment in step S32.

[0021] Preferably, the preset threshold The value is determined in the following way: During the platform deployment phase, data is collected. q Given pairs of equivalent and non-equivalent task flow samples, calculate their similarity respectively. S The threshold that maximizes the Youden Index is selected as the initial default value of the system based on the Receiver Operating Characteristic (ROC) curve. q ≥1000; Alternatively, the threshold The system parameters are configured to be manually adjusted by the system administrator through the management interface. The system provides real-time feedback on the distribution of candidate matching queues under the current threshold, which can be verified and optimized by the administrator.

[0022] S5, Merge suggestion generation and merging steps: A merge suggestion containing the set of differences is generated and pushed to the user. In response to the user's confirmation, the differences are merged as optional branches into the historical master task flow with the highest similarity, thus completing the deduplication and merging.

[0023] Preferably, in step S1, the normalization process includes: performing topological order normalization on the DAG nodes, and performing unified alias mapping on the database objects and field identifiers.

[0024] Preferably, the DAG trajectory editing distance D The calculation is performed in the following way: Let the target task flow trajectory sequence be The historical task flow trajectory sequence is ,in ( i =1,2,…, m )and ( j =1,2,…, n ) represents nodes arranged in topological order.

[0025] Define the node replacement cost function If node and The features in the following three dimensions are completely consistent after normalization: "Completely consistent" means that the string or numerical representations of the features in each dimension are exactly the same after normalization.

[0026] (a) DAG structural features: including node type, out-degree, and in-degree; (b) Characteristics of the operation object: including the database name and table name being operated on; (c) Data flow characteristics: including input field list and output field list, ,otherwise The cost of insertion and deletion operations is always 1.

[0027] Edit distance D Calculated using the dynamic programming recurrence relation: ; in Represents a sequence A The former i Elements and sequences B The former jThe minimum edit cost between elements, ultimately .

[0028] The normalized similarity S The calculation formula is: ; in Indicates the length of the two sequences m and n The maximum value in.

[0029] Secondly, this invention also provides a deduplication system for operation and maintenance task flows based on execution trajectories, used to implement the method described in any of the above-mentioned embodiments. The system includes: Trajectory Feature Extraction Module: Used to execute steps S1 and S2, receive the task flow definition, parse it, and output its execution trajectory vector. V and hash fingerprint H .

[0030] Trajectory Feature Library: Used to persistently store execution trajectory vectors of historical task flows. V and hash fingerprint H It also maintains the ownership relationship between each task flow and the parent task flow.

[0031] Two-level similarity calculation module: used to execute step S3, which contains a coarse screening unit and a fine calculation unit, respectively realizing the candidate set delineation based on hash and the precise similarity calculation based on edit distance.

[0032] Conflict resolution and merging module: Used to execute steps S4 and S5, responsible for threshold determination, difference node location, merging suggestion generation, and merging and updating the master task flow based on user feedback.

[0033] When the master task flow is merged and an optional branch is added, the system asynchronously recalculates the execution trajectory vector of that master. V and hash fingerprint H The new version overwrites the old records in the trajectory feature database. To maintain the consistency of the coarse-screened index, the task identifier of the parent version is retained in the hash bucket corresponding to the old fingerprint, while the new fingerprint is written to a new hash bucket, ensuring that the parent version can be retrieved regardless of which fingerprint is hit in subsequent searches. Old index entries are marked as expired and are asynchronously cleaned up during system off-peak periods. When the expired index entries are cleaned up, the new fingerprint bucket position of the parent version is recorded in the corresponding bucket key, so that requests for retrieval with the old fingerprint can be redirected to the hash bucket where the new fingerprint is located, ensuring that the parent version can still be retrieved after cleanup.

[0034] The beneficial effects of this invention are as follows: (1) By merging equivalent task flows into a structure of "master + optional difference branches", redundant independent storage is avoided, and storage space is greatly saved. At the same time, the retrieval space after merging is significantly narrowed, improving the efficiency of the upper layer process in finding and calling the target task flow.

[0035] (2) The similarity determination of the present invention is based entirely on the structured execution trajectory, which is unrelated to natural language text. This fundamentally eliminates misjudgments caused by word differences and avoids the uncertainty of results brought about by LLM, making the deduplication results highly stable and reliable.

[0036] (3) The two-level computing architecture of “hash coarse screening + edit distance fine calculation” is adopted, which limits the huge cost of sequence alignment calculation to only a very small number of candidate sets, avoids full database comparison, and enables the method to maintain high performance under the task flow base database of tens of thousands or even larger scale.

[0037] (4) Through the mechanism of merging different nodes into the master version, the platform can continuously accumulate and evolve high-quality master version task flow. New users can quickly discover and reuse existing logic, avoiding the repetitive work of "reinventing the wheel", and realizing the explicit accumulation and continuous iteration of operation and maintenance experience. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the architecture and data flow of the operation and maintenance task flow deduplication system based on execution trajectory of the present invention; Figure 2 This is a schematic diagram of the overall process of the operation and maintenance task flow deduplication method based on execution trajectory of the present invention; Figure 3 for Figure 2 A flowchart illustrating the actuarial stage of the two-level similarity calculation. Figure 4 This is a schematic diagram illustrating the principle of difference node positioning and master task flow merging in this invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] This embodiment uses the automated operation and maintenance platform of a large internet company as an application scenario. This platform stores more than 10,000 historical operation and maintenance task flow definitions. The task flow is defined in JSON (JavaScript Object Notation) or YAML (YAML Ain't Markup Language, a data serialization format) format, containing a list of nodes and a list of edges. The nodes define the specific operation type (such as executing SQL scripts, calling APIs, sending notifications, etc.) and its parameters.

[0041] I. System Module Construction: First, construct the following core modules: Trajectory Feature Extraction Module: This module is an independent microservice. Its input is the original definition text of the task flow, and its output is the execution trajectory vector of that task flow. V and hash fingerprint H Internally, it contains a parser and an encoder. The parser is responsible for resolving the original definition into a DAG object in memory, and the encoder is responsible for traversing the DAG and generating feature vectors.

[0042] Trajectory feature library: Implemented using a distributed NoSQL database (such as Elasticsearch or Milvus) with support vector retrieval. It stores unique identifiers for historical task flows. Its execution trajectory vector V Hash fingerprint H and the identifier of its parent task flow. (If it is itself the master version, then) equal ).

[0043] Preferably, when SimHash is used as the fingerprint algorithm, the trajectory feature library extracts the fingerprint of each historical task flow before the index construction stage. K The bit is used as the bucket key, and a secondary index is built based on the bucket key. This allows the coarse screening stage to directly locate the corresponding hash bucket through the bucket key, thereby achieving constant-level (O(1)) index lookup overhead for candidate set retrieval.

[0044] Two-level similarity calculation module: This module is a stateless calculation service that receives the target task flow vector. and fingerprints It initiates a two-level query to the feature database and returns the similarity of the best match. S And alignment details.

[0045] Conflict Resolution and Merging Module: This module is a backend service responsible for executing decision-making logic and interacting with the frontend interface via API. It receives the output from the actuarial module, performs threshold judgment, and if merging is required, generates visual data highlighting the differences based on alignment details, pushes it to the user interface, and listens for user confirmation or rejection events.

[0046] The core data interfaces between the modules are defined as follows: (1) Output interface of trajectory feature extraction module: The data record written to the trajectory feature library contains the following fields: unique task identifier Its execution trajectory vector V Hash fingerprint H and the master task flow identifier to which it belongs. (Initial write) and same); (2) The two-level similarity calculation module and the retrieval interface of the trajectory feature library: the coarse screening stage uses the hash fingerprint of the target task flow. As input, retrieve the candidate set. C , C Includes a unique identifier for each candidate task. Execution trajectory vector V and master label The actuarial phase uses the target task flow vector. With candidate task flow vector As input, calculate and return the normalized similarity. S and alignment details, which include a mapping table of correspondences between nodes in the two trajectories; (3) Output interface of conflict resolution and merging module: The merging suggestion includes the following data field - master task flow identifier Target task flow identifier Normalized similarity S A list of difference nodes, wherein each difference node in the list includes a node identifier, node type, difference description, comparison results of the current task flow parameter key-value pairs and the master task flow parameter key-value pairs, and preset merge operation options (accept / reject / merge after editing).

[0047] The composition architecture and data flow direction of the above modules are as follows: Figure 1 As shown.

[0048] II. Method Implementation Process: The following section uses a specific example of a "Check MySQL deadlock" task flow to illustrate the method and steps in detail: Step S1, perform trajectory analysis and multidimensional feature extraction: Suppose the platform receives a newly created target task flow Task_X, whose definition only contains the user-inputted descriptive text "Check lock table status" and a structured node definition.

[0049] 1. The trajectory feature extraction module receives the definition of Task_X and parses its nodes and edges fields. The reconstructed DAG structure contains three nodes: Node_A (connects to the database), Node_B (executes SQL (Structured Query Language) query SHOW ENGINE INNODB STATUS, and this query node has a timeout parameter timeout=30s set), and Node_C (parses the results and outputs them).

[0050] 2. The extraction module traverses the DAG: DAG structure characteristics: Records the directed edge relationships between nodes, such as Node_A→Node_B→Node_C.

[0051] Operation target characteristics: Node_B was detected as a database operation node, and its target database name prod_db and table name were extracted (this operation has no specific table and is considered empty). For Node_A, the database instance identifier mysql-master-01 that it is connected to was extracted.

[0052] Data flow characteristics: Data flow from Node_B to Node_C was detected, and the field of the flow was extracted as raw_log.

[0053] Step S2, perform trajectory vectorization and hash fingerprint generation: 1. Normalization Processing: The encoder normalizes the above features. For example, it converts database names to lowercase and maps database instance names to their type aliases (e.g., mysql-master-01 is mapped to MySQL_MASTER).

[0054] The topology order normalization adopts a depth-first traversal (DFS) topology sorting algorithm. During the traversal, multiple successor child nodes of the same node are arranged in lexicographical order according to the node type name and then traversed sequentially, thereby ensuring that the node sequence order generated by the same DAG at any time is unique and deterministic. For the case of the same node type, the order in which the nodes appear in the original task flow definition is used as the third-level sorting key to ensure the complete determinism of the sorting result.

[0055] 2. Vectorization: The normalized DAG structure, the set of operation objects, and the set of data flow fields are one-hot encoded and concatenated to form a high-dimensional sparse vector. .

[0056] 3. Generate fingerprint: Encoder selection The most distinctive stable feature subset, namely the operation object features (prod_db and MySQL_MASTER) and the key node sequence types (DB_CONNECT > SQL_QUERY > OUTPUT), is used to generate a fixed-length hash fingerprint according to the hash calculation method defined below. .

[0057] The hash calculation is performed using one of the following two methods: (a) In the conventional deduplication process, the SimHash algorithm is used to encode the stable feature subset into a fixed-length binary fingerprint. By utilizing the locality sensitivity of SimHash, the hash fingerprints generated by task flows with similar execution logic have a small Hamming distance, so that they can fall into the same hash bucket when performing coarse screening. (b) When the system is configured to perform exact deduplication matching, the MD5 algorithm is used to generate a fixed-length hash digest for exact matching of task flows that are logically completely equivalent.

[0058] This embodiment uses the SimHash algorithm to generate a fixed-length hash fingerprint. .

[0059] Step S3, two-level similarity calculation. The two-level calculation process in this step is as follows: Figure 2 As shown, the detailed calculation process in the actuarial stage is as follows: Figure 3 As shown: S31 Hash Coarse Screening: The two-level similarity calculation module will... The data is sent to the trajectory feature database for retrieval. Assume there exists a master task flow, Task_Master, in the history, whose description text is "MySQL deadlock inspection in production environment," but whose execution logic is highly consistent with Task_X. Its hash fingerprint... and The same hash or falling into the same hash bucket. The retrieval returns the candidate set. C ={Task_Master}.

[0060] In this embodiment, the total number of historical task flows in the trajectory feature database is assumed to be... N =10000, preset expected candidate set size =20, then That is, the first 9 bits of the SimHash fingerprint are used as the bucket key to build a secondary index.

[0061] S32 Track Edit Distance Calculation: The module performs a fine calculation of the DAG track edit distance for Task_X and Task_Master.

[0062] Sequence Unfolding: Unfold the DAGs of both tasks into a node sequence using normalized topological order. Assume the two sequences are A (from Task_X) and B (from Task_Master).

[0063] Sliding window alignment: Using the sliding window technique, the short sequence window is slid over the long sequence first to find the longest common segment with completely matching features (such as DB_CONNECT and OUTPUT nodes completely matching) and determine the alignment anchor point.

[0064] The window size of the sliding window W A dynamic determination strategy is adopted: Let the length of the short sequence be... ,but That is, the initial window size is 30% of the short sequence length, and the minimum window length is no less than 3 nodes; each time a match fails at the current position, the window size is reduced by 1 node until the longest completely matching common sub-segment is found or the window size is reduced to 1.

[0065] Edit distance calculation: Based on anchor points, a dynamic programming algorithm is applied to locally misaligned sequence segments. Let the length of sequence A be... m =20, the length of sequence B is n =18. Node replacement cost is determined based on 3D feature consistency. Assuming only one node has a replacement cost of 1 due to different parameter default values, while the rest are perfectly matched, the minimum editing cost is accumulated. D =1 (i.e., a replacement operation).

[0066] Similarity conversion: Based on the formula: .

[0067] The specific process of sliding window alignment and dynamic programming calculation described above is as follows: Figure 3 As shown.

[0068] Step S4, Threshold determination and differential node location: 1. Threshold determination: The system presets a threshold. =0.85. (Based on the calculation...) S =0.95, which satisfies the condition. S ≥ The system determines that Task_X and Task_Master are duplicates.

[0069] 2. Difference Location: During the alignment process in step S32, the actuarial module recorded the correspondence of all node pairs. Based on this relationship, the system accurately located the node that was replaced and identified the difference as the additional timeout parameter timeout=30s set in the SQL_QUERY node of Task_X.

[0070] Step S5, Merge suggestion generation and merging: 1. Generating a suggestion: The conflict resolution and merging module generates a merging suggestion and pushes it to the task creator's front-end interface via API. The suggestion content is: "The execution logic of your created task flow 'Check table locking status' is highly similar to that of the existing master task flow 'Production environment MySQL deadlock inspection' (95% similarity), with only one difference found: the SQL query in node B has a 30-second timeout parameter set." 2. User Confirmation and Merging: After reviewing the visually highlighted differences, if the user deems the differences valuable, they click "Confirm Merge." Upon receiving the confirmation command, the system no longer stores Task_X as an independent task flow. Instead, it adds "Set timeout parameter to 30 seconds" as an optional branch to the definition of the master task flow Task_Master.

[0071] Task_Master thus evolved into an enhanced master that includes the original default execution path and the new timeout parameter path.

[0072] Specifically, the data structure of the master task flow adopts an annotated DAG extension model: based on the original DAG structure, each difference node is marked as an "optional branch node," and a set of execution condition metadata is attached to this node. The execution condition metadata includes a branch identifier (BranchID), branch name, trigger condition label (such as "timeout=30s"), and creator information. The merged master is logically represented as the union of the original execution path and all optional branch paths. In physical storage, it is still stored in the form of a single DAG plus a difference patch list to save storage space and maintain the consistency of the retrieval index.

[0073] At the same time, the vector representation of Task_Master in the trajectory feature library is updated to reflect its new variable structure.

[0074] The principle of locating the difference nodes and merging the master pages in this step is as follows: Figure 4 As shown.

[0075] At this point, a complete deduplication and merging loop has been executed. When similar task flows are created subsequently, the system will be able to more efficiently target and reuse this continuously evolving master template.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for deduplicating operation and maintenance task flows based on execution trajectories, characterized in that, Includes the following steps: S1, Execution trajectory analysis and multi-dimensional feature extraction steps: obtain the target task flow definition, analyze its internal execution logic, extract multi-dimensional structured features, the features include at least DAG structure features, operation object features and data flow features, and explicitly exclude the collection and analysis of task flow name and natural language description text; S2, Execution Trajectory Vectorization and Hash Fingerprint Generation Steps: Normalize the multidimensional features extracted in step S1 and encode them into structured execution trajectory vectors. V ,based on V Hash fingerprints are generated by performing hash calculations on a predefined stable feature subset. H ; S3, Two-level similarity calculation steps: using the hash fingerprint of the target task flow... For indexing, a hash-based coarse screening is performed in a pre-built trajectory feature library to obtain a similar candidate set. C ;right C For each historical task flow, its DAG structure is expanded into a node sequence according to the normalized topological order. The sequence is then locally aligned using a sliding window, and the DAG trajectory edit distance is calculated. D And mapped to normalized similarity S ; S4, Threshold determination and difference node localization steps: Compare the similarity. S With preset threshold ,like S < Then the target task flow is stored as a new master task flow in the trajectory feature library; like S ≥ If the target task flow is found to be duplicated with the historical master task flow with the highest similarity, the set of difference nodes in the trajectories of the two will be located. S5, Merge suggestion generation and merging steps: Generate a merge suggestion containing the set of difference nodes and push it to the user terminal. In response to the user's confirmation operation, merge the difference nodes as optional branches into the historical master task flow with the highest similarity to complete the deduplication and merging.

2. The method according to claim 1, characterized in that, The normalization process described in step S2 includes: performing topological order normalization on DAG nodes, and performing unified alias mapping on database objects and field identifiers.

3. The method according to claim 1, characterized in that, The hash calculation is performed using one of the following two methods: (a) In the conventional deduplication process, the SimHash algorithm is used to encode the stable feature subset into a fixed-length binary fingerprint. By utilizing the locality sensitivity of SimHash, the hash fingerprints generated by task flows with similar execution logic have a small Hamming distance. (b) When the system is configured to perform exact deduplication matching, the MD5 algorithm is used to generate a fixed-length hash digest for exact matching of task flows that are logically completely equivalent.

4. The method according to claim 3, characterized in that, When using the SimHash algorithm, the method for encoding the stable feature subset into a SimHash input feature vector is as follows: First, map each node type in the key node type sequence to a preset unique numerical identifier, forming a sequence of length [length missing]. L numerical vectors, L This represents the total number of nodes in the task flow; Secondly, each library name and table name in the set of operation objects is mapped to a preset unique numerical identifier, and concatenated with the node type sequence to form a one-dimensional numerical sequence; the library names and table names are sorted in lexicographical order before concatenation. Then, the one-dimensional numerical sequence is input into the SimHash algorithm to generate a fixed-length binary fingerprint through a preset deterministic random projection matrix or a weighted summation method. The dimension of the deterministic random projection matrix is d × ,in The length of the one-dimensional numerical sequence is... d This determines the number of bits in the output hash fingerprint.

5. The method according to claim 3, characterized in that, In step S3, the hash bucket partitioning rule for the hash coarse screening is as follows: When using the SimHash algorithm, the first binary fingerprint is used... K The position is used as a bucket identifier, before the fingerprint. K All task streams with the same bit value fall into the same hash bucket, where K It is a positive integer, and its value is determined by the following formula: Where N is the total number of historical task flows in the trajectory feature database. The preset expected candidate set size, The value range is from 1 to 100; When using the MD5 algorithm, the complete MD5 value is used as the bucket identifier, and only tasks with exactly the same fingerprint flow into the same hash bucket.

6. The method according to claim 5, characterized in that, The expected candidate set size The value is determined based on the size of the trajectory feature database. N Determined by the following piecewise function: =10, when N ≤ 1000; =20, when 1000 < N ≤ 10000; =50, when N>10000.

7. The method according to claim 1, characterized in that, The DAG trajectory editing distance D The calculation method is as follows: Let the target task flow trajectory sequence be The historical task flow trajectory sequence is ,in ( i = 1, 2, …, m) and ( j = 1, 2, …, n) are nodes arranged in topological order; Define the node replacement cost function ,like and If the DAG structure features, operation object features, and data flow features are completely consistent after normalization, then... ,otherwise The cost of insertion and deletion operations is always 1; Edit distance D Calculated using the dynamic programming recurrence relation: ； in Represents a sequence A The former i Elements and sequences B The former j The minimum edit cost between elements, ultimately ; The normalized similarity S The calculation formula is: 。 8. The method according to claim 1, characterized in that, The preset threshold The value of is determined in the following way: During the platform deployment phase, collect... q Given pairs of equivalent and non-equivalent task flow samples, calculate their similarity respectively. S The threshold that maximizes the Youden index is selected as the initial default value of the system based on the receiver operating characteristic curve. q ≥1000; Alternatively, the threshold System parameters that can be manually adjusted by the system administrator through the management interface.

9. The method according to claim 1, characterized in that, The window size of the sliding window W A dynamic determination strategy is adopted: Let the length of the shorter sequence of the two trajectories be . ,but That is, the initial window size is 30% of the short sequence length, and the minimum window length is no less than 3 nodes; each time a match fails at the current position, the window size is reduced by 1 node until the longest completely matching common sub-segment is found or the window size is reduced to 1.

10. A deduplication system for operation and maintenance task flows based on execution trajectories, used to implement the method described in any one of claims 1 to 9, characterized in that, The system includes: The trajectory feature extraction module is used to execute steps S1 and S2, receive the task flow definition, parse it, and output its execution trajectory vector. V and hash fingerprint H ; The trajectory feature library is used to persistently store the execution trajectory vectors of historical task flows. V and hash fingerprint H And maintain the ownership relationship between each task flow and the parent task flow; A two-level similarity calculation module is used to execute step S3. It contains a coarse screening unit and a fine calculation unit, which respectively realize the candidate set delineation based on hash and the precise similarity calculation based on edit distance. The conflict resolution and merging module is used to execute steps S4 and S5. It is responsible for threshold determination, difference node location, merging suggestion generation, and merging and updating the master task flow based on user feedback.

Images