Link attribution method based on deep learning and knowledge graph

Abstract

The application discloses a link attribution method based on deep learning and a knowledge graph, belongs to the technical field of cloud computing distributed system monitoring and analysis, and aims at solving the problems that in an end-to-end link across a scheduler, a queue, a node agent, an executor, storage and a network, indexes and logs are split, it is difficult to associate scheduling decision events with execution side abnormalities, and it is difficult to automatically correct deviations. The application generates standardized link events through link data semantic alignment and field normalization, constructs a mode layer causal knowledge graph and an instance layer causal knowledge graph, and applies causal direction and allowed edge and forbidden edge constraints to carry out neural causal discovery, combines scheduling decision event intervention variables to carry out counterfactual reasoning to solve a minimum cost intervention set, and realizes the technical effects of unified root cause attribution and output of an executable scheduling correction strategy.

Description

Technical Field

[0001] This invention relates to the field of monitoring and analysis of cloud computing distributed systems, and in particular to a link attribution method based on deep learning and knowledge graphs. Background Technology

[0002] With the development of cloud computing and distributed computing platforms, resource scheduling and execution for batch processing tasks, streaming tasks, and online services have gradually formed end-to-end links. Tasks typically need to go through stages such as scheduling and distribution, queue queuing, resource allocation, node execution, and result collection. The link spans multiple components, including schedulers, queues, node agents, executors, storage, and networks. To ensure service stability, the industry has formed an observable technology system centered on metric monitoring, log analysis, and distributed link tracing. Examples include alarms and capacity analysis based on time-series metrics, fault discovery based on log templates and anomaly detection, and latency and error localization based on call chains. Meanwhile, some solutions introduce knowledge graphs to model component dependencies and event relationships for alarm correlation and root cause localization; others attempt to use machine learning methods for anomaly detection, event correlation, or analysis based on causal inference.

[0003] However, existing technologies still have shortcomings in the aforementioned end-to-end, multi-component scenarios involving scheduling decisions, mainly including:

[0004] 1. The sources of metrics and logs are diverse and inconsistent in their definitions. The lack of a unified semantic alignment, field normalization, and task-level identification association mechanism makes it difficult to accurately aggregate and align cross-component events in the same chain.

[0005] 2. Most root cause analyses rely on correlations or static topologies, making it difficult to establish a unified causal relationship between scheduling decision events such as migration, preemption, retry, and rate limiting and execution-side anomalies (such as node performance degradation, network jitter, and resource leakage). This can easily lead to false alarms, missed alarms, or the location of broken links.

[0006] 3. Existing solutions typically output "possible causes" or impact paths, lacking intervention semantics oriented towards scheduling strategies and executable counterfactual attribution mechanisms. It is difficult to generate cost-controllable parameter adjustment suggestions based on attribution results, thus failing to form a closed loop of automatic correction.

[0007] Therefore, a link attribution method is needed to address the shortcomings of the existing technologies. Summary of the Invention

[0008] One objective of this invention is to propose a link attribution method based on deep learning and knowledge graphs. Addressing the problems of existing technologies where end-to-end links span multiple components such as schedulers, queues, node agents, executors, storage, and networks suffer from fragmented metrics and inconsistent reporting, making it difficult to uniformly associate scheduling decision events with execution-side anomalies for root cause attribution, and failing to automatically correct scheduling strategies based on attribution results, this invention proposes a method to collect link data and add task identifiers and timestamps. It utilizes a deep learning encoding model to semantically align and normalize the metric and log data to generate a standardized link event set. It constructs a pattern-level causal knowledge graph and an instance-level causal knowledge graph, generating a candidate causal edge set based on causal direction constraints and allowed and prohibited edges. It combines scheduling decision events to generate intervention variables and performs neural causal discovery to learn causal structures and causal effect parameters. Furthermore, it executes counterfactual reasoning to solve for the minimum cost intervention set that satisfies recovery conditions and outputs causal paths and scheduling correction strategies. This invention achieves unified attribution across components, reduces false positives and false negatives, improves fault location efficiency, and outputs executable scheduling correction suggestions.

[0009] This invention provides a link attribution method based on deep learning and knowledge graphs, characterized by comprising:

[0010] S1. Collect end-to-end link data, add task identifiers and timestamps, and call a deep learning encoding model to perform semantic alignment and field normalization on the indicator data and log data in the end-to-end link data to generate a standardized link event set; S2. Construct a pattern-layer causal knowledge graph based on the component types, event types, and inter-component dependencies involved in the end-to-end link, including entity type sets, relation type sets, causal direction constraints, allowed edge type sets, and prohibited edge type sets; S3. Instantiate and map the standardized link event set according to the pattern-layer causal knowledge graph to construct an instance-layer causal knowledge graph with timestamps; S4. Based on the causal direction constraints, allowed edge type sets, prohibited edge type sets, and the temporal sequence of instance nodes, in the actual In instance-layer causal knowledge graph, a candidate causal edge set is generated, and learnable edge parameters are set; S5, scheduling decision events are determined from the standardized link event set, and intervention variables are generated. These are associated with the instance-layer causal knowledge graph, and the neural causal discovery model is called to learn the causal structure model under the constraints of the candidate causal edge set, including the causal structure and causal effect parameters; S6, recovery conditions and cost functions are determined for the target execution side anomalies. Counterfactual reasoning is performed based on the causal structure model and with intervention variables as interventionable quantities. Under the constraints of satisfying the recovery conditions, the minimum cost intervention set that minimizes the cost function is solved, and the corresponding causal path is output as the link attribution result. A scheduling correction strategy for adjusting the scheduling decision event parameters is generated based on the minimum cost intervention set.

[0011] Optionally, S1 includes:

[0012] The system collects end-to-end link data from task scheduling, queuing, allocation, execution to recycling. The end-to-end link data includes scheduler event data, queue event data, node agent event data, executor event data, storage event data, network event data, and corresponding indicator data and log data.

[0013] Add a task identifier and timestamp to each data record in the end-to-end link data, and construct the original link dataset based on the data records after adding the identifier;

[0014] Based on the original link dataset, a deep learning encoding model is invoked, and the deep learning encoding model is used to perform semantic alignment and field normalization on the indicator data and the log data to obtain the normalization result;

[0015] The deep learning encoding model includes a log encoding sub-model and an indicator encoding sub-model. The log encoding sub-model is used to encode log template sequences or log field sequences to obtain log vector representations. The indicator encoding sub-model is used to aggregate indicator data according to a preset time window and encode it to obtain indicator vector representations.

[0016] The log vector representation and the indicator vector representation are merged to generate a unified event representation that includes entity identifier, event type, event parameters, timestamp, and vector representation, and then aggregated to form a standardized link event set.

[0017] Optionally, S2 includes:

[0018] The entity type set is determined based on the component types involved in the end-to-end link. The entity type set includes the entity types corresponding to schedulers, queues, node agents, executors, storage, and networks. The event type corresponding to the entity type set is determined based on the event types involved in the end-to-end link.

[0019] A set of relationship types is determined based on the dependencies between components. The set of relationship types is used to represent the dependencies and data flow relationships between the sets of entity types.

[0020] For each relation type in the set of relation types, a causal direction constraint is configured based on the inter-component dependency relationship. The causal direction constraint is used to limit the allowed edge directions when generating subsequent candidate causal edges.

[0021] Based on the inter-component dependencies, a set of allowed edge types and a set of prohibited edge types are determined. The set of prohibited edge types includes edge types that violate the inter-component dependencies or the data flow relationships.

[0022] Construct a pattern-layer causal knowledge graph based on the entity type set, the relation type set, the causal direction constraint, the allowed edge type set, and the prohibited edge type set.

[0023] Optionally, S3 includes:

[0024] Based on the entity type set in the pattern layer causal knowledge graph, the standardized link event set is assigned to events to determine the entity type corresponding to each standardized link event.

[0025] For each of the standardized link events, an instance node corresponding to the standardized link event is generated based on the task identifier, and a timestamp associated with the standardized link event is retained for the instance node;

[0026] Based on the set of relation types in the pattern layer causal knowledge graph, instance edges are generated between instance nodes with dependencies, and timestamp information associated with each instance edge is retained. The timestamp information of the instance edge includes the call time or data flow time corresponding to the instance edge.

[0027] The instance nodes and instance edges are aggregated to construct an instance-layer causal knowledge graph.

[0028] Optionally, S4 includes:

[0029] Based on the causal direction constraints, allowed edge type set, and prohibited edge type set in the pattern layer causal knowledge graph, the starting entity type and target entity type that are allowed to establish causal relationships in the instance layer causal knowledge graph are determined.

[0030] In the instance-layer causal knowledge graph, for instance node pairs that satisfy the initial entity type and the target entity type and have a dependency relationship, directed candidate causal edges are established according to the causal direction constraints and based on the set of allowed edge types, while candidate causal edges belonging to the set of prohibited edge types are excluded.

[0031] Candidate causal edges are established only when the timestamp of the starting instance node is earlier than the timestamp of the target instance node, and the time difference between the two is within a preset time window.

[0032] The candidate causal edges are aggregated to form a candidate causal edge set;

[0033] For each candidate causal edge in the candidate causal edge set, a corresponding edge weight parameter is set so that the candidate causal edge set carries the edge weight parameter;

[0034] The preset time window is determined by the upper limit of link observation delay and the upper limit of inter-component call delay.

[0035] Optionally, S5 includes:

[0036] Identify scheduling decision events representing migration, preemption, retry, and rate limiting from the standardized link event set, and extract the event type, event occurrence timestamp, and event parameters from each scheduling decision event, and generate intervention variables based on the event type, the timestamp, and the event parameters;

[0037] The intervention variable is associated with the corresponding instance node in the instance layer causal knowledge graph based on the task identifier, and the intervention variable is associated with the corresponding candidate causal edge in the candidate causal edge set based on the timestamp.

[0038] After the intervention variable is associated with the instance layer causal knowledge graph and the candidate causal edge set, the neural causal discovery model is invoked for processing. The neural causal discovery model learns the edge weight parameters of the candidate causal edge set based on graph neural networks or structural equation networks, and constrains the learned causal structure through sparse regularization constraints and acyclic constraints, and outputs a causal structure model, including the causal structure determined by the edge weight parameters and the corresponding causal effect parameters.

[0039] Furthermore, the objective function of the neural causal discovery model further includes: a prior consistency regularization term and an intervention consistency loss term. The prior consistency regularization term is used to increase the edge weight parameters of candidate causal edges that are consistent with the causal knowledge graph of the pattern layer and suppress the edge weight parameters of other candidate causal edges. The intervention consistency loss term is used to increase the consistency between the abnormal prediction changes and the observed changes caused by the change of the intervention variable on the target execution side.

[0040] Optionally, S6 includes:

[0041] Based on the target execution-side anomaly, preset recovery conditions are determined. The preset recovery conditions include at least one observation indicator corresponding to the target execution-side anomaly satisfying a preset threshold or preset range. The observation indicator includes at least one of execution latency, error rate, throughput, queue waiting latency, node resource utilization, and network packet loss rate.

[0042] The cost function is determined based on the impact of parameter adjustments on scheduling decision events. The cost function is used to characterize the number of parameter adjustments, the magnitude of parameter adjustments, or the impact of parameter adjustments on task performance.

[0043] In the causal structure model, the intervention variables are used as modifiable quantities. Counterfactual reasoning is performed on each candidate intervention set composed of the intervention variables to obtain counterfactual results of the target execution side anomaly corresponding to each candidate intervention set.

[0044] The counterfactual reasoning includes an abductive step, an action step, and a prediction step. The abductive step includes estimating exogenous noise in the causal structure model based on the observed standardized link event set. The action step includes cutting the incoming edges of the instance nodes associated with the intervention variable and replacing the structural equation corresponding to the instance node with an assignment function determined by the event parameters of the scheduling decision event. The prediction step includes calculating the counterfactual result of the target execution-side anomaly on the causal structure model after replacing the structural equation.

[0045] By performing a combinatorial search or enumeration on the candidate intervention set, a candidate intervention set that minimizes the cost function is selected from the candidate intervention set that satisfies the preset recovery conditions, and this set is determined as the minimum cost intervention set.

[0046] Based on the causal structure of the causal structure model, extract the causal path from the minimum cost intervention set to the target execution side anomaly, and output the minimum cost intervention set and the causal path together as the link attribution result.

[0047] Based on the link attribution results, the scheduling decision events and their event parameters corresponding to the minimum cost intervention set are converted into parameter adjustment suggestions to generate a scheduling correction strategy;

[0048] Furthermore, the solution to the minimum cost intervention set also satisfies feasibility constraints, which include: the scheduling decision event parameters corresponding to the intervention variables satisfy the parameter value constraint set, and the solution is obtained by mixed integer programming or gradient optimization with projection operators, so that the minimum cost intervention set obtained is an executable parameter adjustment set.

[0049] Optionally, obtaining the deep learning encoding model and the neural causal discovery model includes: constructing a training dataset based on historical end-to-end link data;

[0050] Using the training dataset as input, the deep learning coding model is obtained by optimizing the objective function to increase the similarity of the unified event representations of the indicator data and log data corresponding to the same task identifier and to decrease the similarity of the unified event representations corresponding to different task identifiers.

[0051] Based on the candidate causal edge set defined by the pattern layer causal knowledge graph, the neural causal discovery model is obtained by optimizing the objective function, which includes data fitting terms and sparse constraint terms.

[0052] The beneficial effects of this invention are:

[0053] 1. Achieve unified expression and alignment of cross-component end-to-end link data: By semantically aligning and normalizing the fields of indicator data and log data, and introducing task identifiers and timestamps to form a standardized link event set, heterogeneous data from multiple sources such as schedulers, queues, node agents, executors, storage and networks can be aggregated under the same caliber, improving the consistency and availability of link analysis.

[0054] 2. Improve the accuracy of root cause attribution and reduce false positives and false negatives: By constructing a pattern-level causal knowledge graph and an instance-level causal knowledge graph, and using causal direction constraints, allowed edge type sets and prohibited edge type sets to generate candidate causal edges, neural causal discovery can be performed under structural prior constraints. This enables unified causal association between scheduling decision events and execution-side anomalies, reducing erroneous attributions caused by relying solely on correlation.

[0055] 3. Form an executable error correction closed loop for scheduling strategies: Model scheduling decision events as intervention variables and perform counterfactual reasoning. Under the constraints of meeting the anomaly recovery conditions, solve for the minimum cost intervention set, output causal paths and parameter adjustment suggestions, so that the attribution results can be directly transformed into scheduling error correction strategies, thereby improving fault recovery efficiency and automation. Attached Figure Description

[0056] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0057] Figure 1 This is a flowchart of the link attribution method based on deep learning and knowledge graph proposed in this invention;

[0058] Figure 2 This is a schematic diagram of the counterfactual reasoning and scheduling correction process in step S6 of the present invention;

[0059] Figure 3 This is a schematic diagram of the neural causality discovery process in step S5 of the present invention. Detailed Implementation

[0060] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0061] refer to Figure 1 Link attribution methods based on deep learning and knowledge graphs include:

[0062] S1. Collect end-to-end link data, add task identifiers and timestamps, and call a deep learning encoding model to perform semantic alignment and field normalization on the indicator data and log data in the end-to-end link data to generate a standardized link event set; S2. Construct a pattern-layer causal knowledge graph based on the component types, event types, and inter-component dependencies involved in the end-to-end link, including entity type sets, relation type sets, causal direction constraints, allowed edge type sets, and prohibited edge type sets; S3. Instantiate and map the standardized link event set according to the pattern-layer causal knowledge graph to construct an instance-layer causal knowledge graph with timestamps; S4. Based on the causal direction constraints, allowed edge type sets, prohibited edge type sets, and the temporal sequence of instance nodes, in the actual In instance-layer causal knowledge graph, a candidate causal edge set is generated, and learnable edge parameters are set; S5, scheduling decision events are determined from the standardized link event set, and intervention variables are generated. These are associated with the instance-layer causal knowledge graph, and the neural causal discovery model is called to learn the causal structure model under the constraints of the candidate causal edge set, including the causal structure and causal effect parameters; S6, recovery conditions and cost functions are determined for the target execution side anomalies. Counterfactual reasoning is performed based on the causal structure model and with intervention variables as interventionable quantities. Under the constraints of satisfying the recovery conditions, the minimum cost intervention set that minimizes the cost function is solved, and the corresponding causal path is output as the link attribution result. A scheduling correction strategy for adjusting the scheduling decision event parameters is generated based on the minimum cost intervention set.

[0063] In this specific embodiment, S1 includes:

[0064] Deploy link data acquisition interfaces on schedulers, queues, node agents, executors, storage and network components respectively, and output end-to-end link data records in a unified format. Each data record includes component type, component instance identifier, event type original identifier, event parameter original field, indicator name and indicator sample value or log text, acquisition side host identifier and acquisition time information.

[0065] Write the task identifier corresponding to the same task and the timestamp to each data record. The task identifier is in the form of a string, task_id, and remains unchanged throughout the entire process from task scheduling to recycling. The timestamp is in the millisecond Unix time and is uniformly converted to the UTC time zone and aligned with NTP to ensure that the clock deviation across components does not exceed 50ms.

[0066] The original link dataset is constructed based on the data records after writing the task identifier and timestamp, and then grouped by task_id and sorted by timestamp from smallest to largest to form a task-level ordered sequence;

[0067] The log data is templated and split into fields using the Drain log template extraction algorithm with a fixed parameter of parsing depth. Maximum number of child nodes at each level of the prefix tree Template matching similarity threshold Each log entry is converted into a log template identifier and a sequence of variable fields, while retaining the log-level timestamp.

[0068] The indicator data is aggregated and normalized according to a preset time window, and the preset time window is taken as follows: Furthermore, the index sampling points within the time window are extracted using the event timestamp as the right endpoint, and the mean of the sampling points is calculated using the formula:

[0069] ;

[0070] in Indicates the first Events in the time window Inner The average of the indicators, Indicates the first time within this time window The number of sampling points for each indicator Indicates the first time within this time window The first indicator Each sample value, This represents the event sequence number sorted by task_id and timestamp. This represents the index of an indicator within a predefined set of indicator names. Indicates the sampling point number;

[0071] The mean and maximum value of each indicator within the time window are compared with... The quantile array is used to synthesize indicator features and concatenate them into an indicator input vector in a fixed order according to the indicator names. Unit normalization and scale normalization are then performed on the indicator input vector. Unit normalization converts latency to milliseconds (ms), throughput to ops / s, and utilization to [missing value]. For interval values, the scale normalization is performed by standardizing the mean and standard deviation obtained from historical training data, and the mean and standard deviation remain fixed during the inference phase.

[0072] A deep learning encoding model is invoked to perform semantic alignment and field normalization on logs and metrics. This deep learning encoding model consists of a log encoding sub-model and a metric encoding sub-model, with a fixed output dimension of 128. The log encoding sub-model takes log template identifiers and variable field sequences as input and uses a vocabulary size. Token embedding, embedding dimension 128, maximum sequence length Sequence truncation and padding strategies, number of layers And the number of attention heads Furthermore, the feedforward layer hides a Transformer encoder with a dimension of 256 and takes the first aggregated vector as the log vector representation. The index encoding sub-model takes the index input vector as input and adopts a two-layer fully connected network with layer widths of 256 and 128 respectively. ReLU and LayerNorm are cascaded after each layer to output the index vector representation.

[0073] The log vector representation and the indicator vector representation are fused. The fusion method is to concatenate the vectors and pass them through a fully connected layer with an input dimension of 256 and an output dimension of 128, and then normalize them with L2 to obtain a unified event representation vector representation.

[0074] At the field normalization level, the component instance identifier is uniformly mapped to the entity identifier entity_id according to the predefined standard field dictionary, the original event type identifier is uniformly mapped to the event type event_type, and the original event parameter field is uniformly mapped to the event parameter event_param. Furthermore, the unit and scale normalization consistent with the index are performed on the unit and value range of the same parameter on different component sides, and the timestamp field is uniformly named timestamp.

[0075] Log data and metric data with the same entity ID under the same task ID and timestamp difference not exceeding 5 seconds are aligned and merged to generate standardized link events containing entity ID, event type, event param, timestamp, and a unified event representation vector, and are aggregated to form a standardized link event set.

[0076] In this specific embodiment, S2 includes:

[0077] Based on the standardized link event set field system, a pattern-layer causal knowledge graph is constructed, and component types, event types, and component dependency constraints are solidified in a configuration table-driven manner. The pattern-layer causal knowledge graph is defined as follows:

[0078] ;

[0079] in Represents a pattern-layer causal knowledge graph. This represents a set of entity types, taking six entity types: scheduler, queue, node agent, executor, storage, and network. This represents a set of relation types and is used to characterize the dependencies and data flow relationships between entity types. This represents the set of causal direction constraints and is used to limit the allowed causal directions for each relation type. This represents the set of allowed edge types and is used to limit subsequent establishment of candidate causal edges only on data paths that satisfy dependencies. This represents the set of prohibited edge types and is used to explicitly exclude edge types that violate dependencies or data flow relationships.

[0080] To ensure a deterministic mapping between entity types and event types, an event type dictionary `event_type_dict` is established during system initialization. The `event_type` of standardized link events is mapped to a unique entity type. Specifically, the event types corresponding to the scheduler entity type include distribution, migration, preemption, retry, rate limiting, and recycling; the event types corresponding to the queue entity type include enqueue, dequeue, waiting, and quota; the event types corresponding to the node agent entity type include resource reporting, resource allocation, resource release, and health checks; the event types corresponding to the executor entity type include startup, execution, completion, failure, and timeout; the event types corresponding to the storage entity type include read, write, metadata query, and storage timeout; and the event types corresponding to the network entity type include connection establishment, request sending, response receiving, retransmission, and packet loss.

[0081] Establish a set of relationship types based on component dependencies in the end-to-end link. And register triple records of relation type, starting entity type, and target entity type in relation type dictionary relation_type_dict. The relation type set includes scheduling and dispatching dependencies from scheduler to queue, allocation request dependencies from queue to node agent, startup control dependencies from node agent to executor, read and write data flow relationships from executor to storage, communication data flow relationships from executor to network, and remote access data flow relationships from storage to network.

[0082] For each type of relation, the set of constraints in the causal direction. Configure a unique directed constraint and write the fields cause_type and effect_type, where cause_type takes the starting entity type of the relation type and effect_type takes the target entity type of the relation type, thereby fixing the causal direction as "dependency initiator points to dependency bearer" and using it for subsequent candidate causal edge direction filtering.

[0083] In the set of allowed edge types The system registers all edge types that satisfy component dependencies and data flow direction. Each edge type is uniquely identified by an ordered pair (cause_type, effect_type) and bound to its corresponding relation_type, ensuring that subsequent operations only consider edge types that satisfy component dependencies and data flow direction. Candidate causal edges are generated between the covered entity type pairs;

[0084] In the set of forbidden edge types All registered in China The reverse edge type in the opposite direction and all non-dependent edge types that cross dependent links, including entity type pairs such as scheduler pointing to executor, queue pointing to storage, node agent pointing to storage, network pointing to scheduler, etc., that do not satisfy the component dependency relationship or the data flow relationship, and will... Persist in the form of a blacklist table (forbidden_edge_table) for quick subsequent verification;

[0085] Entity type collection Type nodes are written to the schema layer graph storage, and each type node is assigned a fixed type identifier (entity_type_id) and a type name (entity_type_name), thus creating a collection of relational types. Each type edge is written to the schema layer graph for storage, and a fixed relation identifier (relation_type_id), relation name (relation_type_name), and corresponding cause_type and effect_type are set for each type edge. An allow flag is also synchronously written for use with... Consistency verification is performed to complete the construction of a schema-layer causal knowledge graph that includes a set of entity types, a set of relation types, causal direction constraints, a set of allowed edge types, and a set of prohibited edge types.

[0086] In this specific embodiment, S3 includes:

[0087] The standardized link event set is used as instantiated input and the pattern layer causal knowledge graph is invoked. The mapping of events to instance nodes and instance edges is completed to generate a timestamped instance-layer causal knowledge graph, which is defined as follows:

[0088] ;

[0089] in Represents an instance-level causal knowledge graph. Represents a collection of instance nodes. Represents the set of instance edges. This represents the timestamp mapping function for instance nodes and is used to assign the timestamp of the corresponding normalized link event to each instance node. This represents the instance edge timestamp mapping function, used to assign each instance edge its corresponding call occurrence time or data flow occurrence time. This represents the instance node attribute mapping function, used to write task_id, entity_id, entity type, event_type, event_param, and vector representation. This represents the instance edge attribute mapping function and is used to write the relation type and edge timestamp source identifier;

[0090] For each standardized link event in the standardized link event set, event attribution is performed to determine the entity type. Event attribution is achieved by querying the event_type_dict in the causal knowledge graph of the pattern layer and mapping the event_type of the standardized link event to a unique entity type, thereby ensuring that the same event_type corresponds to only one entity type in the entire system and that the entity type belongs to the entity type set of S2.

[0091] Instance nodes are generated based on each standardized link event, ensuring that each instance node is unique within a single task. The node identifier `node_id` of an instance node is obtained by concatenating `task_id`, `entity_id`, `event_type`, and `timestamp` in a fixed order and then calculating a 64-bit hash. The hash input uses the separator `|` to avoid ambiguity in the concatenation. Instance node attributes are... Write and include at least the following: task_id for task-level isolation, entity_id for component instance location, entity type for alignment with the schema layer, event_type and event_param for event semantic recovery, timestamp for time sorting, and vector representation for numerical input for subsequent causal discovery.

[0092] To ensure that the generation of instance edges is repeatable and traceable, the cross-component association key is permanently stored in event_param in the field normalization result and a unified field name upstream_entity_id, downstream_entity_id, and flow_id are used. Upstream_entity_id represents the entity identifier that initiates the dependency or data flow, downstream_entity_id represents the entity identifier that carries the dependency or receives the data flow, and flow_id represents the shared association identifier of the same call or the same data flow in different component events.

[0093] Based on the set of relation types in the pattern layer causal knowledge graph, instance edges are established between instance nodes. Specifically, within the same task_id group, standardized link events are aggregated by flow_id, and within each flow_id, instance node pairs that satisfy the upstream_entity_id and downstream_entity_id pointing relationship are retrieved. Then, the relation_type_dict of the pattern layer causal knowledge graph is queried to match the starting entity type and target entity type of the instance node pair and generate instance edges of corresponding relation types. The starting point of the instance edge is the instance node corresponding to upstream_entity_id, and the ending point is the instance node corresponding to downstream_entity_id. The relation type identifier is written in the instance edge attribute to maintain consistency with the pattern layer.

[0094] For each instance, retain the edge timestamp information to support subsequent candidate causal edge filtering based on temporal sequence. The edge timestamp is obtained through... The writing and its value retrieval rules are as follows: when event_param contains the field call_timestamp, call_timestamp is taken as the call occurrence time; when event_param contains the field flow_timestamp, flow_timestamp is taken as the data flow occurrence time; when none of the above fields exist, the timestamp of the endpoint instance node of the edge is taken as the edge timestamp and the edge timestamp source identifier is written in the edge attribute to indicate that it comes from the call time, data flow time or endpoint event time;

[0095] The generated instance nodes and instance edges are written to the graph storage, and indexes are created for task_id and node_id to support fast retrieval of subgraphs by task. Instance edges with the same start node_id, end node_id, and relation type identifier under the same task_id are deduplicated, thereby completing the construction of a timestamped instance-layer causal knowledge graph.

[0096] In this specific embodiment, S4 includes:

[0097] Using pattern-level causal knowledge graphs Instance-level causal knowledge graph The set of candidate causal edges is generated as input, and learnable edge parameters are set for each candidate causal edge.

[0098] from Read the set of allowed edge types Forbidden Edge Type Set With causal direction constraint set and will Each edge type in the algorithm is fixed as an ordered pair (cause_type, effect_type) and the ordered pair must simultaneously satisfy the following conditions: The consistency of the registered directions yields the combination of the starting entity type and the target entity type that are allowed to establish causal relationships in the instance layer;

[0099] Subsequently Traverse the instance edge collection by task_id For each instance edge, read its starting instance node and ending instance node, and obtain the starting entity type and ending entity type respectively. Then, backtrack based on the relationship type identifier carried by the instance edge. set of relation types in This confirms that the component dependencies and data flow relationships corresponding to the instance edge have been registered at the schema layer, thereby limiting "instance node pairs with dependencies" to those defined by... Directly connected instance node pairs;

[0100] If and only if the entity type of the instance node pair is an ordered pair belonging to And does not belong to and satisfy Only when the edge direction is specified, a directed candidate causal edge is created in the direction from the starting instance node to the ending instance node and added to the candidate causal edge set. Furthermore, the edge type of the candidate causal edge is taken from the relation type identifier of the instance edge to maintain consistency with the pattern layer;

[0101] To suppress spurious causal connections that do not conform to temporal order, the timestamps of the starting and ending instance nodes of each candidate causal edge are read and a time window is applied for filtering. This time window is denoted as... And note:

[0102] ;

[0103] in This represents the maximum allowed time difference window for candidate causal edges, expressed in milliseconds. This indicates the upper limit of the link observation delay, expressed in milliseconds (ms), and is buffered by the acquisition end and fixed at 2000ms for the reporting link. This represents the upper limit of inter-component call latency in milliseconds (ms). The maximum timeout threshold for control plane calls from the scheduler to the executor and data plane calls from the executor to storage and network is uniformly configured to 5000ms, thus obtaining... And only if the timestamp of the starting instance node is earlier than the timestamp of the ending instance node and the time difference between the two does not exceed The candidate causal edge is retained at that time;

[0104] Regarding edge parameter settings, Each candidate causal edge is assigned a unique candidate edge identifier, cand_edge_id. The cand_edge_id is obtained by concatenating the starting instance node_id, the ending instance node_id, and the relation type identifier in a fixed order and then calculating a 64-bit hash. A learnable edge weight parameter is created for each cand_edge_id. ,in Indicate candidate causal edges The edge weight parameters are used for updating by the neural causal discovery model in S5. The parameter data type is float32 and initialized to 0. They are stored in a parameter table with cand_edge_id as the key to ensure consistent addressing across batch training and inference. At the same time, weight_param_key is written to the candidate causal edge record to point to the corresponding edge weight. Then, edge_mask=1 is written as a learnable marker, forming a set of candidate causal edges that carry learnable edge parameters and satisfy direction constraints, allowed and prohibited edge constraints, and time window constraints. .

[0105] In this specific embodiment, S5 includes:

[0106] With standardized link event sets and instance-level causal knowledge graphs and candidate causal edge set Intervention variables are constructed for the input and a neural causal discovery model is trained to output a causal structure model;

[0107] In the standardized link event set, retrieve standardized link events whose entity type is scheduler and whose event_type belongs to migration, preemption, retry, or rate limiting. Extract event parameter key-value pairs for scheduling control from the event_param of each standardized link event, and keep the parameter name and unit unchanged after the field normalization process in S1. Thus, generate an intervention variable record intervention_record for each scheduling decision event. The intervention_record contains intervention_id, task_id, timestamp, intervention_type, and intervention_param. The intervention_id is obtained by concatenating task_id, the entity_id of the scheduler instance, and timestamp and calculating a 64-bit hash. The intervention_type is the event_type. The intervention_param is the event parameter key-value pair. Perform unit normalization and scale normalization on the numerical parameters in the same way as in step S1.

[0108] Then, based on the task_id and timestamp, each intervention_record is... The association between instance nodes is established by locating instance nodes of type scheduler within the same task_id subgraph whose timestamp matches the timestamp of intervention_record, and marking these instance nodes as intervention nodes. Simultaneously, each intervention_record is associated with a set of candidate causal edges based on its timestamp. Establish a correlation by selecting candidate causal edges whose starting point is the intervention node and whose endpoint timestamp is later than the intervention node timestamp and the time difference is no more than 1000ms. Write intervention_id into the candidate causal edge attribute to indicate that the candidate causal edge is affected by the intervention variable.

[0109] After completing the association of intervention variables, a neural causal discovery model is trained. This model is implemented using a structural equation model and employs task subgraphs as training samples. For each task_id, [the model is trained from...]. The task's instance node set and candidate causal edge set are taken, and node observation vectors and edge weight parameters are constructed as inputs. The node observation vectors are obtained by concatenating the unified event representation vector with key observation indicators extracted from event_param, with the concatenation order fixed and missing indicators filled with 0. The edge weight parameters are taken from the edge weight parameters of each candidate causal edge set in S4. And it was trained using float32;

[0110] The computational structure of a structural equation network is fixed at two message passing layers and two prediction head layers. The number of message passing layers is 2, and the hidden dimension of each layer is 128. In each layer, the hidden vectors of all incoming edge originating nodes of each instance node are aggregated and used as candidate causal edges. As the aggregation coefficient of the incoming edge, where Use the Sigmoid function to limit the edge strength. In the interval, the prediction head uses a two-layer fully connected network with a layer width of 256 and the observation vector dimension, respectively, and ReLU and LayerNorm are used between the layers to output the predicted value of the observation vector for each instance node and use the prediction residual for data fitting.

[0111] The objective function for model training is fixed as follows:

[0112] ;

[0113] in Describe the overall objective function. This represents the mean squared error of the data fitting term and the prediction residuals obtained by taking the observation vectors of all instance nodes. Denotes the sparse regularization coefficient and takes The summation symbol represents the summation of the elements of a set. Indicates the candidate causal edge index. Denotes the set of candidate causal edges. This represents the absolute value operation. Indicate candidate causal edges The edge strength and the edge weight parameters Obtained via the Sigmoid function, Denotes the coefficient of acyclic constraint and takes Indicates based on all The acyclic penalty term is calculated from the assembled task-level adjacency matrix, and acyclicity is achieved through matrix exponent trace constraints. This represents a vector consisting of the weight parameters of all candidate causal edges. Denotes the prior consistency regularity coefficient and takes This represents a prior consistency regularization term and a weighted average by setting different penalty weights for different relation types. This approach aims to increase the edge strength of candidate causal edges that are more consistent with the dependency chains of the pattern-layer causal knowledge graph, while suppressing the edge strength of other candidate causal edges. Represents the consistency coefficient of intervention and takes This represents the intervention consistency loss term, and the mean square error is calculated between the observation vector difference within a fixed time window before and after each intervention_record and the prediction difference obtained by the model after encoding the corresponding intervention variable as an exogenous input. This increases the consistency between the predicted changes and observed changes of the target execution side caused by the change of intervention variable.

[0114] The neural causal discovery model is trained using the Adam optimizer with a learning rate of [value missing]. Iterate and update the data according to the batch size of task_id (32) until the overall objective function converges;

[0115] After training The candidate causal edge set is pruned into a causal structure using a retention threshold, and the retained edges are... As a causal strength parameter, the prediction head parameters of the structural equation network are used as causal effect parameters, and the output is a causal structure model that includes causal structure and causal effect parameters.

[0116] In this specific embodiment, S6 includes:

[0117] The causal structure model is used as the structural and parameter basis for counterfactual reasoning, and the intervention variables generated by S5 and associated with instance nodes are used as interventionable quantities to complete link attribution and scheduling correction.

[0118] First, identify the target execution-side anomaly and solidify its recovery conditions and observation indicators. The target execution-side anomaly corresponds to an anomaly node in the causal structure model, and the observation indicators of the anomaly node are execution latency (exec_latency) and error rate (error_rate). The preset recovery condition is defined as the counterfactual result satisfying that exec_latency is not greater than 500ms and error_rate is not greater than 0.01. This recovery condition is written into the recovery condition table as a subsequent screening constraint.

[0119] Subsequently, a candidate intervention set was determined and the search space was limited. The candidate intervention set consisted of intervention variables that appeared within a 300-second time backtracking window before the anomaly occurrence time stamp, and only intervention variables that could reach the anomaly node along the directed edge in the causal structure model were retained. The adjustable parameters of each intervention variable in the set were extracted as parameter vectors and concatenated in a fixed order to form an intervention parameter vector. The fixed order is retry_limit, rate_limit, preempt_quota, migrate_parallelism, where retry_limit and migrate_parallelism are integer parameters, and rate_limit and preempt_quota are real number parameters.

[0120] Further determine the set of parameter value constraints and write them as feasibility constraints. The value range of retry_limit is: The range of values ​​for rate_limit is: The range of values ​​for `preempt_quota` is: And the unit is ms, the range of values ​​for migration_parallelism is... Furthermore, all parameters are based on the unit normalization of S1, thereby ensuring that the intervention parameter vector obtained by solving is an executable set of parameter adjustments;

[0121] In the counterfactual reasoning phase, abductive, action, and prediction steps are performed for each candidate intervention set. In the abductive step, the standardized link event set prior to the anomaly occurrence is used as the observation input, and the predicted values ​​for each instance node are calculated forward in the causal structure model. Subsequently, the difference between the observed and predicted values ​​is used as an exogenous noise estimate and embedded in the exogenous term of the corresponding structural equation to ensure consistency between counterfactual reasoning and the observed links. In the action step, incoming edges are cut off for the intervention nodes corresponding to the candidate intervention sets, and the structural equation of the intervention node is replaced with one derived from the intervention parameter vector. The assignment function that directly assigns values ​​thus achieves this. In the semantic prediction step, the counterfactual indices of anomalous nodes are calculated and counterfactual results are obtained by forward propagation layer by layer in topological order on the causal structural model after the structural equation is replaced. ;

[0122] When solving for the minimum cost intervention set under the constraint of satisfying the recovery condition, the cost function is defined as the sum of the weighted absolute changes of all adjusted parameters relative to the current online parameters, and the weights are fixed as retry_limit weight 1, rate_limit weight 5, preempt_quota weight 2, and migration_parallelism weight 1. The "combinatorial search of candidate intervention sets" is fixed as an enumeration strategy that increases from 1 to 3 in order of the number of intervention variables, and gradient optimization with a projection operator is applied to each enumerated candidate intervention set within the feasible region. The minimum cost parameter value that satisfies the recovery condition is searched within the inner search. The projection operator performs interval truncation on real-valued parameters and first performs interval truncation on integer-valued parameters before taking the nearest integer to ensure executability.

[0123] The constrained optimization problem is defined in the following form:

[0124] s. t. ;

[0125] in This represents the vector of intervention parameters that satisfy the recovery conditions and minimize the cost. This represents the vector of intervention parameters to be solved. This represents the feasible region determined by the set of parameter value constraints. Indicates the dimension index of the intervention parameter. This represents the total number of dimensions of the intervention parameter vector, which is obtained by concatenating the adjustable parameter dimensions of each intervention variable in the candidate intervention set. Indicates the first The cost weights of the dimension parameters are assigned according to the aforementioned weighting rules. express The Dimensional parameter values, This indicates the currently effective number of online [number]. The values ​​of the dimension parameters are obtained and read from the intervention_param of the corresponding intervention variable. This represents the absolute value operation. Indicates in The following is a counterfactual observation index vector of anomalous nodes predicted by the causal structure model. This represents the set of acceptable counterfactual outcomes determined by the recovery conditions and corresponds to the threshold constraints of exec_latency and error_rate.

[0126] After enumeration, from all satisfying... The solution selects the candidate intervention set with the lowest cost and determines it as the minimum cost intervention set. Simultaneously, in the directed causal graph of the causal structure model, starting from the intervention node corresponding to the minimum cost intervention set, a directed reachable path search is performed on the abnormal nodes. The path selection rule is to extract a unique causal path based on the maximum product value of the edge strength parameters and the minimum number of edges. The minimum cost intervention set and the causal path are output together as the link attribution result. The solution for each intervention variable in the minimum cost intervention set is then defined. The parameter name is denormalized to an intervention_param update value that can be written by the scheduling system, and a scheduling correction strategy is generated that includes the parameter name, target value, effective scope (corresponding task_id), and effective timestamp.

[0127] In this specific embodiment, the deep learning encoding model and the neural causal discovery model are obtained through offline training using historical end-to-end link data;

[0128] The training dataset is constructed by extracting end-to-end link data for 30 consecutive days from the historical monitoring system and log system and writing it into task_id and timestamp according to the S1 rule. The training dataset forms task-level samples with task_id as the primary key and sorts them by timestamp within each task sample to obtain a standardized link event sequence. At the same time, a pair of aligned samples is constructed as positive sample pairs with log data and indicator data with the same entity_id and timestamp difference of no more than 5s within the same task_id. Negative sample pairs are constructed with log data and indicator data between different task_ids.

[0129] The deep learning encoding model has a fixed output dimension of 128, and the log encoding sub-model uses a vocabulary size of [missing information]. The token is embedded with an embedding dimension of 128, and the maximum sequence length is... Transformer encoder layer number Attention count The feedforward layer has a hidden dimension of 256, and the first aggregated vector is used as the log vector representation. The index encoding sub-model uses a two-layer fully connected network with layer widths of 256 and 128 respectively, and outputs an index vector representation. The two are then concatenated and passed through a fully connected layer with an input dimension of 256 and an output dimension of 128, and L2 normalization is performed to obtain a unified event representation vector for S1 to generate a standardized link event set.

[0130] The training objective function of the deep learning encoding model uses contrastive learning loss and is fixed as follows:

[0131] ;

[0132] in This represents the training loss of a deep learning encoding model. Indicate batch size and take Indicates the index of positive sample pairs within a batch. This indicates the negative sample index within the batch. Indicates the first Log vector representations obtained from log samples through the log coding sub-model Indicates and The first task that belongs to the same task_id and meets the alignment condition The indicator vector representation obtained from each indicator sample through the indicator coding sub-model Indicates the first batch The indicator vector representation of each indicator sample, Represents the cosine similarity function. This represents a temperature coefficient and is set to 0.07, thereby increasing the similarity between the log vector representation and the indicator vector representation corresponding to the same task_id, and decreasing the similarity between different task_ids.

[0133] The training process uses the Adam optimizer and the learning rate is set to... The training rounds are 20, and the validation set is used with... The minimum model parameters are used as the final deep learning encoded model for S1 inference;

[0134] After obtaining the deep learning encoding model and generating a standardized link event set of historical samples, the candidate causal edge generation rules are defined based on the pattern-layer causal knowledge graph, and the instance-layer causal knowledge graph and candidate causal edge set of historical samples are constructed according to S3 and S4. Each task_id subgraph is used as a training sample for the neural causal discovery model. The unified event representation vector of the instance node is concatenated with key observation metrics extracted from event_param in a fixed order to form a node observation vector. The neural causal discovery model is implemented using a structural equation network and is only applied to… The candidate causal edges in the set can be learned edge weight parameters Candidate edge masks are used to ensure that the weights of non-candidate edges are always 0. The training objective function includes a data fitting term and a sparse constraint term. The data fitting term is taken as the mean square error of the observation vectors of all instance nodes in the structural equation network, and the sparse constraint term is taken as the edge strength of all candidate causal edges. Norm and fixed sparsity coefficient The training process uses the Adam optimizer and the learning rate is set to... The batch size is set to 32 task_id subgraphs and iterated until the fitting error of the validation set data converges, resulting in a neural causal discovery model for S5 to learn causal structure and causal effect parameters under the constraint of candidate causal edge set.

[0135] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0136] This invention employs a combined approach of "deep learning encoding model, causal knowledge graph, neural causal discovery, and counterfactual reasoning." It integrates indicator and log data scattered across components such as schedulers, queues, node agents, executors, storage, and networks in an end-to-end process. First, it uses semantic alignment and field normalization to create a standardized set of link events with task identifiers and timestamps, eliminating event fragmentation caused by inconsistent reporting. Then, it explicitly models component dependencies as a causal knowledge graph and generates a set of candidate causal edges, allowing causal structure learning to be completed within a constrained search space. This further integrates scheduling decision events and execution-side anomalies into the same causal structure model. Finally, using scheduling decision events as interventionable variables, counterfactual reasoning is performed to find the minimum cost intervention set under the premise of anomaly recovery conditions, outputting corresponding causal paths and correction suggestions. This enables the link attribution results to drive the scheduling strategy in an executable reverse direction, improving fault location speed and reducing false positives and false negatives.

[0137] To address the aforementioned technical issues, this invention makes scenario-oriented improvements to the algorithm structure: First, it constructs a causal knowledge graph at the pattern layer and an causal knowledge graph at the instance layer, and introduces causal direction constraints, allowed edge type sets, and prohibited edge type sets to impose structural constraints on candidate edges. Combined with time window constraints, it suppresses pseudo-causal edges that do not conform to link dependencies and temporal order, improving the interpretability and stability of causal discovery. Second, it explicitly models scheduling decision events such as migration, preemption, retry, and rate limiting as intervention variables, and implements intervention semantics in counterfactual reasoning by cutting inbound edges and replacing structural equations, enabling scheduling decisions to act causally on execution-side anomalies. Third, it aims at the minimum cost intervention set in the attribution output and introduces feasibility constraints, so that the attribution results not only provide causal paths but also generate scheduling correction strategies that meet parameter value constraints and have controllable costs, thereby better achieving closed-loop governance of end-to-end link anomalies.

Claims

1. A link attribution method based on deep learning and knowledge graphs, characterized in that, include: S1. Collect end-to-end link data of the task, add task identifier and timestamp, call the deep learning encoding model to perform semantic alignment and field normalization on the indicator data and log data in the end-to-end link data, and generate a standardized link event set; S2. Construct a pattern-layer causal knowledge graph based on the component types, event types, and inter-component dependencies involved in the end-to-end link, including a set of entity types, a set of relationship types, causal direction constraints, a set of allowed edge types, and a set of prohibited edge types. S3. Based on the pattern-layer causal knowledge graph, instantiate and map the standardized link event set to construct an instance-layer causal knowledge graph with timestamps. S4. Based on the causal direction constraints, the set of allowed edge types, the set of prohibited edge types, and the temporal order of instance nodes, generate a set of candidate causal edges in the instance layer causal knowledge graph and set the learnable edge parameters. S5. Determine scheduling decision events from the standardized link event set and generate intervention variables, associate them with the instance layer causal knowledge graph, and call the neural causal discovery model to learn the causal structure model under the constraints of the candidate causal edge set, including the causal structure and causal effect parameters. S6. For the target execution side anomaly, determine the preset recovery conditions and cost function, perform counterfactual reasoning based on the causal structure model and with the intervention variables as the interventionable quantities, solve the minimum cost intervention set that minimizes the cost function under the constraint of satisfying the preset recovery conditions, output the corresponding causal path as the link attribution result, and generate a scheduling correction strategy for adjusting the scheduling decision event parameters based on the minimum cost intervention set.

2. The link attribution method based on deep learning and knowledge graphs according to claim 1, characterized in that, S1 includes: The system collects end-to-end link data from task scheduling, queuing, allocation, execution to recycling. The end-to-end link data includes scheduler event data, queue event data, node agent event data, executor event data, storage event data, network event data, and corresponding indicator data and log data. Add a task identifier and timestamp to each data record in the end-to-end link data, and construct the original link dataset based on the data records after adding the identifier; Based on the original link dataset, a deep learning encoding model is invoked, and the deep learning encoding model is used to perform semantic alignment and field normalization on the indicator data and the log data to obtain the normalization result; The deep learning encoding model includes a log encoding sub-model and an indicator encoding sub-model. The log encoding sub-model is used to encode log template sequences or log field sequences to obtain log vector representations. The indicator encoding sub-model is used to aggregate indicator data according to a preset time window and encode it to obtain indicator vector representations. The log vector representation and the indicator vector representation are merged to generate a unified event representation that includes entity identifier, event type, event parameters, timestamp, and vector representation, and then aggregated to form a standardized link event set.

3. The link attribution method based on deep learning and knowledge graphs according to claim 1, characterized in that, S2 include: The entity type set is determined based on the component types involved in the end-to-end link. The entity type set includes the entity types corresponding to schedulers, queues, node agents, executors, storage, and networks. The event type corresponding to the entity type set is determined based on the event types involved in the end-to-end link. A set of relationship types is determined based on the dependencies between components. The set of relationship types is used to represent the dependencies and data flow relationships between the sets of entity types. For each relation type in the set of relation types, a causal direction constraint is configured based on the inter-component dependency relationship. The causal direction constraint is used to limit the allowed edge directions when generating subsequent candidate causal edges. Based on the inter-component dependencies, a set of allowed edge types and a set of prohibited edge types are determined. The set of prohibited edge types includes edge types that violate the inter-component dependencies or the data flow relationships. Construct a pattern-layer causal knowledge graph based on the entity type set, the relation type set, the causal direction constraint, the allowed edge type set, and the prohibited edge type set.

4. The link attribution method based on deep learning and knowledge graphs according to claim 1, characterized in that, S3 includes: Based on the entity type set in the pattern layer causal knowledge graph, the standardized link event set is assigned to events to determine the entity type corresponding to each standardized link event; For each of the standardized link events, an instance node corresponding to the standardized link event is generated based on the task identifier, and a timestamp associated with the standardized link event is retained for the instance node; Based on the set of relation types in the pattern layer causal knowledge graph, instance edges are generated between instance nodes with dependencies, and timestamp information associated with each instance edge is retained. The timestamp information of the instance edge includes the call time or data flow time corresponding to the instance edge. The instance nodes and instance edges are aggregated to construct an instance-layer causal knowledge graph.

5. The link attribution method based on deep learning and knowledge graphs according to claim 1, characterized in that, S4 includes: Based on the causal direction constraints, allowed edge type set, and prohibited edge type set in the pattern layer causal knowledge graph, the starting entity type and target entity type that are allowed to establish causal relationships in the instance layer causal knowledge graph are determined. In the instance-layer causal knowledge graph, for instance node pairs that satisfy the initial entity type and the target entity type and have a dependency relationship, directed candidate causal edges are established according to the causal direction constraints and based on the set of allowed edge types, while candidate causal edges belonging to the set of prohibited edge types are excluded. Candidate causal edges are established only when the timestamp of the starting instance node is earlier than the timestamp of the target instance node, and the time difference between the two is within a preset time window. The candidate causal edges are aggregated to form a candidate causal edge set; For each candidate causal edge in the candidate causal edge set, a corresponding edge weight parameter is set so that the candidate causal edge set carries the edge weight parameter; The preset time window is determined by the upper limit of link observation delay and the upper limit of inter-component call delay.

6. The link attribution method based on deep learning and knowledge graphs according to claim 1, characterized in that, S5 include: Identify scheduling decision events representing migration, preemption, retry, and rate limiting from the standardized link event set, and extract the event type, event occurrence timestamp, and event parameters from each scheduling decision event. Generate intervention variables based on the event type, the event occurrence timestamp, and the event parameters. The intervention variable is associated with the corresponding instance node in the instance layer causal knowledge graph based on the task identifier, and the intervention variable is associated with the corresponding candidate causal edge in the candidate causal edge set based on the event occurrence timestamp. After the intervention variable is associated with the instance layer causal knowledge graph and the candidate causal edge set, the neural causal discovery model is invoked for processing. The neural causal discovery model learns the edge weight parameters of the candidate causal edge set based on graph neural networks or structural equation networks, and constrains the learned causal structure through sparse regularization constraints and acyclic constraints, and outputs a causal structure model, including the causal structure determined by the edge weight parameters and the corresponding causal effect parameters.

7. The link attribution method based on deep learning and knowledge graphs according to claim 1, characterized in that, S6 include: Based on the target execution-side anomaly, preset recovery conditions are determined. The preset recovery conditions include at least one observation indicator corresponding to the target execution-side anomaly satisfying a preset threshold or preset range. The observation indicator includes at least one of execution latency, error rate, throughput, queue waiting latency, node resource utilization, and network packet loss rate. The cost function is determined based on the impact of parameter adjustments on scheduling decision events. The cost function is used to characterize the number of parameter adjustments, the magnitude of parameter adjustments, or the impact of parameter adjustments on task performance. In the causal structure model, the intervention variables are used as modifiable quantities. Counterfactual reasoning is performed on each candidate intervention set composed of the intervention variables to obtain counterfactual results of the target execution side anomaly corresponding to each candidate intervention set. The counterfactual reasoning includes an abductive step, an action step, and a prediction step. The abductive step includes estimating exogenous noise in the causal structure model based on the observed standardized link event set. The action step includes cutting the incoming edges of the instance nodes associated with the intervention variable and replacing the structural equation corresponding to the instance node with an assignment function determined by the event parameters of the scheduling decision event. The prediction step includes calculating the counterfactual result of the target execution-side anomaly on the causal structure model after replacing the structural equation. By performing a combinatorial search or enumeration on the candidate intervention set, a candidate intervention set that minimizes the cost function is selected from the candidate intervention set that satisfies the preset recovery conditions, and this set is determined as the minimum cost intervention set. Based on the causal structure of the causal structure model, extract the causal path from the minimum cost intervention set to the target execution side anomaly, and output the minimum cost intervention set and the causal path together as the link attribution result. Based on the link attribution results, the scheduling decision events and their parameters corresponding to the minimum cost intervention set are converted into parameter adjustment suggestions to generate a scheduling correction strategy.

8. The link attribution method based on deep learning and knowledge graphs according to claim 1, characterized in that, The acquisition of the deep learning coding model includes: A training dataset was constructed based on historical end-to-end link data; Using the training dataset as input, the deep learning coding model is obtained by optimizing the objective function that increases the similarity of the unified event representation of the indicator data and log data corresponding to the same task identifier and decreases the similarity of the unified event representation of the indicator data and log data corresponding to different task identifiers. The acquisition of the neural causal discovery model includes: Based on the candidate causal edge set defined by the pattern layer causal knowledge graph, the neural causal discovery model is obtained by optimizing the objective function, which includes data fitting terms and sparse constraint terms.

9. The link attribution method based on deep learning and knowledge graphs according to claim 6, characterized in that, The objective function of the neural causal discovery model further includes: a prior consistency regularization term and an intervention consistency loss term. The prior consistency regularization term is used to increase the edge weight parameters of candidate causal edges that are consistent with the causal knowledge graph of the pattern layer and suppress the edge weight parameters of other candidate causal edges. The intervention consistency loss term is used to increase the consistency between the abnormal prediction changes and the observed changes caused by the changes in the intervention variable on the target execution side.

10. The link attribution method based on deep learning and knowledge graphs according to claim 7, characterized in that, The solution to the minimum cost intervention set also satisfies feasibility constraints, which include: The scheduling decision event parameters corresponding to the intervention variables satisfy the parameter value constraint set, and are solved by mixed integer programming or gradient optimization with projection operators, so that the minimum cost intervention set obtained by the solution is an executable parameter adjustment set.

Images