A multi-source heterogeneous log feature fusion and root cause intelligent positioning method and system for big data Spark job

Abstract

The application discloses a multi-source heterogeneous log feature fusion and root cause intelligent positioning method and system for big data Spark jobs. The method collects Spark job running original logs from multiple cluster components and nodes, generates uniform log event sequences through format normalization, timestamp alignment and structured analysis; extracts multi-dimensional heterogeneous fault features, dynamically weights and fuses through attention mechanism, and reduces dimensions to generate fusion feature representation; constructs a Spark job hierarchical execution topology and a fault propagation graph, maps the fusion features to each node of the propagation graph, calculates the root cause confidence through an iterative propagation mechanism, locates to a specific execution component or code line, and outputs a root cause positioning report containing a fault propagation path and a confidence score. The application realizes cross-source log alignment and multi-dimensional feature fusion, can accurately locate the root cause to the code line level, and significantly improves the efficiency and accuracy of Spark job fault diagnosis.

Description

Technical Field

[0001] This application relates to the fields of big data processing and intelligent operation and maintenance technology, and in particular to a method and system for multi-source heterogeneous log feature fusion and root cause intelligent localization for big data Spark jobs. Background Technology

[0002] With the widespread adoption of big data technology and the Spark distributed computing framework, the scale and complexity of modern big data platform operations have increased dramatically. The operational data generated during their runtime, such as Driver logs, Executor logs, event logs, and container logs, are characterized by massive volume, multiple sources, and heterogeneity. Intelligent Operations and Maintenance (AIOps) technology has emerged to address this, aiming to automatically identify problems from this complex operational data through big data and artificial intelligence algorithms, thereby ensuring the stability and reliability of big data operations. Among these technologies, the comprehensive analysis of multi-source heterogeneous log data from Spark jobs and the automatic root cause localization are core technical components for achieving automated and intelligent big data operations and maintenance, greatly assisting operations personnel in fault diagnosis and job recovery.

[0003] While existing technologies possess some log analysis capabilities, they still face significant challenges in practical applications. First, current solutions primarily analyze general logs or single-type logs, failing to effectively collect, normalize, and align heterogeneous logs from multiple dimensions such as Driver, Executor, History Server, and YARN. This creates "data silos," preventing a global perspective on the runtime status of Spark jobs. Second, traditional log feature extraction methods often focus on a single category of textual semantics or statistical features, lacking effective integration of multi-dimensional heterogeneous features such as abnormal keywords, stack traces, execution stage latency, and resource usage metrics. The detection results are often one-sided and singular. Most critically, existing technologies generally lack the ability to pinpoint the root cause of Spark job execution topologies. When a job fails, operations personnel still rely heavily on manual experience, cross-referencing and troubleshooting job dependencies across multiple tools. This process is time-consuming, labor-intensive, inefficient, and prone to misjudgment, making it difficult to quickly and accurately pinpoint the root cause to the specific Executor, Stage, or even line of code. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for multi-source heterogeneous log feature fusion and root cause intelligent localization for big data Spark jobs. It can realize unified collection and accurate time alignment of cross-source logs, effectively integrate multi-dimensional heterogeneous fault features, and accurately locate the root cause to the line level based on the hierarchical topology of Spark jobs, significantly improving the efficiency and accuracy of fault diagnosis.

[0005] Firstly, this application proposes a method for multi-source heterogeneous log feature fusion and root cause intelligent localization for big data Spark jobs, including the following steps: Raw log data generated during the execution of Spark jobs is collected from multiple log sources, including YARN ResourceManager, YARN NodeManager, Spark History Server, HDFS, and the local file system of cluster nodes. The original log data is subjected to format normalization, timestamp alignment, and structured parsing to generate a unified log event sequence. Each log event in the unified log event sequence includes an aligned timestamp, log source identifier, log level, log content, and metadata fields. Multidimensional heterogeneous features are extracted from the unified log event sequence, including abnormal keyword features, stack trace features, execution phase latency features, and resource usage index features. The multidimensional heterogeneous features are projected onto a unified feature space, and attention weights of various features are calculated based on an attention mechanism and dynamic weighted fusion is performed to generate a fused feature vector. The fused feature vector is then subjected to dimensionality reduction processing to generate a fused feature representation. A Spark job execution topology is constructed based on Spark History Server event logs and YARN application information. A fault propagation graph is constructed based on the Spark job execution topology and historical fault data. The nodes of the fault propagation graph correspond to the components at each level: Application, Job, Stage, Task, and Executor, and the edges correspond to the fault propagation weights between components. The fused feature representation is mapped to each node of the fault propagation graph. The root cause confidence of each node is calculated based on the iterative propagation mechanism. The node with the highest root cause confidence is selected as the root cause node, and the specific Executor, Stage or line of code of the Spark job is located. The output includes a root cause localization report containing root cause node identifiers, fault propagation paths, and confidence scores.

[0006] Furthermore, the process of collecting raw log data generated during the execution of Spark jobs from multiple log sources, and executing it using a multi-source adaptable collector with a pluggable architecture, includes: Configure an independent adapter for each type of log source. All adapters implement the log collection interface in a unified manner. The corresponding adapter is dynamically loaded, unloaded and called according to the source identifier through the adapter registry. Log collection tasks from different sources are submitted to a thread pool for parallel execution via an asynchronous task scheduler, which uniformly controls the collection frequency, timeout, and retry strategy. The asynchronous task scheduler adopts a producer-consumer model. The main thread encapsulates the collection tasks from each source into collection task objects and submits them to a priority blocking queue. The worker thread pool consumes tasks according to source priority and arrival order. Each collection task carries a source type, target job identifier, collection time window, and callback handle. After the task is completed, the result merging module is notified through an asynchronous callback. The adapters include a YARN REST API adapter, an HDFS client adapter, a local file adapter, and an SSH / SCP remote adapter. The YARN REST API adapter maintains an HTTP connection pool and performs incremental parsing on the returned data. The HDFS client adapter supports streaming block reading and Kerberos security authentication. The local file adapter captures new log lines in real time through a file listening service. The SSH / SCP remote adapter supports breakpoint resume and batch compressed transmission. It supports a hot-swappable plugin mechanism based on the service provider interface. When adding a new log source, the corresponding adapter deployment package can be placed in the plugin directory and the refresh interface can be called to register it dynamically. When uninstalling, the unregister interface can be called to release resources without restarting the collection service.

[0007] Furthermore, the timestamp alignment process employs a two-level alignment mechanism, including: The first stage is the NTP bias estimation stage: multiple probe packets are sent to the NTP server, the request sending time, response receiving time and local time returned by the NTP server are recorded for each probe, the round-trip delay and one-way bias estimate are calculated, and abnormal probes with round-trip delay greater than a preset multiple of the median are eliminated. The least squares method is used to fit and merge the fixed deviation and clock drift rate of the log source relative to the reference clock. The original timestamp is initially corrected according to the fixed deviation and the clock drift rate to obtain the first-level aligned timestamp. The second stage is the event causality correction stage: a set of causal constraints is constructed based on the causal prior relationships of events within the Spark job. These causal prior relationships include events such as Application startup before Job submission, Job submission before Stage scheduling, Stage scheduling before Task execution, and Task execution before Task completion. For cross-source event pairs that violate causal constraints, a quadratic programming problem is constructed with the causal constraints as the optimization objective and minimizing the adjustment amount of clock skew from each source as the objective function. Sequential least squares programming is used to solve for the quadratic fine-tuning amount of clock skew from each source. Based on the quadratic fine-tuning amount, the first-level aligned timestamp is refined to obtain the final aligned timestamp. Wherein, the final alignment timestamp satisfies: In the formula This is the original timestamp. For source The estimated fixed bias, For source The estimated clock drift rate, For source The second fine-tuning amount.

[0008] Furthermore, the extraction of multidimensional heterogeneous features from the unified log event sequence includes: The abnormal keyword features are extracted from log text based on a pre-trained language model or keyword dictionary, and include text semantic feature vectors containing abnormal keywords, error patterns, and WARN / ERROR level events. The stack trace features are obtained by performing structured parsing on the exception stack trace, extracting the exception class name, exception method name, source code file path and line number to form a stack trace feature vector; The execution phase latency features are extracted from Spark History Server event logs and cover the execution time, waiting time, number of retries, and data skewness indicators at each level of Application, Job, Stage, and Task. The resource usage metrics are extracted from YARN logs and Executor logs, including CPU utilization, memory usage, off-heap memory, disk I / O throughput, network I / O throughput, and Shuffle read / write metrics. Each feature is associated with a Spark job execution component, binding the feature vector to the corresponding Application, Job, Stage, Task, or Executor node.

[0009] Furthermore, the step of calculating attention weights for various features based on the attention mechanism and performing dynamic weighted fusion includes: The first class original feature vector Through projection matrix and bias vector Projected to a unified dimension , to obtain projection features The projection matrix is ​​initialized using Xavier, and the bias vector is initialized to a zero vector. Learnable query vectors With each projection feature Similarity is calculated, and attention weights are obtained through a scaled dot product attention mechanism. ： In the formula For a set of feature types, query vector Initialize to a standard normal distribution and perform L2 normalization; The projected features are weighted and summed according to the attention weights to obtain the fused feature vector. ; The training process uses historical fault samples as input and manually labeled root cause nodes as supervision labels. It employs the cross-entropy loss function and Adam optimizer for end-to-end training and sets an early stopping mechanism to stop training when the validation set loss no longer decreases.

[0010] Further, the step of performing dimensionality reduction processing on the fused feature vector to generate a fused feature representation includes: The dimensionality reduction method is automatically selected based on the data characteristics. Principal component analysis is used by default, retaining the minimum number of principal components that explain the preset proportion of variance, and generating a dimensionality-reduced fusion feature representation. When the root cause localization accuracy after principal component analysis dimensionality reduction on the validation set is lower than the preset threshold of autoencoder dimensionality reduction accuracy, the system automatically switches to autoencoder dimensionality reduction. The autoencoder adopts a symmetrical structure of input layer-hidden layer-output layer, the activation function is a linear rectified function, the loss function is mean square error, and the hidden layer dimension is determined by the reconstruction error inflection point method. Dimensionally reduced fusion feature representation satisfy: In the formula For activation function, and These are the parameters for the dimensionality reduction layer.

[0011] Furthermore, the construction of the fault propagation graph based on the Spark job execution topology and historical fault data includes: Based on the Spark History Server event logs, reconstruct the hierarchical and data dependencies of Application→Job→Stage→Task→Executor, and construct the Spark job execution topology; Based on historical fault sample statistics, anomalies were observed from nodes. propagation to nodes frequency Combined with the data dependency strength in the Spark job topology Calculate the edge propagation weights: In the formula For nodes The set of direct predecessor nodes, This is a balance coefficient between empirical frequency and topological dependence; Each node in the fault propagation graph stores a corresponding fusion feature component. and local anomaly weights The local anomaly weights are obtained by mapping the fused features through a fully connected layer.

[0012] Furthermore, the calculation of the root cause confidence of each node based on the iterative propagation mechanism includes: Initialize the root cause confidence of each node as follows: ; The confidence propagation is performed repeatedly according to the iterative update equation: In the formula is the propagation factor, which controls the proportion of confidence propagation among neighboring nodes, and its value ranges from [0,1]. When the maximum value of the change in confidence of all nodes is less than the convergence threshold ε Or the number of iterations reaches the maximum number of iterations. When the iteration ends, the result of the last iteration is taken as the final root cause confidence of each node. The propagation weights have been normalized, and the iterative process is a compression mapping that converges to a unique fixed point.

[0013] Furthermore, the selection of the node with the highest root cause confidence as the root cause node, employing a determination strategy combining dynamic and absolute thresholds, includes: Set absolute threshold If the highest confidence level If the value is less than the absolute threshold, it is marked as unable to be automatically located or requires manual verification; Set relative difference threshold If the difference between the highest confidence level and the second highest confidence level If the relative gap is less than the threshold, the top K candidate root causes are output for manual judgment. The absolute threshold and the relative difference threshold are periodically calibrated through a grid search to define the positioning accuracy. The threshold parameter is the proportion of historical samples where the true root cause is successfully located under different thresholds. It is searched within a preset interval with a fixed step size to maximize the localization accuracy.

[0014] Secondly, this application also proposes a multi-source heterogeneous log feature fusion and root cause intelligent localization system for big data Spark jobs, used to perform the above-mentioned methods, including: The multi-source heterogeneous log collection and preprocessing module includes a multi-source adaptor collector, a format normalization unit, a timestamp alignment unit, and a structured parsing unit. It is used to asynchronously collect Spark job logs from multiple log sources and generate a unified log event sequence. The multi-source adaptor collector adopts a pluggable architecture, supports YARN REST API, HDFS client, local file and SSH / SCP adapters, and achieves parallel collection through an asynchronous task scheduler. The feature extraction and fusion module includes an abnormal keyword feature extraction unit, a stack trajectory feature extraction unit, an execution phase delay feature extraction unit, a resource usage index feature extraction unit, an attention mechanism feature fusion network, and a dimensionality reduction unit. It is used to extract four types of heterogeneous features and perform attention-weighted fusion and dimensionality reduction to generate a fused feature representation. The root cause intelligent localization module includes a Spark job topology construction unit, a fault propagation graph construction unit, a root cause confidence iterative calculation unit, and a root cause determination unit. It is used to construct the Spark job execution topology and fault propagation graph, calculate the root cause confidence of each node based on the iterative propagation mechanism, and locate the root cause to a specific Executor, Stage, or line of code. The root cause report generation module is used to output a root cause localization report that includes root cause node identifiers, fault propagation paths, confidence scores, and related component information.

[0015] Compared to existing technologies, this application provides a method and system for multi-source heterogeneous log feature fusion and root cause intelligent localization for big data Spark jobs. By uniformly collecting multi-source heterogeneous logs and performing format normalization and timestamp alignment, a global view of job execution is constructed. Multi-dimensional fault features are fused and root cause reasoning is performed based on Spark's hierarchical topology, realizing the transformation from data silos to feature fusion. This effectively solves the problem of difficult localization caused by scattered logs, single features, and missing topology in existing technologies. It can achieve unified collection and accurate time alignment of cross-source logs, effectively fuse multi-dimensional heterogeneous fault features, and accurately locate the root cause to the line level based on Spark job hierarchical topology, significantly improving the efficiency and accuracy of fault diagnosis. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a method for multi-source heterogeneous log feature fusion and root cause intelligent localization for big data Spark jobs, provided as an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of a multi-source heterogeneous log feature fusion and root cause intelligent localization system for big data Spark jobs, provided as an embodiment of this application. Detailed Implementation

[0018] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0020] like Figure 1 As shown, this application proposes a method for multi-source heterogeneous log feature fusion and root cause intelligent localization for big data Spark jobs, including the following steps: S100 collects raw log data generated during the execution of Spark jobs from multiple log sources, including YARN ResourceManager, YARN NodeManager, Spark History Server, HDFS, and the local file system of cluster nodes. Raw log data refers to unprocessed log information directly obtained from various sources within the Spark job runtime environment. These logs may exist in different formats, such as text files, JSON, or binary formats, and may contain inconsistent timestamps. Specifically, log data can be collected by manually configuring the connection parameters and file paths for each log source and periodically pulling log files using simple scripts or tools. For example, you can configure an HTTP GET request for YARNResourceManager to retrieve log information returned by its REST API; configure an HDFS client for HDFS to download files from specified paths; and configure a local file system configuration file reader to periodically scan for log files in a specified directory.

[0021] S200. Perform format normalization, timestamp alignment, and structured parsing on the raw log data to generate a unified log event sequence. Each log event in the unified log event sequence includes an aligned timestamp, log source identifier, log level, log content, and metadata fields. A unified log event sequence refers to the process of transforming raw log data from different sources into a unified, standardized event stream after format normalization, timestamp alignment, and structured parsing. Format normalization uses preset regular expressions or fixed delimiter rules to parse log text of different formats, extract key fields, and convert them into unified key-value pairs or JSON format. Timestamp alignment simply reads the timestamp field contained in each log event and converts it into a unified UTC time format. Structured parsing uses predefined log templates or pattern matching rules to convert unstructured text in the log content into structured fields. Thus, the processed log events are simply merged into a sequence in chronological order.

[0022] S300. Extract multidimensional heterogeneous features from the unified log event sequence. The multidimensional heterogeneous features include abnormal keyword features, stack trace features, execution phase latency features, and resource usage index features. Anomaly keyword features can be developed by maintaining a static list of anomaly keywords, performing string matching in log text, and counting the frequency of matched keywords. Stack trace features can be developed through simple text pattern recognition, such as searching for lines starting with "at", to identify the stack trace and extract the top-level exception class name and method name. Execution phase latency features can be developed by parsing specific events in the Spark History Server event logs and manually calculating the start and end times of stages to obtain the execution duration. Resource usage metrics features can be developed by periodically extracting metrics such as CPU usage and memory usage from the YARN UI or Executor logs and performing simple average calculations. The extraction of these features aims to describe the health status of jobs from different dimensions, such as anomaly patterns in text content, error messages in the program execution path, performance bottlenecks in job scheduling and resource allocation, and system resource consumption.

[0023] S400. Project the multidimensional heterogeneous features onto a unified feature space, calculate the attention weights of various features based on the attention mechanism and perform dynamic weighted fusion to generate a fused feature vector, and perform dimensionality reduction processing on the fused feature vector to generate a fused feature representation. A unified feature space refers to representing and comparing features of different types and dimensions in a common mathematical space through projection or transformation. This helps eliminate differences between heterogeneous features, providing a foundation for subsequent fusion and analysis. Subsequently, attention weights for various features are calculated based on an attention mechanism, and dynamic weighted fusion is performed to generate a fused feature vector. The attention mechanism is a machine learning technique that assigns different weights to different parts of the input to highlight information more relevant to the current task. In feature fusion, the attention mechanism can dynamically identify and emphasize feature types that contribute more to root cause localization. The fused feature vector is a single vector representation composed of multidimensional heterogeneous features weighted and fused by the attention mechanism. This vector integrates information from different feature sources, providing a more comprehensive reflection of the complex state of the Spark job. Specifically, projecting to the unified feature space can be achieved by using matrix multiplication with a fixed weight matrix for each feature type, mapping feature vectors of different dimensions to a preset unified dimension. The calculation of the attention mechanism and dynamic weighted fusion can be performed using a simple fully connected layer, concatenating all projected feature vectors, and then outputting a single fused vector through another fully connected layer. Finally, dimensionality reduction is performed on the fused feature vector to generate a fused feature representation. The fused feature representation is a simplified representation obtained after dimensionality reduction of the fused feature vector. Dimensionality reduction helps remove redundant information, improves computational efficiency, and may enhance the discriminative power of the features. Dimensionality reduction can be performed using a fixed-dimensional truncation method, such as directly selecting the first N dimensions of the fused feature vector as the dimensionality-reduced representation, or using a pre-defined linear projection matrix for dimensionality reduction.

[0024] S500: Construct a Spark job execution topology based on Spark History Server event logs and YARN application information; construct a fault propagation graph based on the Spark job execution topology and historical fault data; the nodes of the fault propagation graph correspond to the components at each level of Application, Job, Stage, Task, and Executor, and the edges correspond to the fault propagation weights between components. Spark job execution topology is a graph structure describing the dependencies and execution flow between components within a Spark job (such as Application, Job, Stage, Task, and Executor). This topology graph visually displays the job's hierarchical structure and data flow. Fault propagation graphs, built upon the Spark job execution topology and incorporating historical fault data, represent the propagation paths and intensity of faults between different components. Nodes in this graph correspond to components at each level (Application, Job, Stage, Task, Executor), and edges correspond to the fault propagation weights between components. Specifically, Spark job execution topology can identify parent-child relationships between Application, Job, Stage, and Task by parsing Spark History Server event logs and constructing a simple tree structure. Fault propagation graphs can use the number of times a component fails in its direct downstream components as propagation weights, based on historical fault data.

[0025] S700. Map the fused feature representation to each node of the fault propagation graph, calculate the root cause confidence of each node based on the iterative propagation mechanism, select the node with the highest root cause confidence as the root cause node, and locate the specific Executor, Stage or line of code of the Spark job. Root cause confidence is a numerical value calculated through an iterative propagation mechanism, representing the probability or likelihood that a certain job component is the root cause of a failure. The higher the confidence, the greater the likelihood that the component is the root cause. Specifically, the fused feature representation of each node can be directly used as the initial "anomaly score" for that node. Then, a simple iterative algorithm based on the PageRank concept is used, where the confidence of each node is the weighted sum of the confidences of its neighboring nodes, iterating until convergence. Subsequently, the node with the highest root cause confidence is selected as the root cause node. The root cause node is the node in the failure propagation graph that is determined to have the highest root cause confidence. This node represents the direct or primary cause of the Spark job failure. This allows locating a specific Executor, Stage, or line of code within the Spark job. Selecting the node with the highest root cause confidence can directly choose the single node with the highest confidence after the iteration is complete as the root cause. For locating a specific Executor, Stage, or line of code, its identifier can be directly output based on the type of the root cause node (Executor, Stage).

[0026] S800 outputs a root cause localization report that includes root cause node identifiers, fault propagation paths, and confidence scores.

[0027] A root cause analysis report is a document that provides a detailed description of the root cause of a fault, which is the final output of the system. This report typically includes the identifier of the root cause node, the propagation path of the fault in the system, and the confidence score of each node, providing maintenance personnel with a basis for fault diagnosis. Specifically, the root cause node identifier, its direct predecessor and successor nodes in the fault propagation graph, and their confidence scores can be output in plain text format or a simple table format.

[0028] In some possible embodiments, the collection of raw log data generated during the execution of Spark jobs from multiple log sources, executed using a multi-source adaptable collector with a pluggable architecture, includes: Configure an independent adapter for each type of log source. All adapters implement the log collection interface in a unified manner. The corresponding adapter is dynamically loaded, unloaded and called according to the source identifier through the adapter registry. Log collection tasks from different sources are submitted to a thread pool for parallel execution via an asynchronous task scheduler, which uniformly controls the collection frequency, timeout, and retry strategy. The asynchronous task scheduler adopts a producer-consumer model. The main thread encapsulates the collection tasks from each source into collection task objects and submits them to a priority blocking queue. The worker thread pool consumes tasks according to source priority and arrival order. Each collection task carries a source type, target job identifier, collection time window, and callback handle. After the task is completed, the result merging module is notified through an asynchronous callback. The adapters include a YARN REST API adapter, an HDFS client adapter, a local file adapter, and an SSH / SCP remote adapter. The YARN REST API adapter maintains an HTTP connection pool and performs incremental parsing on the returned data. The HDFS client adapter supports streaming block reading and Kerberos security authentication. The local file adapter captures new log lines in real time through a file listening service. The SSH / SCP remote adapter supports breakpoint resume and batch compressed transmission. It supports a hot-swappable plugin mechanism based on the service provider interface. When adding a new log source, the corresponding adapter deployment package can be placed in the plugin directory and the refresh interface can be called to register it dynamically. When uninstalling, the unregister interface can be called to release resources without restarting the collection service.

[0029] This embodiment proposes a pluggable architecture for a multi-source adapter log collector, decoupling the log collection function into pluggable modules. This architecture allows the system to flexibly support different types of log sources without modifying the core collection logic. By encapsulating the collection logic for specific sources in independent adapters, the system can easily cope with the addition or change of log sources, significantly improving the system's scalability and maintainability. For each specific log source, such as the YARNREST API, HDFS file system, local files, or remote SSH / SCP, an independent adapter is configured. Although these adapters handle different data sources and protocols, they all conform to and implement a unified log collection interface. This interface defines the standard behavior of log collection operations, ensuring consistency between different adapters, enabling the upper-layer scheduling and management modules to interact with all adapters in a unified manner, simplifying system design and integration. To achieve high flexibility and resource optimization, the system maintains an adapter registry. When logs from a specific source need to be collected, the system can dynamically load, instantiate, and call the corresponding adapter through the registry based on the unique identifier of that log source. When an adapter is no longer needed, it can also be dynamically unloaded, releasing the occupied system resources. This dynamic management mechanism avoids loading all adapters at system startup, reducing memory usage and allowing acquisition capabilities to be adjusted on demand at runtime.

[0030] To efficiently handle log collection tasks from multiple heterogeneous sources, this application introduces an asynchronous task scheduler. This scheduler encapsulates collection requests from different log sources into independent tasks and submits them to a shared thread pool for parallel processing. This parallel execution significantly improves the throughput and response speed of log collection. Simultaneously, the scheduler also uniformly manages the frequency of collection tasks, sets timeout mechanisms to prevent tasks from blocking for extended periods, and implements retry strategies to address issues such as transient network failures or resource unavailability, thereby ensuring the robustness and reliability of the collection process. The asynchronous task scheduler is specifically implemented using a producer-consumer model. The main thread acts as the "producer," responsible for encapsulating collection tasks from different log sources into collection task objects containing information such as source type, target job identifier, collection time window, and callback handle, based on configuration or real-time requirements. These task objects are then submitted to a priority blocking queue. The worker thread pool acts as the "consumer," retrieving tasks from the queue and executing them according to their priority and arrival order. After the task is completed, the result merging module is notified asynchronously through a preset callback handle, realizing non-blocking processing and summarization of the collected results, which further improves the system's concurrent processing capability and efficiency.

[0031] To cover common log sources in Spark jobs, this application provides several specific adapter types. The YARNREST API adapter is specifically designed to retrieve logs and application status information from the YARN ResourceManager and NodeManager via HTTP. It maintains an HTTP connection pool to improve connection reuse efficiency and performs incremental parsing on the returned JSON or XML data, reducing memory consumption. The HDFS client adapter utilizes the HDFS client library to support streaming, chunked reading of log files stored on HDFS and handles Kerberos security authentication, ensuring data access in a secure cluster environment. The local file adapter uses a file monitoring service (such as inotify or Polling) to capture newly added log lines on the local file system of cluster nodes in real time, achieving near real-time log collection. The SSH / SCP remote adapter is used to securely transfer log files from remote nodes, supporting breakpoint resumption to handle network interruptions and batch compression to reduce network bandwidth consumption. These dedicated adapters ensure effective and efficient processing of various log sources. To maximize system flexibility and availability, this application supports a hot-plug mechanism based on the Service Provider Interface (SPI). This means that when a new log source needs to be supported, the newly developed adapter can be packaged as a plugin, its deployment package placed in the preset plugin directory, and a refresh interface called. The system can then dynamically discover and register the new adapter at runtime without stopping or restarting the entire log collection service. Similarly, when an adapter is no longer needed, it can be dynamically unloaded and its occupied resources released by calling the deregistration interface. This hot-swappable capability greatly simplifies the system's upgrade, expansion, and maintenance processes, ensuring the continuous operation and high availability of the log collection service.

[0032] In some possible embodiments, the timestamp alignment process employs a two-level alignment mechanism, including: The first stage is the NTP bias estimation stage: multiple probe packets are sent to the NTP server, the request sending time, response receiving time and local time returned by the NTP server are recorded for each probe, the round-trip delay and one-way bias estimate are calculated, and abnormal probes with round-trip delay greater than a preset multiple of the median are eliminated. The least squares method is used to fit and merge the fixed deviation and clock drift rate of the log source relative to the reference clock. The original timestamp is initially corrected according to the fixed deviation and the clock drift rate to obtain the first-level aligned timestamp. Specifically, to improve the accuracy and robustness of the estimation, the system calculates the round-trip delay and one-way bias estimates for each probe, and removes abnormal probe data with round-trip delays greater than a preset multiple of the median to avoid the impact of instantaneous network fluctuations on the results. Then, using the remaining valid probe data, the system employs least squares to fit the fixed bias and clock drift rate of the log source relative to the reference clock. This fitting method yields a more stable and accurate clock correction model, considering that clock bias is not constant but may drift over time. Finally, based on the fitted fixed bias and clock drift rate, a preliminary correction is performed on the original timestamp to obtain a first-level aligned timestamp, which already possesses a high degree of consistency in physical time.

[0033] The second stage is the event causality correction stage: a set of causal constraints is constructed based on the causal prior relationships of events within the Spark job. These causal prior relationships include events such as Application startup before Job submission, Job submission before Stage scheduling, Stage scheduling before Task execution, and Task execution before Task completion. For cross-source event pairs that violate causal constraints, a quadratic programming problem is constructed with the causal constraints as the optimization objective and minimizing the adjustment amount of clock skew from each source as the objective function. Sequential least squares programming is used to solve for the quadratic fine-tuning amount of clock skew from each source. Based on the quadratic fine-tuning amount, the first-level aligned timestamp is refined to obtain the final aligned timestamp. These prior causal relationships are abstracted into a set of causal constraints. For cross-source event pairs that may still violate causal constraints after first-level alignment, the system constructs a quadratic programming problem with the causal constraints as the optimization objective and minimizing the adjustment amount of clock skew from each source as the objective function. This means that, under the premise of satisfying logical causality, the modification of the timestamp should be as small as possible to preserve the original time information to the greatest extent. The constructed quadratic programming problem is solved using optimization algorithms such as sequential least squares programming to obtain the quadratic fine-tuning amount of clock skew from each source. Finally, the obtained quadratic fine-tuning amount is applied to the first-level aligned timestamp for further fine-tuning to obtain the final aligned timestamp. The final aligned timestamp satisfies: In the formula This is the original timestamp. For source The estimated fixed bias, For source The estimated clock drift rate, For source The formula takes into account both physical clock skew and logical causality correction, ensuring the accuracy of the timestamp.

[0034] Through the aforementioned two-level alignment mechanism, this application effectively solves the problem of inconsistent timestamps in distributed Spark job logs. The first-level NTP bias estimation stage provides a globally unified physical time reference for all log events through physical clock synchronization, effectively correcting system clock drift and fixed biases. Building upon this, the second-level event causality correction stage further introduces logical causal prior knowledge from within the Spark job to fine-tune potential causal order conflicts. This method, combining physical time correction and logical causality correction, ensures that the log event sequence is not only highly consistent in physical time but also accurate in logical order. This provides a high-precision, high-reliability time basis for subsequent multi-dimensional heterogeneous feature extraction (such as execution phase latency features) and fault propagation graph construction, greatly improving the accuracy and efficiency of root cause localization.

[0035] In some possible embodiments, extracting multidimensional heterogeneous features from the unified log event sequence includes: The abnormal keyword features are extracted from log text based on a pre-trained language model or keyword dictionary, and include text semantic feature vectors containing abnormal keywords, error patterns, and WARN / ERROR level events. Anomaly keyword features aim to directly capture specific words, phrases, or patterns in logs that explicitly indicate anomalies, errors, or warnings. In practice, a keyword dictionary containing common anomaly identifiers (such as "ERROR," "FAIL," "EXCEPTION," "OOM," "Timeout," etc.) can be constructed, and string matching or regular expression techniques can be used to quickly identify and extract these features from the log content. Furthermore, to gain a deeper understanding of the semantic information in the log text, pre-trained language models (such as BERT, Word2Vec, etc.) can be used to encode the log text, converting it into high-dimensional semantic vectors, thereby capturing more complex error patterns and contextual information. These extracted keywords, error patterns, or semantic vectors collectively constitute anomaly keyword features, providing direct anomaly signals for subsequent fault analysis.

[0036] The stack trace features are obtained by performing structured parsing on the exception stack trace, extracting the exception class name, exception method name, source code file path and line number to form a stack trace feature vector; A stack trace is a sequence of function calls recorded when a program encounters an exception, precisely indicating the specific code location where the exception occurred. This application uses a specially designed parser or regular expressions to structure the stack trace text appearing in the logs, extracting key information such as the class name that caused the exception (e.g., java.lang.NullPointerException), the specific method name (e.g., com.example.MyClass.myMethod), and the corresponding source file path and line number (e.g., MyClass.java:123). This structured information is then encoded into a stack trace feature vector, for example, through hash encoding, one-hot encoding, or more complex embedding techniques, thereby transforming code-level fault information into a computable numerical representation, providing developers with precise information for quickly locating problems.

[0037] The execution phase latency features are extracted from Spark History Server event logs and cover the execution time, waiting time, number of retries, and data skewness indicators at each level of Application, Job, Stage, and Task. Performance issues in Spark jobs often manifest in the time consumption of different execution stages. This application, by parsing event logs generated by the Spark History Server (such as SparkListenerApplicationStart, SparkListenerJobStart, SparkListenerStageCompleted, SparkListenerTaskEnd events), can accurately extract the execution duration (total time from start to finish), waiting time (time spent waiting for resources in the scheduling queue), retries (number of times re-execution due to failure), and data skew metrics of Spark jobs at different levels such as Application, Job, Stage, and Task. Data skew can be measured by calculating the variance of the amount of data processed by a Task or the ratio of the maximum value to the average value. These metrics collectively constitute the latency characteristics of the execution stages, effectively revealing potential faults such as performance bottlenecks, scheduling efficiency issues, or uneven data distribution in Spark jobs.

[0038] The resource usage metrics are extracted from YARN logs and Executor logs, including CPU utilization, memory usage, off-heap memory, disk I / O throughput, network I / O throughput, and Shuffle read / write metrics. Resource bottlenecks are a common cause of Spark job failures. This application analyzes YARN NodeManager logs, YARN ResourceManager API data, and Spark Executor's own logs to extract key metrics related to resource consumption. These metrics include CPU utilization (CPU utilization of the Executor or Container), memory usage (including heap memory, non-heap memory, and off-heap memory consumption), disk I / O throughput (the amount of data read and written to disk), network I / O throughput (the amount of data transferred over the network), and Shuffle read / write operations (the amount of data read and written during Spark Shuffle). These resource usage metrics directly reflect whether Spark jobs experience CPU contention, Out of Memory (OOM), disk I / O bottlenecks, or inefficient network transmission during operation, providing a quantitative basis for identifying resource-related failures.

[0039] Each feature is associated with a Spark job execution component, binding the feature vector to the corresponding Application, Job, Stage, Task, or Executor node.

[0040] To ensure that the extracted feature information is accurately applied to its corresponding fault domain, this application utilizes unique identifiers (such as Application ID, Job ID, Stage ID, Task ID, and Executor ID) contained in the logs during the feature extraction process to establish a clear logical association between the extracted feature vectors and the corresponding components in the Spark job execution topology. For example, the stack trace feature of a specific Task will be bound to that Task node, while the CPU utilization feature of an Executor will be bound to the corresponding Executor node. This association mapping is maintained by constructing data structures (such as hash tables or graph structures), ensuring that feature information can be accurately assigned to the components it affects, providing a solid foundation for subsequent fault propagation graph construction and root cause confidence calculation.

[0041] In some possible embodiments, the step of calculating attention weights for various features based on an attention mechanism and performing dynamic weighted fusion includes: The first class original feature vector Through projection matrix and bias vector Projected to a unified dimension , to obtain projection features , wherein the projection matrix It is a learnable weight matrix whose function is to weight the original feature vectors. The bias vector maps from its original dimension to a unified dimension d to eliminate dimensional differences between heterogeneous features; It is a learnable bias term. The ReLU (Rectified Linear Unit) activation function is applied to the linear transformation result to introduce nonlinearity and enhance the expressive power of the model. To ensure the stability and efficiency of model training, the projection matrix is ​​initialized with Xavier, an initialization method designed to keep the variance of activation values ​​and gradients stable during forward and backward propagation, helping to avoid gradient vanishing or gradient exploding problems; the bias vector is initialized to a zero vector.

[0042] Learnable query vectors With each projection feature Similarity is calculated, and attention weights are obtained through a scaled dot product attention mechanism. ： In the formula For a set of feature types, a query vector can be learned. It is a projection feature Vectors of the same dimension are learned through backpropagation during training to capture task-relevant feature preferences, i.e., which types of features the model wants to focus on. The scaled dot product attention mechanism computes the query vector... With each projection feature The dot product similarity between them, divided by the dimension. The square root ( The scale is applied to prevent the dot product from becoming too large, which would cause the gradient of the softmax function to be too small. Then, the similarity score is converted into a probability distribution between 0 and 1 using the softmax function, ensuring that all attention weights... The sum of these values ​​equals 1, thus representing the importance ratio of different feature types. Query vector Initialize to a standard normal distribution and perform L2 normalization; to provide a random but bounded initial state and prevent its value from being too large or too small, thereby stabilizing the similarity calculation.

[0043] The projected features are weighted and summed according to the attention weights to obtain the fused feature vector. This step is the final fusion stage of the attention mechanism, which involves fusing each projected feature. Multiply by its corresponding attention weight Then, all the weighted features are summed to obtain a comprehensive fused feature vector. The fused feature vector It contains information about all heterogeneous features, and the importance of different features is determined by their weights. It is dynamically reflected.

[0044] The training process uses historical fault samples as input and manually labeled root cause nodes as supervision labels. It employs the cross-entropy loss function and Adam optimizer for end-to-end training and sets an early stopping mechanism to stop training when the validation set loss no longer decreases.

[0045] The training process uses historical failure samples as input and manually labeled root cause nodes as supervision labels, employing end-to-end training with the cross-entropy loss function and the Adam optimizer. The cross-entropy loss function measures the difference between the model's predicted root cause distribution and the true root cause distribution, aiming to minimize this difference. The Adam optimizer is an adaptive learning rate optimization algorithm that efficiently adjusts the learning rate, accelerating model convergence. End-to-end training means that all parameters, from feature projection to attention fusion, and finally to the root cause prediction layer (if present), are trained as a whole. Furthermore, an early stopping mechanism is implemented, stopping training when the validation set loss no longer decreases. This is an effective regularization technique to prevent the model from overlearning on the training set and losing its generalization ability.

[0046] In some possible embodiments, performing dimensionality reduction on the fused feature vector to generate a fused feature representation includes: The dimensionality reduction method is automatically selected based on the data characteristics. Principal component analysis is used by default, retaining the minimum number of principal components that explain the preset proportion of variance, and generating a dimensionality-reduced fusion feature representation. This automatic selection mechanism aims to intelligently determine the appropriate dimensionality reduction strategy based on the inherent characteristics of the current Spark job log data. This avoids the limitations of human experience-based judgment, making the dimensionality reduction process more adaptive. Principal Component Analysis (PCA) is a classic linear dimensionality reduction technique. Its core idea is to project the original data onto a new set of orthogonal coordinate axes, called principal components, through orthogonal transformation. Each principal component represents the direction of the largest variance in the data, i.e., it contains the most information. In implementation, the covariance matrix of the fused eigenvectors is first calculated, and then its eigenvalues ​​and eigenvectors are solved. The eigenvectors define the direction of the principal components, while the eigenvalues ​​measure the magnitude of the data variance in that direction. By retaining the minimum number of principal components that explain a preset proportion of variance, such as setting the retention of 95% or 99% of the original data variance, the system can dynamically determine the number of principal components to retain, thereby achieving effective dimensionality compression while maximizing the preservation of data information.

[0047] When the root cause localization accuracy after principal component analysis dimensionality reduction on the validation set is lower than the preset threshold of autoencoder dimensionality reduction accuracy, the system automatically switches to autoencoder dimensionality reduction. The autoencoder adopts a symmetrical structure of input layer-hidden layer-output layer, the activation function is a linear rectified function, the loss function is mean square error, and the hidden layer dimension is determined by the reconstruction error inflection point method. This performance-based dynamic switching mechanism reflects the adaptability of the method. An autoencoder is an unsupervised neural network model, whose objective is to learn an effective, low-dimensional encoded representation of input data. It consists of an encoder (which maps high-dimensional inputs to a low-dimensional hidden layer) and a decoder (which reconstructs the low-dimensional hidden layer back to the original input dimension). If the input layer has D neurons and the hidden layer has d neurons (d<D), the output layer typically also has D neurons to realize the reconstruction of the input. The activation function is Rectified Linear Unit (ReLU), i.e., f(x) = max(0, x), which helps introduce non-linear capability, alleviate the gradient vanishing problem, and accelerate model training. The loss function is Mean Squared Error (MSE), whose objective is to minimize the difference between the reconstructed output and the original input, thereby forcing the hidden layer to learn a compact representation of the input data. The hidden layer dimension is determined by the reconstruction error inflection point method, which means that by experimenting with reconstruction errors under different hidden layer dimensions, the inflection point where the error decline trend slows down significantly is found, and the dimension corresponding to this inflection point is regarded as the optimal low-dimensional representation, as it achieves the maximum dimensionality reduction while ensuring reconstruction quality.

[0048] The fused feature representation after dimensionality reduction satisfies: , where is the activation function, and are parameters of the dimensionality reduction layer; This expression describes the mapping process from the fused feature vector to the fused feature representation after dimensionality reduction . Herein, and respectively represent the weight matrix and the bias vector of the dimensionality reduction layer, which are obtained through training and learning. denotes the activation function, which can be a linear function or a non-linear function (such as ReLU), and is used to introduce non-linear transformation capability so as to better capture complex patterns in data.

[0049] In some possible embodiments, the constructing a fault propagation graph based on the Spark job execution topology and historical fault data comprises: restoring the hierarchical dependency relationship and data dependency relationship of Application→Job→Stage→Task→Executor according to the Spark History Server event log, and constructing the Spark job execution topology; This step, by parsing various events recorded in the Spark History Server logs, such as Application startup, Job submission, Stage scheduling, Task execution, and Executor lifecycle events, clearly establishes the parent-child hierarchical relationships between components such as Application, Job, Stage, Task, and Executor. Simultaneously, by analyzing Shuffle read / write metrics and RDD lineage information, data dependencies between different Stages can be identified, thus forming a complete Directed Acyclic Graph (DAG) structure that clearly depicts the execution flow and data direction of Spark jobs.

[0050] Based on historical fault sample statistics, anomalies were observed from nodes. propagation to nodes frequency Combined with the data dependency strength in the Spark job topology Calculate the edge propagation weights: To quantify the propagation intensity of faults across different components, this application utilizes historical fault sample data to statistically analyze the propagation intensity of anomalies from any predecessor node. Propagate to successor nodes frequency This reflects the empirical probability that a failure in a specific component will lead to a failure in downstream components in real-world failure scenarios. Simultaneously, it considers the nodes in the already constructed Spark job topology. With nodes Data dependency strength between For example, this can be quantified by measuring the amount of data transmitted or the tightness of dependencies. In the formula... For nodes The set of direct predecessor nodes, This is a balance coefficient between empirical frequency and topological dependence; used to adjust historical empirical frequencies. Dependence strength with topological data The relative importance in weight calculation. This weight. After normalization, the sum of weights from all direct predecessor nodes to node v is consistent, thus accurately reflecting the tendency of faults to propagate along the topology.

[0051] Each node in the fault propagation graph stores a corresponding fusion feature component. and local anomaly weights The local anomaly weights are obtained by mapping the fused features through a fully connected layer; In the constructed fault propagation graph, each node (representing a Spark component such as Application, Job, Stage, Task, or Executor) is assigned two key attributes. The first is the fused feature components. This is the feature representation obtained after extracting, fusing, and dimensionality-reducing the relevant log events of the node through the aforementioned steps. It comprehensively reflects the component's operating status and anomaly patterns. Secondly, there is the local anomaly weight. This weight is achieved by fusing feature components. The input is passed to a pre-trained fully connected layer (e.g., a single-layer or multi-layer perceptron) and mapped to obtain the local anomaly weights. This represents the initial degree of anomalousness or suspiciousness of the node before considering propagation effects, providing an initial anomalous signal for the subsequent iterative propagation mechanism.

[0052] In some possible embodiments, the calculation of the root cause confidence of each node based on the iterative propagation mechanism includes: Initialize the root cause confidence of each node as follows: ; Before the iterative propagation process begins, each node in the fault propagation graph needs to be configured. Set an initial root cause confidence level. This initialization step is designed to integrate nodes. Local anomaly weights and its fused feature components This provides a reasonable starting point for subsequent iterative propagation. Local anomaly weights. This reflects the degree of anomaly exhibited by the node based on its own characteristics, while the fused feature components... This includes a comprehensive representation of the node's multidimensional heterogeneous log features. By multiplying the two, the node can be initially quantified. As a potential root cause, it provides an initial assessment of iterative propagation based on local information.

[0053] The confidence propagation is performed repeatedly according to the iterative update equation: In the formula is the propagation factor, which controls the proportion of confidence propagation among neighboring nodes, and its value ranges from [0,1]. Confidence propagation is the core mechanism of this method. It iteratively transmits and updates fault information between nodes in the fault propagation graph. In each iteration... In the middle, node New root cause confidence It consists of two parts: one part comes from its predecessor node. Confidence propagation, i.e. The other part is nodes. Its own local abnormal information, that is .in, Represents a node The set of all direct predecessor nodes, It is a predecessor node To the node The propagation weight reflects the fault's spread from... spread to The probability or intensity of transmission. Transmission factor. It is a hyperparameter between 0 and 1, used to balance the contribution ratio of propagated information from neighboring nodes and local anomalies from the node itself during the update process. When The propagation effect is stronger when the value is close to 1; when... When the value is close to 0, the local information of the node itself has a greater impact. This iterative update mechanism allows the root cause confidence to comprehensively consider both the local manifestation of the fault and its propagation path in the topology.

[0054] When the maximum value of the change in confidence of all nodes is less than the convergence threshold ε Or the number of iterations reaches the maximum number of iterations. When the iteration ends, the result of the last iteration is taken as the final root cause confidence of each node. The propagation weights have been normalized, and the iterative process is a compression mapping that converges to a unique fixed point. A clear termination condition is set: when the change in root cause confidence of all nodes between two consecutive iterations (usually the maximum absolute value) is less than a preset convergence threshold ε, it indicates that the system has stabilized and can be considered to have reached convergence, at which point the iteration terminates. Furthermore, to prevent the iteration process from getting stuck in an infinite loop or converging too slowly, a maximum number of iterations is also set. Once the number of iterations reaches... Even if it hasn't fully converged, the iteration will be forcibly terminated. Finally, the confidence scores of each node calculated in the last iteration are used as its final root cause confidence scores. The propagation weights have been normalized, and the iterative process is a compression mapping that converges to a unique fixed point. Propagation weights The normalization process ensures that the sum of confidence scores propagated from predecessor nodes does not grow or decay indefinitely, thus maintaining numerical stability. The iterative process is designed as a compression mapping, meaning that the magnitude of change in the node confidence vector gradually decreases in each iteration. An important property of compression mappings is that they have a unique fixed point in the complete metric space, and any iterative sequence converges to this fixed point. Therefore, this design guarantees that the iterative process will eventually converge to a stable and unique root cause confidence distribution, avoiding uncertainty or divergence in the results.

[0055] In some possible embodiments, the selection of the node with the highest root cause confidence as the root cause node, using a determination strategy combining dynamic thresholds and absolute thresholds, includes: Set absolute threshold If the highest confidence level If the value is less than the absolute threshold, it is marked as unable to be automatically located or requires manual verification; The absolute threshold Used to assess the highest confidence level The absolute reliability. If the calculated highest confidence level... Less than the absolute threshold This indicates that even the node with the highest confidence level has not reached the preset minimum reliability standard. In this case, the system will mark it as unable to automatically locate the root cause, or suggest manual review to avoid misjudgments due to insufficient model confidence. The absolute threshold... This setting ensures that only results with sufficient confidence will be adopted by the system as the root cause for automatic location.

[0056] Set relative difference threshold If the difference between the highest confidence level and the second highest confidence level If the relative gap is less than the threshold, the top K candidate root causes are output for manual judgment. The relative difference threshold Used to assess the highest confidence level With the second highest confidence level The significance of the differences between them. If the highest confidence level... With the second highest confidence level difference (Right now () less than the relative gap threshold This means that the confidence levels of the top-ranked candidate root causes are very close, making it difficult for the model to clearly distinguish which is the sole root cause. In such ambiguous situations, the system does not force a selection of a root cause but instead outputs the top K candidate root causes for human judgment. This allows experts to intervene, combining their domain knowledge to make the final decision, thereby improving the accuracy and practicality of root cause localization in complex scenarios.

[0057] The absolute threshold and the relative difference threshold are periodically calibrated through a grid search to define the positioning accuracy. The threshold parameter that maximizes the localization accuracy is searched within a preset range by a fixed step size, representing the proportion of the true root cause successfully located in historical samples under different thresholds. During the calibration process, the positioning accuracy is first defined. This represents the proportion of historical samples where the true root cause was successfully located under different threshold combinations. Then, within a preset threshold range, all possible threshold combinations are systematically searched with a fixed step size. For each combination, its corresponding localization accuracy is calculated. And ultimately choose the one that improves positioning accuracy. The largest threshold parameter is used as the current optimal absolute threshold. and relative gap threshold This dynamic calibration mechanism enables the root cause localization method to continuously optimize and adapt to changes in the system's operating environment, ensuring its long-term effectiveness and robustness.

[0058] It should be noted that the above description describes some embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0059] Based on the same inventive concept, and corresponding to the methods of any of the above embodiments, this application also proposes a multi-source heterogeneous log feature fusion and root cause intelligent localization system for big data Spark jobs, used to perform the above methods, such as... Figure 2 As shown.

[0060] The multi-source heterogeneous log collection and preprocessing module 100 includes a multi-source adaptor collector, a format normalization unit, a timestamp alignment unit, and a structured parsing unit. It is used to asynchronously collect Spark job logs from multiple log sources and generate a unified log event sequence. The multi-source adaptor collector adopts a pluggable architecture, supports YARN REST API, HDFS client, local file and SSH / SCP adapters, and achieves parallel collection through an asynchronous task scheduler. The feature extraction and fusion module 200 includes an abnormal keyword feature extraction unit, a stack trajectory feature extraction unit, an execution phase delay feature extraction unit, a resource usage index feature extraction unit, an attention mechanism feature fusion network, and a dimensionality reduction unit. It is used to extract four types of heterogeneous features and perform attention-weighted fusion and dimensionality reduction to generate a fused feature representation. The root cause intelligent localization module 300 includes a Spark job topology construction unit, a fault propagation graph construction unit, a root cause confidence iterative calculation unit, and a root cause determination unit. It is used to construct the Spark job execution topology and fault propagation graph, calculate the root cause confidence of each node based on the iterative propagation mechanism, and locate the root cause to a specific Executor, Stage, or line of code. The root cause report generation module 400 is used to output a root cause localization report containing root cause node identifiers, fault propagation paths, confidence scores, and related component information.

[0061] The apparatus described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0062] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for multi-source heterogeneous log feature fusion and root cause intelligent localization for big data Spark jobs, characterized in that, Includes the following steps: Raw log data generated during the execution of Spark jobs is collected from multiple log sources, including YARN ResourceManager, YARN NodeManager, Spark History Server, HDFS, and the local file system of cluster nodes. The original log data is subjected to format normalization, timestamp alignment, and structured parsing to generate a unified log event sequence. Each log event in the unified log event sequence includes an aligned timestamp, log source identifier, log level, log content, and metadata fields. Multidimensional heterogeneous features are extracted from the unified log event sequence, including abnormal keyword features, stack trace features, execution phase latency features, and resource usage index features. The multidimensional heterogeneous features are projected onto a unified feature space, and attention weights of various features are calculated based on an attention mechanism and dynamic weighted fusion is performed to generate a fused feature vector. The fused feature vector is then subjected to dimensionality reduction processing to generate a fused feature representation. A Spark job execution topology is constructed based on Spark History Server event logs and YARN application information. A fault propagation graph is constructed based on the Spark job execution topology and historical fault data. The nodes of the fault propagation graph correspond to the components at each level: Application, Job, Stage, Task, and Executor, and the edges correspond to the fault propagation weights between components. The fused feature representation is mapped to each node of the fault propagation graph. The root cause confidence of each node is calculated based on the iterative propagation mechanism. The node with the highest root cause confidence is selected as the root cause node, and the specific Executor, Stage or line of code of the Spark job is located. The output includes a root cause localization report containing root cause node identifiers, fault propagation paths, and confidence scores.

2. The method according to claim 1, characterized in that, The process of collecting raw log data generated during the execution of Spark jobs from multiple log sources, using a pluggable architecture multi-source adaptor collector, includes: Configure an independent adapter for each type of log source. All adapters implement the log collection interface in a unified manner. The corresponding adapter is dynamically loaded, unloaded and called according to the source identifier through the adapter registry. Log collection tasks from different sources are submitted to a thread pool for parallel execution via an asynchronous task scheduler, which uniformly controls the collection frequency, timeout, and retry strategy. The asynchronous task scheduler adopts a producer-consumer model. The main thread encapsulates the collection tasks from each source into collection task objects and submits them to a priority blocking queue. The worker thread pool consumes tasks according to source priority and arrival order. Each collection task carries a source type, target job identifier, collection time window, and callback handle. After the task is completed, the result merging module is notified through an asynchronous callback. The adapters include a YARN REST API adapter, an HDFS client adapter, a local file adapter, and an SSH / SCP remote adapter. The YARN REST API adapter maintains an HTTP connection pool and performs incremental parsing on the returned data. The HDFS client adapter supports streaming block reading and Kerberos security authentication. The local file adapter captures new log lines in real time through a file listening service. The SSH / SCP remote adapter supports breakpoint resume and batch compressed transmission. It supports a hot-swappable plugin mechanism based on the service provider interface. When adding a new log source, the corresponding adapter deployment package can be placed in the plugin directory and the refresh interface can be called to register it dynamically. When uninstalling, the unregister interface can be called to release resources without restarting the collection service.

3. The method according to claim 1, characterized in that, The timestamp alignment process employs a two-level alignment mechanism, including: The first stage is the NTP bias estimation stage: multiple probe packets are sent to the NTP server, the request sending time, response receiving time and local time returned by the NTP server are recorded for each probe, the round-trip delay and one-way bias estimate are calculated, and abnormal probes with round-trip delay greater than a preset multiple of the median are eliminated. The least squares method is used to fit and merge the fixed deviation and clock drift rate of the log source relative to the reference clock. The original timestamp is initially corrected according to the fixed deviation and the clock drift rate to obtain the first-level aligned timestamp. The second stage is the event causality correction stage: a set of causal constraints is constructed based on the causal prior relationships of events within the Spark job. These causal prior relationships include events such as Application startup before Job submission, Job submission before Stage scheduling, Stage scheduling before Task execution, and Task execution before Task completion. For cross-source event pairs that violate causal constraints, a quadratic programming problem is constructed with the causal constraints as the optimization objective and minimizing the adjustment amount of clock skew from each source as the objective function. Sequential least squares programming is used to solve for the quadratic fine-tuning amount of clock skew from each source. Based on the quadratic fine-tuning amount, the first-level aligned timestamp is refined to obtain the final aligned timestamp. Wherein, the final alignment timestamp satisfies: In the formula This is the original timestamp. For source The estimated fixed bias, For source The estimated clock drift rate, For source The second fine-tuning amount.

4. The method according to claim 1, characterized in that, The extraction of multidimensional heterogeneous features from the unified log event sequence includes: The abnormal keyword features are extracted from log text based on a pre-trained language model or keyword dictionary, and include text semantic feature vectors containing abnormal keywords, error patterns, and WARN / ERROR level events. The stack trace features are obtained by performing structured parsing on the exception stack trace, extracting the exception class name, exception method name, source code file path and line number to form a stack trace feature vector; The execution phase latency features are extracted from Spark History Server event logs and cover the execution time, waiting time, number of retries, and data skewness indicators at each level of Application, Job, Stage, and Task. The resource usage metrics are extracted from YARN logs and Executor logs, including CPU utilization, memory usage, off-heap memory, disk I / O throughput, network I / O throughput, and Shuffle read / write metrics. Each feature is associated with a Spark job execution component, binding the feature vector to the corresponding Application, Job, Stage, Task, or Executor node.

5. The method according to claim 1, characterized in that, The process of calculating attention weights for various features based on an attention mechanism and performing dynamic weighted fusion includes: The first class original feature vector Through projection matrix and bias vector Projected to a unified dimension , to obtain projection features The projection matrix is ​​initialized using Xavier, and the bias vector is initialized to a zero vector. Learnable query vectors With each projection feature Similarity is calculated, and attention weights are obtained through a scaled dot product attention mechanism. ： In the formula For a set of feature types, query vector Initialize to a standard normal distribution and perform L2 normalization; The projected features are weighted and summed according to the attention weights to obtain the fused feature vector. ; The training process uses historical fault samples as input and manually labeled root cause nodes as supervision labels. It employs the cross-entropy loss function and Adam optimizer for end-to-end training and sets an early stopping mechanism to stop training when the validation set loss no longer decreases.

6. The method according to claim 5, characterized in that, The step of performing dimensionality reduction processing on the fused feature vector to generate a fused feature representation includes: The dimensionality reduction method is automatically selected based on the data characteristics. Principal component analysis is used by default, retaining the minimum number of principal components that explain the preset proportion of variance, and generating a dimensionality-reduced fusion feature representation. When the root cause localization accuracy after principal component analysis dimensionality reduction on the validation set is lower than the preset threshold of autoencoder dimensionality reduction accuracy, the system automatically switches to autoencoder dimensionality reduction. The autoencoder adopts a symmetrical structure of input layer-hidden layer-output layer, the activation function is a linear rectified function, the loss function is mean square error, and the hidden layer dimension is determined by the reconstruction error inflection point method. Dimensionally reduced fusion feature representation satisfy: In the formula For activation function, and These are the parameters for the dimensionality reduction layer.

7. The method according to claim 1, characterized in that, The construction of the fault propagation graph based on the Spark job execution topology and historical fault data includes: Based on the Spark History Server event logs, reconstruct the hierarchical and data dependencies of Application→Job→Stage→Task→Executor, and construct the Spark job execution topology; Based on historical fault sample statistics, anomalies were observed from nodes. propagation to nodes frequency Combined with the data dependency strength in the Spark job topology Calculate the edge propagation weights: In the formula For nodes The set of direct predecessor nodes, This is a balance coefficient between empirical frequency and topological dependence; Each node in the fault propagation graph stores a corresponding fusion feature component. and local anomaly weights The local anomaly weights are obtained by mapping the fused features through a fully connected layer.

8. The method according to claim 1, characterized in that, The calculation of the root cause confidence of each node based on the iterative propagation mechanism includes: Initialize the root cause confidence of each node as follows: ; The confidence propagation is performed repeatedly according to the iterative update equation: In the formula is the propagation factor, which controls the proportion of confidence propagation among neighboring nodes, and its value ranges from [0,1]. When the maximum value of the change in confidence of all nodes is less than the convergence threshold ε Or the number of iterations reaches the maximum number of iterations. When the iteration ends, the result of the last iteration is taken as the final root cause confidence of each node. The propagation weights have been normalized, and the iterative process is a compression mapping that converges to a unique fixed point.

9. The method according to claim 8, characterized in that, The selection of the node with the highest root cause confidence as the root cause node employs a decision-making strategy combining dynamic and absolute thresholds, including: Set absolute threshold If the highest confidence level If the value is less than the absolute threshold, it is marked as unable to be automatically located or requires manual verification; Set relative difference threshold If the difference between the highest confidence level and the second highest confidence level If the relative gap is less than the threshold, the top K candidate root causes are output for manual judgment. The absolute threshold and the relative difference threshold are periodically calibrated through a grid search to define the positioning accuracy. The threshold parameter is the proportion of historical samples where the true root cause is successfully located under different thresholds. It is searched within a preset interval with a fixed step size to maximize the localization accuracy.

10. A multi-source heterogeneous log feature fusion and root cause intelligent localization system for big data Spark jobs, used to execute the method as described in any one of claims 1 to 9, characterized in that, include: The multi-source heterogeneous log collection and preprocessing module includes a multi-source adaptor collector, a format normalization unit, a timestamp alignment unit, and a structured parsing unit. It is used to asynchronously collect Spark job logs from multiple log sources and generate a unified log event sequence. The multi-source adaptor collector adopts a pluggable architecture, supports YARN REST API, HDFS client, local file and SSH / SCP adapters, and achieves parallel collection through an asynchronous task scheduler. The feature extraction and fusion module includes an abnormal keyword feature extraction unit, a stack trajectory feature extraction unit, an execution phase delay feature extraction unit, a resource usage index feature extraction unit, an attention mechanism feature fusion network, and a dimensionality reduction unit. It is used to extract four types of heterogeneous features and perform attention-weighted fusion and dimensionality reduction to generate a fused feature representation. The root cause intelligent localization module includes a Spark job topology construction unit, a fault propagation graph construction unit, a root cause confidence iterative calculation unit, and a root cause determination unit. It is used to construct the Spark job execution topology and fault propagation graph, calculate the root cause confidence of each node based on the iterative propagation mechanism, and locate the root cause to a specific Executor, Stage, or line of code. The root cause report generation module is used to output a root cause localization report that includes root cause node identifiers, fault propagation paths, confidence scores, and related component information.

Images