Network fault location fast response system based on distributed log analysis

Abstract

The application discloses a network fault positioning fast response system based on distributed log analysis, relates to the technical field of computer network and operation and maintenance, and comprises the following modules: a data acquisition module, which generates time series data frames based on a sliding window; a state vector construction module, which extracts indexes and constructs standardized state vectors; a covariance matrix updating module, which recursively updates a dynamic covariance matrix by using a forgetting factor; a feature space decomposition module, which constructs a residual projection matrix based on dynamic principal component dimension parameters; an anomaly detection module, which calculates residual anomaly energy and determines a fault candidate set based on contribution degrees; and a parameter self-adaptive module, which calculates residual space entropy values and feeds back to adjust principal component dimension parameters of the next frame; and the application realizes adaptive and accurate bounding of gray faults through entropy feedback closed loop and orthogonal projection.

Description

Technical Field

[0001] This invention relates to the field of computer network and operation and maintenance technology, specifically to a rapid response system for network fault location based on distributed log parsing. Background Technology

[0002] With the increasing popularity of cloud-native and microservice architectures, the massive logs and monitoring metrics generated by large-scale distributed systems form the cornerstone of system observability. In order to ensure the high availability of services, intelligent operation and maintenance systems usually rely on automated analysis tools to mine these multi-dimensional time-series data in real time, in order to quickly locate and block risks before a full-blown failure occurs.

[0003] Existing mainstream fault detection technologies often employ multivariate statistical analysis methods. By learning from historical data, a baseline model of the system's normal operating state is established, and then the degree to which real-time data deviates from this baseline model is monitored to determine whether any anomalies exist.

[0004] However, this anomaly detection mechanism based on fixed statistical logic has significant blind spots when dealing with complex "grayscale faults." In highly dynamic distributed environments, many hidden faults (such as resource waiting in the early stages of deadlock or chronic memory leaks) do not initially manifest as drastic changes in the value of a single indicator, but rather as a slight disruption of the synergistic relationship between multiple indicators. This logical flaw of ignoring the micro-distribution characteristics of data makes the system prone to mistakenly filtering early, weak but fatal structural anomaly signals as random noise, resulting in serious missed detections. Or, when the business load undergoes normal pattern drift, it generates a large number of false alarms due to the inability to dynamically adapt.

[0005] Therefore, how to adaptively adjust the sensitivity of the detection model by utilizing the distribution characteristics of the data itself, so as to keenly capture the subtle breaks in the potential logical relationship between indicators, when the values ​​of multidimensional indicators have not yet reached the traditional threshold, is the core technical problem that urgently needs to be solved in the early delimitation of gray-scale faults. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a rapid response system for network fault location based on distributed log parsing.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] In a first aspect, this invention discloses a rapid response system for network fault location based on distributed log parsing, comprising:

[0009] The data acquisition module is used to acquire the distributed log stream of the target system, and performs periodic slicing of the distributed log stream based on a preset sliding time window to generate a time-series data frame sequence arranged in chronological order.

[0010] The state vector construction module is used to extract multiple monitoring indicators from the current time series data frame in the time series data frame sequence, perform standardization processing, and construct a standardized state vector for the current moment containing multiple feature dimensions.

[0011] The covariance matrix update module is used to obtain the reference covariance matrix at the current time. The reference covariance matrix is ​​either a preset initial matrix or a historical covariance matrix corresponding to the previous time series data frame. Based on the preset forgetting factor, the reference covariance matrix is ​​updated according to the standardized state vector to obtain the covariance matrix at the current time.

[0012] The feature space decomposition module is used to obtain the principal component dimension parameters at the current time step. The principal component dimension parameters are either preset initial values ​​or updated values ​​based on feedback from the previous time series data frame. Based on the principal component dimension parameters, the covariance matrix at the current time step is decomposed into feature space to construct the residual projection matrix.

[0013] The anomaly detection module projects the standardized state vector onto the subspace defined by the residual projection matrix to obtain the residual vector and its corresponding anomaly energy value. When the anomaly energy value meets the preset anomaly triggering condition, the contribution of each feature dimension is calculated based on the residual vector, and the set of fault candidate indicators is determined according to the contribution.

[0014] The parameter adaptation module is used to calculate the spatial entropy value of the residual vector in each feature dimension, and determine the principal component dimension parameters used to process the next time series data frame based on the spatial entropy value.

[0015] Secondly, this invention discloses a rapid response method for network fault location based on distributed log parsing, comprising the following steps:

[0016] The distributed log stream of the target system is obtained, and the distributed log stream is periodically sliced ​​based on a preset sliding time window to generate a time-series data frame sequence arranged in chronological order.

[0017] For the current time-series data frame in the time-series data frame sequence, multiple monitoring indicators are extracted and standardized to construct a standardized state vector for the current moment containing multiple feature dimensions.

[0018] Obtain the reference covariance matrix at the current time. The reference covariance matrix can be a preset initial matrix or a historical covariance matrix corresponding to the previous time series data frame. Based on the preset forgetting factor, update the reference covariance matrix according to the standardized state vector to obtain the covariance matrix at the current time.

[0019] Obtain the principal component dimension parameters at the current time step. The principal component dimension parameters are either preset initial values ​​or updated values ​​based on feedback from the previous time series data frame. Based on the principal component dimension parameters, perform feature space decomposition on the covariance matrix at the current time step and construct the residual projection matrix.

[0020] The standardized state vector is projected onto the subspace defined by the residual projection matrix to obtain the residual vector and its corresponding anomaly energy value;

[0021] When the abnormal energy value meets the preset abnormal triggering conditions, the contribution of each feature dimension is calculated based on the residual vector, and the set of fault candidate indicators is determined according to the contribution.

[0022] Calculate the spatial entropy value of the residual vector in each feature dimension, and determine the principal component dimension parameters to be used to process the next time series data frame based on the spatial entropy value.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] 1. By constructing a dynamic covariance matrix and an orthogonal feature space decomposition mechanism, the state of a high-dimensional system is decoupled from the principal component space and the residual space. This mechanism uses the residual projection matrix to map the state vector to the residual subspace, which can keenly capture the subtle disruption of the cooperative relationship between indicators (such as the logical deviation of increased CPU but not increased throughput). Thus, even before the values ​​of each indicator reach the traditional alarm threshold, the abnormal energy value can accurately identify hidden gray-scale faults such as the early stage of deadlock and chronic memory leakage, which significantly reduces the false alarm rate.

[0025] 2. By calculating the spatial entropy value of the residual vector, the system can quantify the disorder of the current residual signal in real time. When the entropy value decreases (indicating the presence of structured anomalies), the system automatically tightens the principal component dimension parameters to improve sensitivity. When the entropy value increases (indicating regression white noise), the system automatically relaxes the boundary to tolerate normal fluctuations. This effectively solves the false alarm problem caused by model rigidity and achieves a dynamic balance between high sensitivity and low false alarm rate.

[0026] 3. The calculated contribution of candidate fault indicators is mapped to the initial fault energy of nodes in the physical topology graph. Combined with the service dependency topology graph, a reverse random walk algorithm is executed. This algorithm utilizes the transitivity of service call relationships to penetrate the complex call relationships between microservices and accurately locate the real source of downstream faults (such as database deadlocks) that cause upstream symptoms (such as high latency). This achieves automated attribution from discrete indicator anomalies to physical node faults, greatly shortening the troubleshooting time for operations and maintenance personnel. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is an overall block diagram of the system according to Embodiment 1 of the present invention;

[0029] Figure 2 This is an overall block diagram of the method in Embodiment 2 of the present invention;

[0030] Figure 3 This is a flowchart illustrating the overall execution process of the method in Embodiment 2 of the present invention. Detailed Implementation

[0031] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Application Overview: In the field of intelligent operation and maintenance of modern large-scale distributed systems, especially in fault delimitation for cloud-native microservice architectures, the co-movement pattern and structured correlation among multi-dimensional monitoring indicators are regarded as key indicators for measuring system health. This high-confidence health status is essentially a self-consistent process of orthogonal projection of multi-dimensional features at the statistical level. That is, load fluctuations at the business level (such as an increase in QPS) should be transmitted linearly or non-linearly to the infrastructure layer (such as an increase in CPU and I / O levels) through the system call chain, using computing resources as the bearing medium, thereby maintaining a stable principal component subset space structure in the feature space defined by the covariance matrix.

[0033] However, existing monitoring technologies lack a dynamic verification mechanism for the consistency of the correlation structure among multi-source heterogeneous indicators, resulting in an inability to accurately identify logical deviations and adaptive lags in the system during the gray-scale fault latency period. Logical deviations manifest as a break in the coupling relationship between business indicators and resource indicators, even though business indicators fluctuate within the normal range (e.g., increased traffic due to deadlock but stagnant resource consumption). Adaptive lags manifest as the detection model failing to follow the conceptual drift of business patterns, rigidly using a fixed-dimensional subspace to explain dynamically changing loads, preventing normal high-frequency business fluctuations from being effectively absorbed by principal components and causing them to overflow into the residual space. Consequently, a strict mathematical correspondence cannot be established between the system's operating state and the projection space of the monitoring model, leading to missed detections of hidden faults or false alarms of normal fluctuations, thus affecting the accuracy of fault location and the timeliness of emergency response.

[0034] For example, in a microservice cluster during a major e-commerce promotion, when a hidden connection pool leak occurs, conventional monitoring systems can only capture the appearance of a single indicator (such as memory usage) not exceeding limits through static threshold analysis, but cannot distinguish whether it is accompanied by an inverse logical decoupling of throughput and response time. Furthermore, when the business load undergoes a drastic but normal mode switch due to promotional activities, the system model only records the appearance of a surge in residual energy, failing to detect the changing characteristics of residual space entropy and the dynamic adaptation requirements of principal component dimension parameters. Specifically, the system misjudges the structural anomaly of connection pool leakage as normal fluctuation and ignores it, or misclassifies normal business surges as abnormal intrusions and triggers false alarms, resulting in continuous interference from invalid alarms to maintenance personnel, and failing to form an adaptive defense mechanism that conforms to the dynamic characteristics of the system.

[0035] If the above problems are not addressed, the fault location system will continue to lose its objective ability to distinguish non-numerical out-of-limit faults. In particular, the failure to identify logical deviations will cause the system to mask deep-seated architectural defects, leading to the deterioration and spread of faults without alarms, and eventually triggering a cascading avalanche. At the same time, the failure to correct adaptive hysteresis will cause residual inflation in the detection model, making the model unable to distinguish between noise and signal, and ultimately losing its sensitivity to weak anomalies. As a result, inaccurate monitoring feedback will systematically hinder the operation and maintenance platform from grasping the deep logic of system operation, affecting the achievement of high availability assurance goals.

[0036] Example 1:

[0037] like Figure 1 As shown, a network fault location and rapid response system based on distributed log parsing includes:

[0038] The data acquisition module is used to acquire the distributed log stream of the target system, and performs periodic slicing of the distributed log stream based on a preset sliding time window to generate a time-series data frame sequence arranged in chronological order.

[0039] As the perception front end of the entire network fault location and rapid response system, the data acquisition module has constructed a proactive collection and time-series reconstruction mechanism for large-scale distributed heterogeneous data sources. In this embodiment, the data acquisition module first establishes a deep connection with the target system, which is typically deployed in a containerized orchestration environment such as Kubernetes and contains hundreds to thousands of microservice nodes. To achieve full capture of distributed log streams, the system adopts a collection architecture based on the Sidecar pattern or DaemonSet, pre-installing lightweight collection probes on each service container or physical host. These probes can intercept unstructured text logs output by business applications (such as error stacks generated by Log4j), access logs generated by middleware (such as Nginx Access Logs), and structured monitoring metrics of the underlying infrastructure (such as CPU utilization, memory level, and network I / O rate in Prometheus format) in real time. To address the "out-of-order arrival" problem caused by inconsistent data production speeds in a distributed environment, these heterogeneous data do not directly enter the analysis process. Instead, they are first aggregated into a high-throughput distributed message queue (such as Apache Kafka) to form a globally ordered raw log stream.

[0040] After acquiring the raw log stream, to transform the continuous streaming data into discrete units that the algorithm model can process, this embodiment designs a preset sliding time window mechanism based on event time. Specifically, this mechanism abandons the traditional approach of relying on data arrival time (processing time) to avoid data attribution errors caused by network jitter. The system is configured with a window length... and sliding step size The two-parameter tuples formed, for example, in real-world high-frequency trading scenarios, include the window length. It can be set to 60 seconds to cover the context information before and after the fault occurred, while the sliding step size... The interval is set to 10 seconds to enable high-frequency fault scanning. This means that every 10 seconds, the system generates an analysis unit containing all data from the past 60 seconds. This overlapping slicing design effectively prevents fault features from being physically cut into two unrelated windows.

[0041] Furthermore, this embodiment introduces a watermark mechanism in its engineering implementation to handle late data. The system maintains a logical clock, and when the logical clock advances to a certain time point... At that time, the system will force a slicing operation to place the timestamp within the interval. All log records and metric data points within the system are physically packaged. During this process, the data acquisition module performs preliminary cleaning and alignment of the raw data: for unstructured logs, the module parses the timestamp field in the header and uniformly calibrates it to a Unix timestamp with millisecond precision; for monitoring metrics sampled at different frequencies (e.g., CPU samples every 5 seconds, while memory samples every 15 seconds), the module uses linear interpolation or zero-order hold algorithms to align them to a unified timeline.

[0042] After the above processing, the originally chaotic, heterogeneous, and asynchronous distributed log stream was reconstructed into a sequence of time-series data frames arranged strictly in chronological order. Each time-series data frame is essentially a snapshot container containing the overall system runtime state within that time window. For example, a data frame might contain a dimension... The log feature matrix (where For the number of log entries, (a feature dimension) and a dimension of The index tensors. These data frames, serving as standard inputs for subsequent processing, not only preserve the causal dependencies of the data in the time dimension, but also retain the spatial coupling relationships between different indicators at the same moment through alignment operations. This provides an accurate and clean data foundation for subsequent analysis, thereby ensuring the purity of the entire fault location system at the source of the data.

[0043] The state vector construction module is used to extract multiple monitoring indicators from the current time series data frame in the time series data frame sequence, perform standardization processing, and construct a standardized state vector for the current moment containing multiple feature dimensions.

[0044] In this embodiment, the design of this module follows the dual principles of feature orthogonalization and dimensional normalization to eliminate the huge differences in physical units and orders of magnitude between different monitoring indicators (such as the percentage of CPU utilization and the number of milliseconds of network latency), and to ensure the numerical stability of subsequent covariance matrix calculations.

[0045] Specifically, the operating logic of this module includes the following sub-steps:

[0046] First, the system pre-configures a multi-dimensional feature mapping configuration table. This configuration table defines the features the system is interested in. Key performance indicators (KPIs) dimensions, such as Represents container CPU utilization. Represents the average latency of HTTP responses. Represents disk I / O throughput, etc. When the state vector construction module receives the current time... When processing time-series data frames, the system extracts corresponding values ​​from the complex structure of the data frame based on this configuration table and assembles them into a... The original feature vector of dimension It is worth noting that if a certain indicator is missing in the current frame, the module will use a strategy based on nearest neighbor interpolation or last observation carried forward to complete it, so as to ensure the integrity of the vector.

[0047] Secondly, this embodiment does not use the extremely time-consuming method of recalculating all historical data, but instead designs a lightweight incremental update mechanism.

[0048] The system memory contains two state variables of the same length as the feature dimension: the current moving mean vector. and the current sliding variance vector When the new original feature vector Upon arrival, the module dynamically updates these two statistics using an exponentially weighted moving average (EWMA) algorithm. This design enables the statistical benchmark to dynamically follow the system's concept drift, adapting to normal fluctuations in workload.

[0049] Next, the module performs the core standardization process. For each dimension of the original feature vector... (in The module uses the latest statistics to perform Z-Score normalization. The calculation logic is shown in the following formula:

[0050]

[0051] in: The transformed standardized state vector is at the th... The value represents the degree to which the current indicator deviates from the historical normal level (i.e., how many standard deviations it has deviated from).

[0052] The original feature vector is at the th Real-time measurement values ​​in each dimension;

[0053] It is an online-maintained moving average, representing the recent baseline level of the indicator;

[0054] It is the sliding standard deviation maintained online, representing the recent fluctuation range of the indicator;

[0055] It is a very small positive number (such as $1e-6$) used to prevent division by zero errors caused by certain indicators remaining unchanged for a long time (standard deviation of 0).

[0056] Through this process, data that originally had very different physical meanings were uniformly mapped into the same dimensionless statistical space. This standardized data representation laid a solid mathematical foundation for the subsequent accurate measurement of the correlation between indicators by the covariance matrix (instead of being dominated by large numerical indicators).

[0057] The covariance matrix update module is used to obtain the reference covariance matrix at the current time. The reference covariance matrix is ​​either a preset initial matrix or a historical covariance matrix corresponding to the previous time series data frame. Based on the preset forgetting factor, the reference covariance matrix is ​​updated according to the standardized state vector to obtain the covariance matrix at the current time.

[0058] This module does not simply statistically analyze the static correlation of historical data, but rather uses an online recursive update mechanism to capture the evolving trends (i.e., concept drift) of the collaborative relationships between multi-dimensional indicators of the system in real time. Specifically, its detailed processing procedure is as follows:

[0059] First, the module performs reference benchmark acquisition and cold start determination. Then, it processes the current time step. When retrieving data, the module first attempts to retrieve the data from the previous time step. Output covariance matrix .

[0060] If the system is in the initial startup state ( If the system has just completed a reset and no historical matrix exists, the module will automatically initialize an identity matrix. (Identity Matrix) as the reference covariance matrix This means that initially, the system assumes that the indicators are independent and have normalized variance. Alternatively, to accelerate convergence, the module can load a sample covariance matrix pre-calculated based on historical golden hour data as... .

[0061] If the system is running continuously, the module will directly update the result from the previous moment. Load it into memory as the reference covariance matrix at the current moment. This chain-like dependency ensures that historical information can be passed down from generation to generation.

[0062] Secondly, the module introduces a forgetting factor adjustment mechanism. The system presets a value range within... Forgetting factors between This parameter physically defines the memory half-life of the system:

[0063] higher (e.g., 0.99) means that the system has a high dependence on historical patterns, strong noise resistance, and is suitable for periods of stable business.

[0064] lower A value of 0.90 (e.g., 0.90) means the system is more forgetful, highly sensitive to instantaneous changes, and can adapt to sudden business changes more quickly. In actual deployments, this factor is usually configured between 0.95 and 0.99.

[0065] Next, the module performs the core recursive update calculation, using the standardized state vector at the current moment. and reference covariance matrix The module generates the covariance matrix at the current time step through weighted fusion. The calculation formula is as follows:

[0066]

[0067] in: It is the instantaneous autocorrelation matrix at the current moment, with dimensions of . This captures the immediate coupling relationships between various indicators at the current moment. For example, if... and If both are large positive values, then the matrix's first... The item shows a significant increase, indicating a strong positive correlation between the two. This represents the preservation of the exponential decay of the historical correlation structure. It is the weight of the new information injection that ensures the conservation of the overall energy of the matrix.

[0068] For example, suppose the system feature dimension Current forgetting factor .

[0069] Input status: Current normalized vector (That is, CPU usage is relatively high, and latency is slightly high). Refer to the covariance matrix. (The state at the previous time step) is assumed to be the identity matrix: (This means that historically, CPU and latency were considered unrelated.)

[0070] Calculate instantaneous correlation: Note the off-diagonal elements. The simultaneous increase in CPU and latency at this moment was captured.

[0071] Weighted fusion update:

[0072] .

[0073] Results Analysis: Updated Covariance Matrix middle, This indicates that the system model has begun to learn a positive correlation between CPU and latency (from 0 to 0.2). If this synchronous increase continues, the off-diagonal element will increase with iterations, and eventually the model will consider high CPU accompanied by high latency as a normal feature of the system (i.e., a principal component).

[0074] In this way, the covariance matrix update module achieves dynamic tracking of complex topological relationships within the system with extremely low computational cost, providing a precise mathematical basis for the subsequent feature space decomposition module to distinguish between normal principal component patterns and abnormal residual patterns.

[0075] The feature space decomposition module is used to obtain the principal component dimension parameters at the current time step. The principal component dimension parameters are either preset initial values ​​or updated values ​​based on feedback from the previous time series data frame. Based on the principal component dimension parameters, the covariance matrix at the current time step is decomposed into feature space to construct the residual projection matrix.

[0076] The current time covariance matrix is ​​output by the covariance matrix update module. The feature space decomposition module acts as a pattern filter for the system. Based on statistical principles, this module mathematically decouples the current multidimensional operating state of the system into principal component space (representing normal business fluctuations and indicator coordination patterns) and residual space (representing noise and potential abnormal patterns). In this embodiment, this process is not static but relies on dynamically adjusted principal component dimension parameters to achieve adaptive anomaly sensitivity control.

[0077] Specifically, first, the module performs eigenvalue decomposition (EVD).

[0078] Module receiving 1-dimensional real symmetric matrix As input, it is decomposed using numerical linear algebra algorithms (such as the QR algorithm or divide-and-conquer method) to solve the characteristic equation. This step will produce two key sets:

[0079] Eigenvalue sequence: The module sorts these feature values ​​in descending order of their numerical values ​​(i.e., ...). Physically, eigenvalues The system data was quantified in the first quarter. The variance (i.e., energy) in each orthogonal direction. Larger eigenvalues ​​typically correspond to major load trends (such as traffic spikes causing simultaneous increases in CPU and memory), while smaller eigenvalues ​​correspond to random noise or weak, unmodeled variations.

[0080] Feature vector set: .in It corresponds to The unit orthogonal vector represents the first... The direction of each change pattern.

[0081] It is worth noting that in real-world scenarios, some monitoring indicators may remain constant for a short period, resulting in zero variance, or there may be a strong linear correlation between indicators. This could lead to an increase in the covariance matrix. This can transform the matrix into a singular or ill-conditioned matrix, thus affecting the numerical stability of eigenvalue decomposition. To address this issue, this embodiment introduces Tikhonov regularization before performing the decomposition. Specifically, the system will... Add a very small perturbation on the diagonal (For example ),Right now This step ensures that the matrix is ​​always positive definite, guaranteeing that the eigenvalue decomposition algorithm can converge stably, and that this small perturbation will not change the principal direction of the eigenvectors, thus not affecting the subsequent anomaly detection accuracy.

[0082] Secondly, the module performs dynamic subspace partitioning and principal subspace construction. In this step, the module obtains the principal component dimension parameters at the current time step. It is worth noting that this parameter It is not a fixed value, but rather a dynamic value derived from the parameter adaptive module based on the residual entropy feedback from the previous time step (or in Always use preset initial values, such as ).

[0083] Based on this parameter The module starts from the sorted set of feature vectors. Before the middle cut vectors They are pieced together horizontally to form a Principal subspace matrix of dimension :

[0084]

[0085] The matrix This forms the normal behavior subspace of the system. Any state change falling within this subspace is considered a legitimate fluctuation that conforms to the current relevance logic by the system.

[0086] Finally, the module constructs the residual projection matrix. To detect anomalies, it's necessary to focus on components that cannot be interpreted by the principal subspace. The module utilizes projection operator theory, through the identity matrix... Subtract the projection operator of the principal subspace Construct the residual projection matrix The calculation formula is as follows:

[0087]

[0088] in: yes An identity matrix of dimension 1 It is the projection matrix that projects a vector onto the principal subspace. It is the projection matrix that projects a vector onto the residual subspace (i.e., the orthogonal complement of the principal subspace).

[0089] For example, suppose the system feature dimension (Dimension 1: CPU, Dimension 2: Latency), the current matrix output by the covariance matrix update module is: .

[0090] Eigenvalue decomposition: Solving for eigenvalues The corresponding unit eigenvector is approximately: (This represents the main direction of change, namely, CPU and latency are positively correlated, and CPU fluctuations are greater.) (This represents the secondary direction, i.e., the direction of the noise).

[0091] Subspace construction: Assume the principal component dimension parameters of the current feedback. (That is, assuming the system is primarily driven by one business model), then the principal subspace matrix Only contains : ;

[0092] Constructing the residual projection matrix: First, calculate the principal projection operator:

[0093]

[0094] Then calculate the residual projection matrix. :

[0095]

[0096] Result Analysis: When the state vector is subsequently... When multiplied by this matrix, all that meet the requirements The component of direction (normal mode) will be significantly reduced, while any component perpendicular to the direction will be reduced. The directional components (i.e., those that violate normal correlations) will be preserved and highlighted. This step ensures that the system can filter out large fluctuations in normal business operations and focus on small but illogical structural anomalies.

[0097] The anomaly detection module projects the standardized state vector onto the subspace defined by the residual projection matrix to obtain the residual vector and its corresponding anomaly energy value. When the anomaly energy value meets the preset anomaly triggering condition, the contribution of each feature dimension is calculated based on the residual vector, and the set of fault candidate indicators is determined according to the contribution.

[0098] Following the feature space decomposition module, the anomaly detection module maps the high-dimensional state vector to an orthogonal residual subspace. By quantifying the energy that cannot be explained by the normal pattern, it determines whether the system has deviated from the established operating logic, and further deconstructs and locates the source of the anomaly at a micro-level.

[0099] Specifically, first, the module performs residual vector projection. The module receives two key inputs: the current-time standardized state vector from the state vector construction module. , and the residual projection matrix from the feature space decomposition module .

[0100] Utilizing the projection transformation principle in linear algebra, the module transforms the state vector through matrix multiplication. Mapping to the residual subspace yields the residual vector. The calculation formula is as follows:

[0101]

[0102] Among them, the residual vector Each component in , representing the first Each feature dimension represents a unique variation after removing normal co-variations with other metrics (principal components). If the system operates under normal logic (e.g., increased traffic leading to increased CPU usage), most of the variation will be absorbed by the principal subspace, and the residual vector should approach a zero vector (containing only white noise).

[0103] Secondly, the module performs anomaly energy quantification and threshold determination. To compress the multidimensional residual vector into a single scalar index for easier monitoring, the module calculates the squared magnitude of the residual vector, i.e., the anomaly energy value (also known as...). (Statistic). The calculation formula is:

[0104]

[0105] In obtaining The module then compares it with preset anomaly triggering conditions. In this embodiment, to adapt to the characteristics of non-Gaussian distributed data, the triggering condition is not a simple fixed threshold, but a confidence limit dynamically calculated based on the chi-squared distribution. The system presets a confidence parameter. (For example ), and estimate the dynamic threshold based on the degrees of freedom of the current model. .

[0106] like If the system determines that the current state is a normal fluctuation, the process will terminate or enter a silent period.

[0107] like The system determines that a grayscale anomaly has occurred and immediately triggers the subsequent root cause localization subprocess.

[0108] Specifically, this embodiment uses the Jackson-Mudholkar approximation method to determine the dynamic threshold. The system is based on a sequence of feature values. Calculate the three partial statistics: , , Based on these three statistics, the dynamic threshold... The calculation formula is as follows:

[0109]

[0110] in, , This represents the standard normal distribution quantile (e.g., 2.33) at a confidence level (e.g., 99%). Using this formula, the system can determine the actual distribution characteristics of the current residual space (derived from the eigenvalues). (Decision), to calculate the precise anomaly detection boundary in real time, rather than relying on fixed values ​​based on experience.

[0111] Finally, the module performs root cause candidate screening based on contribution. If an anomaly is triggered, the module needs to further analyze which metric caused it. Exceeding the limit. The module calculates the contribution of each feature dimension to the total anomaly energy. The calculation formula is as follows:

[0112]

[0113] To determine the set of candidate fault indicators, the module executes the following filtering logic:

[0114] Sorting: All feature dimensions are sorted according to their contribution. Sort from largest to smallest.

[0115] Filter: Select those ranked first. Individual indicators (e.g.) ), or select a cumulative contribution level that reaches a preset percentage (such as ) is a set of indicators.

[0116] Furthermore, in order to address the false alarm problem caused by noise (such as instantaneous network jitter or GC pauses) commonly found in distributed network environments, this embodiment does not directly use the initial candidate indicator list as the delimitation result after initially screening it, but instead introduces a continuous verification mechanism based on a time-series buffer pool.

[0117] Specifically, the system allocates a candidate metric timing buffer pool in memory, which physically manifests as a pool with a capacity of (For example A first-in, first-out (FIFO) queue used to store the most recently accessed items. The initial candidate index list generated from each time-series data frame. After the initial list for the current time $t$ is generated, the system pushes it to the head of the queue and removes it from the oldest historical list, thus maintaining the sliding update of the time window.

[0118] Based on this, the module performs frequency statistics and noise removal. The system iterates through each monitoring metric in the initial list at the current moment, and calculates its value in the entire buffer pool (i.e., past data). The total frequency of occurrence within a given time period (at any given moment) is defined as the time-series persistence characteristic value of this indicator. .

[0119] The module then compares the feature value with a preset persistence threshold. (For example Compare:

[0120] like The system determined that the indicator was transient noise, and considered that its abnormal behavior lacked continuity over time, so it was removed from the list.

[0121] like The system determines that the indicator is a valid fault feature and formally includes it in the final set of candidate fault indicators.

[0122] Through this mechanism, the system can effectively filter out occasional index fluctuations, ensuring that the indexes that subsequently enter the topology analysis have a high degree of fault confidence.

[0123] For example, assume the system dimension (Dimension 1: CPU, Dimension 2: Latency), Input Vector Residual projection matrix .

[0124] Calculate the residual vector: .

[0125] The values ​​here are approximate calculations for ease of understanding. In actual high-precision calculations, if... If the principal component direction is fully met, the residual should be close to 0; Includes abnormal structures, such as (That is, high CPU usage but low latency will result in very significant residuals.)

[0126] Assuming a logical violation occurs, the input... (CPU speeds up but latency decreases, violating the logic of positive correlation).

[0127]

[0128] Calculate the anomalous energy: Assuming a dynamic threshold ,but This triggers an abnormal alarm.

[0129] Calculating contribution and defining boundaries: CPU-based contribution: Contribution of delay dimension: .

[0130] Conclusion: Although the absolute value of CPU (2.0) appears larger than the absolute value of latency (-1.0), there is an anomalous change in the latency dimension in the residual space ( This module contributes the vast majority of abnormal energy. Therefore, latency is identified as the primary candidate indicator for failure, prompting operations personnel to investigate why latency decreases in a seemingly illogical way (e.g., it may be due to probe failure or service circuit breaking causing direct request rejection).

[0131] In this way, the anomaly detection module can accurately capture and delimit grayscale anomalies that are numerically compliant but logically erroneous.

[0132] The parameter adaptation module is used to calculate the spatial entropy value of the residual vector in each feature dimension, and determine the principal component dimension parameters used to process the next time series data frame based on the spatial entropy value.

[0133] This module assesses the health of the model in real time based on the information disorder of the current residual data and dynamically adjusts the observation window (i.e., the principal component dimension parameter) for the next time step. This design enables the system to keenly detect subtle structural changes in the early stages of grayscale faults.

[0134] Specifically, the operating logic of this module is a closed-loop process from micro-information measurement to macro-decision control, and the detailed steps are as follows:

[0135] The module first receives the residual vector from the anomaly detection module. and their corresponding abnormal energy values .

[0136] To measure the degree of clustering of the residual signal across various feature dimensions, the module first transforms the physically distinct residual components into dimensionless probability distributions. Specifically, it calculates the... The proportion of energy carried by each feature dimension in the total anomaly energy, i.e., the normalized probability. : .

[0137] Here, because It will necessarily satisfy and This step mathematically maps the residual vector to a discrete probability distribution function.

[0138] Based on information theory principles, the module uses the Shannon Entropy formula to quantify the disorder of the current residual vector, i.e., the spatial entropy value. :

[0139]

[0140] High Entropy State: When Approaching the maximum value (i.e.) When ), it means The distribution tends to be uniform. This indicates that the residual vector is mainly composed of irregular random white noise, and the anomalous energy of the system is diffuse, suggesting that the current principal component subspace has captured the main structured information well, and the model is in a healthy state.

[0141] Low Entropy State: When A significant decrease means that energy is highly concentrated in a few specific dimensions (i.e., (Sharp distribution). This indicates that the residuals contain structured signals with specific directions. In an operational scenario, this usually means that a novel failure mode (such as a specific deadlock or resource contention) has occurred that is not explained by the principal component space, resulting in underfitting of the model.

[0142] Because single-frame data is highly susceptible to instantaneous network jitter or acquisition noise, directly using... Control measures may cause parameter oscillations. Therefore, the module maintains an entropy-based time-series buffer pool to store the most recent... Each time step (e.g.) Find the entropy value of ) and calculate the smoothed entropy value. :

[0143]

[0144] This step is equivalent to adding a low-pass filter; the smoothing entropy value will only decrease significantly when the system continues to exhibit structured anomalous characteristics.

[0145] The module will smooth the entropy value. With the preset entropy threshold (For example, taking the theoretical maximum entropy) The comparison is performed, and the principal component dimension parameters used to process the next time-series data frame are dynamically adjusted based on the comparison results. :

[0146] Scenario 1: Structured anomaly manifestation ( )

[0147] Judgment: The presence of obvious structural information in the residuals indicates that the current model has assigned too much variance to the principal component space, causing some abnormal features to be masked in the normal pattern.

[0148] Decision: Tighten the model boundaries and reduce the number of principal component dimension parameters in each module.

[0149]

[0150] (in This is the step size, usually set to 1; As a safety net dimension, such as 1).

[0151] By reducing This forces minor features that were originally considered principal components to be squeezed out into the residual space. This amplifies the projection of anomalous signals into the residuals, causing fault features to be exposed earlier in the next time step.

[0152] Scenario 2: Regression to Noise State ( )

[0153] Judgment: The residual regression is uniform noise, indicating that the model is too sensitive or the system has returned to stability.

[0154] Decision: Relax the model boundaries, maintain the module dimensions or slowly increase the principal component dimension parameters:

[0155]

[0156] Increase This allows the model to tolerate more normal business fluctuations (such as periodic loads), thereby reducing the false alarm rate.

[0157] In addition, to ensure the system's basic detection capabilities, this embodiment specifies the principal component dimension parameters. Strict safety barriers have been set up. On the one hand, an upper limit has been set. To ensure that at least one dimension is retained as the residual space, the logical flaw of overfitting leading to perfect fit across all dimensions should be prevented; on the other hand, a lower limit should be set. (Or 5% of the total dimensions) to prevent oversensitivity from causing all fluctuations to be viewed as abnormal. Simultaneously, to cope with the recovery period after system cold starts or data interruptions, when the number of entropy samples in the buffer pool is insufficient... At that time, the system will be forcibly locked. The value is the preset initial value. The adaptive adjustment logic is only activated after a sufficient number of data frames have been accumulated. This mechanism avoids drastic fluctuations in model parameters during the unstable initialization phase.

[0158] Through this parameter adaptive mechanism, the system can automatically adjust to high sensitivity (low sensitivity) without human intervention. ) and low false alarm rate (high By finding a dynamic balance between these factors, intelligent and rapid response to network faults can be achieved.

[0159] While the parameter adaptive module dynamically adjusts the detection model, a topological root cause localization module is further configured to achieve physical tracing from abnormal indicators to fault nodes. This module maps discrete statistical abnormal signals (candidate indicators) back to the physical topology of the system, uses graph algorithms to infer the propagation path of the fault in the service call chain, and thus locates the true source of the fault.

[0160] Specifically, firstly, the module performs panoramic topology construction and entity mapping. Before performing analysis, the system pre-constructs a service dependency topology diagram of the target system by connecting to trace data from a CMDB (Configuration Management Database) or APM (Application Performance Management) system. .

[0161] Node set : Represents the entity unit in the system, including microservice instances, database nodes, and underlying physical machine or container nodes.

[0162] edge set : Represents the calling or deployment relationship between entities. In this embodiment, the direction of the edge is specifically defined as the service calling direction (i.e., the request initiator points to the responder), which forms the basis for the subsequent random walk.

[0163] After obtaining the set of candidate fault indicators, the module uses a pre-built metadata mapping table to map each monitoring indicator in the set to a specific node in the topology graph. For example, the indicator service_A_response_time is mapped to the node Service_A, and the indicator db_cpu_usage is mapped to the node Database_Node_1.

[0164] Secondly, the module performs fault energy injection. In order to transform the severity of anomalies at the statistical level into physical states at the graph level, the module initializes the fault energy of each node in the topology graph based on the contribution calculated in the previous steps.

[0165] The system constructs an initial fault energy vector equal to the number of nodes. For each node in the topology Its initial energy value is the sum of the contributions of all candidate indicators mapped to the node;

[0166] For nodes not associated with any abnormal indicators, their initial energy is set to 0. Subsequently, for... Perform L1 normalization to make it a valid probability distribution vector.

[0167] Finally, the module performs a reverse random walk to locate the root cause. Considering that in a microservice architecture, failures typically propagate from the callee (downstream) to the caller (upstream), but the symptoms (such as high latency) are often more pronounced upstream, this embodiment designs a walk algorithm that reverses the service call direction to find the source of the anomaly.

[0168] Constructing the transition matrix: Based on the adjacency matrix of the topological graph, construct a column-normalized transition matrix. Among them, elements Indicates from node Transfer to node The probability of this is proportional to the frequency of service calls.

[0169] Iterative calculation: The module introduces the restart probability. (For example This parameter determines the extent to which the walk process reverts to the observed initial anomalous node, thus balancing the weights of direct causes and underlying causes. The module uses the following iterative formula to calculate the steady-state probability distribution of each node:

[0170]

[0171] in: For the first The probability distribution vector after the nth iteration; at the initial time ( ),set up .

[0172] The transpose of the transition matrix means that the walk is performed in the opposite direction of the edges (i.e., from the callee to the caller).

[0173] The system continues to execute the above iterations until the norm of the difference between the results of two consecutive iterations is reached. Less than the preset convergence threshold (like At this point, the vector Steady state is reached. Module retrieval. The node corresponding to the element with the largest value is identified as the target root cause node and output to the operations and maintenance personnel.

[0174] For example, suppose service A calls service B, service B calls service C, and monitoring shows that service A has an extremely high response time (large contribution), service B has a slight delay (small contribution), and service C is normal.

[0175] Initial energy: mainly concentrated in A (symptom point).

[0176] Reverse flow: Since A depends on B and B depends on C, energy will flow in reverse from A to B, and then from B to C.

[0177] If, after iteration, it is found that energy is highly concentrated in B or C (even though C itself does not trigger an alarm, it may be a hidden root cause), then it can effectively point out that C or B is the culprit that caused A's collapse, thereby achieving in-depth localization by inferring the cause from the effect.

[0178] In summary, the system first utilizes the recursive covariance matrix to capture subtle coupling changes between high-dimensional indicators in real time, addressing the pain point that static thresholds cannot detect logical deviations. Then, it introduces spatial entropy as a sensor to drive the principal component boundaries of the monitoring model to adaptively shrink or expand, achieving both keen detection of unknown fault modes and robust filtering of normal business fluctuations without manual intervention. Finally, it maps the statistical residual energy to fault potential energy in the physical topology, and through a reverse walk algorithm, penetrates the complex calls between microservices, achieving an automated transition from symptom alerts to root cause delimitation. This provides a defense solution for building observability in large-scale distributed systems.

[0179] Example 2:

[0180] like Figures 2-3 As shown, the fast response method for network fault location based on distributed log parsing includes the following steps:

[0181] The distributed log stream of the target system is obtained, and the distributed log stream is periodically sliced ​​based on a preset sliding time window to generate a time-series data frame sequence arranged in chronological order.

[0182] For the current time-series data frame in the time-series data frame sequence, multiple monitoring indicators are extracted and standardized to construct a standardized state vector for the current moment containing multiple feature dimensions.

[0183] Obtain the reference covariance matrix at the current time. The reference covariance matrix can be a preset initial matrix or a historical covariance matrix corresponding to the previous time series data frame. Based on the preset forgetting factor, update the reference covariance matrix according to the standardized state vector to obtain the covariance matrix at the current time.

[0184] Obtain the principal component dimension parameters at the current time step. The principal component dimension parameters are either preset initial values ​​or updated values ​​based on feedback from the previous time series data frame. Based on the principal component dimension parameters, perform feature space decomposition on the covariance matrix at the current time step and construct the residual projection matrix.

[0185] The standardized state vector is projected onto the subspace defined by the residual projection matrix to obtain the residual vector and its corresponding anomaly energy value;

[0186] When the abnormal energy value meets the preset abnormal triggering conditions, the contribution of each feature dimension is calculated based on the residual vector, and the set of fault candidate indicators is determined according to the contribution.

[0187] Calculate the spatial entropy value of the residual vector in each feature dimension, and determine the principal component dimension parameters to be used to process the next time series data frame based on the spatial entropy value.

[0188] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.

[0189] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0190] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A network fault location and rapid response system based on distributed log parsing, characterized in that, include: The data acquisition module is used to acquire the distributed log stream of the target system, and perform periodic slicing of the distributed log stream based on a preset sliding time window to generate a time-series data frame sequence arranged in chronological order. The state vector construction module is used to extract multiple monitoring indicators and perform standardization processing on the current time-series data frame in the time-series data frame sequence, and construct a standardized state vector for the current moment containing multiple feature dimensions. The covariance matrix update module is used to obtain the reference covariance matrix at the current time. The reference covariance matrix is ​​either a preset initial matrix or a historical covariance matrix corresponding to the previous time series data frame. Based on the normalized state vector, the reference covariance matrix is ​​updated according to the preset forgetting factor to obtain the covariance matrix at the current time. The feature space decomposition module is used to obtain the principal component dimension parameters at the current time step. The principal component dimension parameters are either preset initial values ​​or updated values ​​based on feedback from the previous time-series data frame. Based on the principal component dimension parameters, the covariance matrix at the current time step is decomposed into feature space to construct the residual projection matrix. An anomaly detection module is used to project the standardized state vector onto the subspace defined by the residual projection matrix to obtain the residual vector and its corresponding anomaly energy value; when the anomaly energy value meets the preset anomaly triggering condition, the contribution of each feature dimension is calculated based on the residual vector, and a set of fault candidate indicators is determined according to the contribution. The process of determining the set of candidate fault indicators based on the contribution degree includes: sorting the monitoring indicators corresponding to each feature dimension based on the contribution degree, selecting monitoring indicators whose contribution degree meets the preset screening conditions, and generating an initial candidate indicator list for the current time; obtaining a candidate indicator time-series buffer pool, which is used to store the initial candidate indicator list corresponding to the most recent preset number of historical time-series data frames; traversing each monitoring indicator in the initial candidate indicator list for the current time, counting the frequency of a specific monitoring indicator appearing in the candidate indicator time-series buffer pool, and obtaining the time-series persistence characteristic value of the monitoring indicator; if the time-series persistence characteristic value is greater than or equal to a preset persistence threshold, the monitoring indicator is confirmed as a valid candidate indicator and included in the final set of candidate fault indicators; if the time-series persistence characteristic value is less than the persistence threshold, the monitoring indicator is marked as transient noise and removed from the initial candidate indicator list for the current time. The parameter adaptive module is used to calculate the spatial entropy value of the residual vector in each feature dimension, and determine the principal component dimension parameters for processing the next time series data frame based on the spatial entropy value. The calculation process of the spatial entropy value includes: calculating the ratio of the component energy of the residual vector to the abnormal energy value in each feature dimension to obtain the normalized probability of each feature dimension; based on the normalized probability of each feature dimension, calculating the spatial entropy value of the current time-series data frame according to the Shannon entropy formula; the formula for calculating the spatial entropy value is: in, This represents the spatial entropy value of the current time-series data frame. This represents the total number of feature dimensions. Indicates the first The normalized probability of each feature dimension is calculated as follows: , For the residual vector at the th The numerical values ​​of each feature dimension, This is an abnormal energy value; Determining the principal component dimension parameters for processing the next time-series data frame based on the spatial entropy value includes: obtaining the spatial entropy values ​​of the most recent preset number of time-series data frames, including the current time-series data frame; smoothing the spatial entropy values ​​to obtain a smoothed entropy value; if the smoothed entropy value is less than a preset entropy threshold, then the principal component dimension parameters for processing the next time-series data frame are determined as: the principal component dimension parameters at the current time minus a preset step size; if the smoothed entropy value is not less than the entropy threshold, then the principal component dimension parameters for processing the next time-series data frame are determined as: keeping the principal component dimension parameters at the current time unchanged, or increasing the principal component dimension parameters according to a preset rule.

2. The network fault location and rapid response system based on distributed log parsing according to claim 1, characterized in that: The calculation process of the covariance matrix at the current moment includes: Obtain a preset forgetting factor, wherein the forgetting factor has a value range between zero and one; Based on the reference covariance matrix, the standardized state vector at the current time, and the forgetting factor, the covariance matrix at the current time is calculated according to the recursive update formula; the recursive update formula is: in, Let the covariance matrix at the current time be denoted as . Represents the reference covariance matrix. This represents the standardized state vector at the current moment. Represents the transpose of the normalized state vector. This represents the forgetting factor.

3. The network fault location and rapid response system based on distributed log parsing according to claim 1, characterized in that: The process of constructing the residual projection matrix includes: Perform eigenvalue decomposition on the covariance matrix at the current time to obtain an eigenvalue sequence arranged in descending order of values ​​and a set of eigenvectors that correspond one-to-one with the eigenvalue sequence; Select the top-ranked feature vectors from the feature vector set, and select an amount equal to the principal component dimension parameter. The selected feature vectors form the principal subspace matrix. The residual projection matrix is ​​constructed based on the difference between the identity matrix and the projection matrix of the principal subspace matrix; the formula for constructing the residual projection matrix is: in, Represents the residual projection matrix. Represents the identity matrix. This represents the principal subspace matrix composed of the selected eigenvectors. This represents the transpose of the principal subspace matrix.

4. The network fault location and rapid response system based on distributed log parsing according to claim 1, characterized in that: Projecting the standardized state vector onto the subspace defined by the residual projection matrix yields the residual vector and its corresponding anomaly energy value, including: The residual vector is calculated using the projection formula based on the residual projection matrix and the standardized state vector at the current time. The anomalous energy value is obtained by calculating the square of the magnitude of the residual vector.

5. The network fault location and rapid response system based on distributed log parsing according to claim 1, characterized in that: After determining the set of candidate fault indicators, the following is also included: Obtain the service dependency topology graph of the target system, which includes nodes representing services and directed edges representing call relationships; Establish a mapping relationship between the monitoring indicators in the set of candidate fault indicators and the nodes in the service dependency topology graph; The contribution of each monitoring indicator in the set of candidate fault indicators is accumulated and assigned to its mapped node as the initial fault energy, generating a fault propagation map with energy state. A random walk is performed on the fault propagation map, and the probability distribution of each node is updated by iterative calculation until the preset convergence condition is met, so as to obtain the steady-state probability distribution. The node with the highest steady-state distribution probability is determined as the target root cause node.

6. The network fault location and rapid response system based on distributed log parsing according to claim 5, characterized in that: The probability distribution of each node is updated iteratively until a preset convergence condition is met, including: The direction of the random walk is defined as along the direction of the service call; Obtain the preset transition matrix and restart probability, and determine the normalized vector corresponding to the initial fault energy as the initial probability vector; Calculate the first using the iterative formula The probability vector after the nth iteration, up to the nth iteration The probability vector after the nth iteration and the nth iteration The difference between the probability vectors after each iteration is less than a preset convergence threshold; the iteration formula is: in, Indicates the first The probability vector after the nth iteration. Indicates the first The probability vector after the nth iteration, when The initial probability vector is given by time. Indicates the probability of restarting. Describes the transpose of the transition matrix. This represents a normalized vector consisting of the initial fault energy.

7. A fast response method for network fault location based on distributed log parsing, characterized in that, Includes the following steps: The distributed log stream of the target system is acquired, and the distributed log stream is periodically sliced ​​based on a preset sliding time window to generate a time-series data frame sequence arranged in chronological order. For the current time-series data frame in the time-series data frame sequence, multiple monitoring indicators are extracted and standardized to construct a standardized state vector for the current moment containing multiple feature dimensions. Obtain the reference covariance matrix at the current time, which is either a preset initial matrix or a historical covariance matrix corresponding to the previous time-series data frame; update the reference covariance matrix based on the standardized state vector according to a preset forgetting factor to obtain the covariance matrix at the current time. Obtain the principal component dimension parameters at the current time step. The principal component dimension parameters are either preset initial values ​​or updated values ​​based on feedback from the previous time-series data frame. Based on the principal component dimension parameters, perform feature space decomposition on the covariance matrix at the current time step to construct the residual projection matrix. The standardized state vector is projected onto the subspace defined by the residual projection matrix to obtain the residual vector and its corresponding anomaly energy value; When the abnormal energy value meets the preset abnormal triggering condition, the contribution of each feature dimension is calculated based on the residual vector, and the set of fault candidate indicators is determined according to the contribution. The process of determining the set of candidate fault indicators based on the contribution degree includes: sorting the monitoring indicators corresponding to each feature dimension based on the contribution degree, selecting monitoring indicators whose contribution degree meets the preset screening conditions, and generating an initial candidate indicator list for the current time; obtaining a candidate indicator time-series buffer pool, which is used to store the initial candidate indicator list corresponding to the most recent preset number of historical time-series data frames; traversing each monitoring indicator in the initial candidate indicator list for the current time, counting the frequency of a specific monitoring indicator appearing in the candidate indicator time-series buffer pool, and obtaining the time-series persistence characteristic value of the monitoring indicator; if the time-series persistence characteristic value is greater than or equal to a preset persistence threshold, the monitoring indicator is confirmed as a valid candidate indicator and included in the final set of candidate fault indicators; if the time-series persistence characteristic value is less than the persistence threshold, the monitoring indicator is marked as transient noise and removed from the initial candidate indicator list for the current time. Calculate the spatial entropy value of the residual vector in each feature dimension, and determine the principal component dimension parameters for processing the next time-series data frame based on the spatial entropy value; The calculation process of the spatial entropy value includes: calculating the ratio of the component energy of the residual vector to the abnormal energy value in each feature dimension to obtain the normalized probability of each feature dimension; based on the normalized probability of each feature dimension, calculating the spatial entropy value of the current time-series data frame according to the Shannon entropy formula; the formula for calculating the spatial entropy value is: in, This represents the spatial entropy value of the current time-series data frame. This represents the total number of feature dimensions. Indicates the first The normalized probability of each feature dimension is calculated as follows: , For the residual vector at the th The numerical values ​​of each feature dimension, This is an abnormal energy value; Determining the principal component dimension parameters for processing the next time-series data frame based on the spatial entropy value includes: obtaining the spatial entropy values ​​of the most recent preset number of time-series data frames, including the current time-series data frame; smoothing the spatial entropy values ​​to obtain a smoothed entropy value; if the smoothed entropy value is less than a preset entropy threshold, then the principal component dimension parameters for processing the next time-series data frame are determined as: the principal component dimension parameters at the current time minus a preset step size; if the smoothed entropy value is not less than the entropy threshold, then the principal component dimension parameters for processing the next time-series data frame are determined as: keeping the principal component dimension parameters at the current time unchanged, or increasing the principal component dimension parameters according to a preset rule.

Images