System anomaly intelligent diagnosis and recovery method and system fusing time sequence logs

Abstract

The application provides a system anomaly intelligent diagnosis and recovery method and system for fusing time sequence logs, relates to the distributed system operation and maintenance field, and comprises the following steps: collecting running logs from a plurality of service nodes of a distributed system and performing time sequence alignment, and extracting calling relationship and performance measurement data; calculating time sequence change characteristics in a sliding time window, generating a dynamic feature vector through fusion coding, and constructing a cross-service time sequence correlation graph; identifying abnormal service nodes and matching them with historical fault data to obtain historical propagation paths and feature vectors; calculating a propagation convergence coefficient and a feature deviation degree based on the historical propagation paths to obtain a cause-effect strength score; and identifying the fault influence degree according to the score and adaptively adjusting resource isolation and traffic scheduling strategies. The application realizes accurate diagnosis and efficient recovery of distributed system faults.

Description

TECHNICAL FIELD

[0001] The present application relates to distributed system operation and maintenance technology, and in particular to a system anomaly intelligent diagnosis and recovery method and system fusing time sequence logs. BACKGROUND

[0002] With the wide application of distributed systems, intelligent fault diagnosis and recovery has become a research hotspot. Existing fault diagnosis methods mainly perform anomaly detection based on performance indicators of a single service node, which is difficult to effectively identify fault propagation paths and root cause positioning in distributed systems.

[0003] Current distributed system fault diagnosis methods usually process the running logs and performance indicators of service nodes separately, lacking fusion analysis of multi-source heterogeneous data. This makes it difficult to fully grasp the running state of the system and accurately identify the propagation characteristics and impact range of faults.

[0004] Existing technologies also lack in-depth analysis of the calling relationship between service nodes, making it difficult to accurately grasp the propagation rules of faults in the system, and failing to fully utilize historical fault data for experience learning, resulting in poor diagnosis effect for similar fault scenarios; lacking self-adaptive ability in resource isolation and traffic scheduling, making it difficult to achieve precise fault control and system recovery.

[0005] With the continuous expansion of the scale of distributed systems and the continuous improvement of business complexity, traditional fault diagnosis and recovery methods have been unable to meet the actual needs. An intelligent solution that can fuse time sequence log data, identify fault propagation paths, and adaptively adjust recovery strategies is urgently needed. SUMMARY

[0006] The embodiments of the present application provide a system anomaly intelligent diagnosis and recovery method and system fusing time sequence logs, which can solve the problems in the prior art.

[0007] In a first aspect, the embodiments of the present application provide a system anomaly intelligent diagnosis and recovery method fusing time sequence logs, comprising:

[0008] Collecting running logs from multiple service nodes of a distributed system and performing time sequence alignment according to timestamps, extracting calling relationships and performance measurement data from the time sequence aligned running logs;

[0009] Calculating the time sequence variation characteristics of the performance measurement data within a sliding time window, fusing and encoding the time sequence variation characteristics and the calling relationships to generate dynamic feature vectors corresponding to the service nodes, and constructing a cross-service time sequence correlation graph based on the calling relationships and the dynamic feature vectors;

[0010] identify an abnormal service node from the cross-service timing association graph, match the dynamic feature vector corresponding to the abnormal service node with the historical fault records in the historical fault database, obtain the historical propagation path and the historical feature vector in the historical fault records;

[0011] Calculate the propagation convergence coefficient of each service node in the cross-service timing association graph based on the historical propagation path, calculate the feature deviation degree of the dynamic feature vector and the historical feature vector of each service node, and fuse the propagation convergence coefficient and the feature deviation degree to obtain the causal strength score of each service node;

[0012] According to the causal strength score, identify the fault influence degree, and adaptively adjust the resource isolation strategy and the traffic scheduling strategy of the service node based on the fault influence degree, to realize intelligent diagnosis and recovery of system fault.

[0013] Collect running logs from multiple service nodes of a distributed system and time-align them according to timestamps, extract call relationship and performance measurement data from the time-aligned running logs, including:

[0014] Collect original running logs from multiple service nodes, globally synchronize and calibrate the original running logs based on timestamps, sort the calibrated original running logs according to the chronological order of timestamps to construct time-aligned running logs;

[0015] Extract request tracking identifiers from the time-aligned running logs, associate log records spanning multiple service nodes as complete call links according to the request tracking identifiers, and construct weighted call relationships based on the call order and call frequency of the service nodes in the call links;

[0016] Extract performance indicator data of each service node from the time-aligned running logs, time-slice group the performance indicator data according to timestamps, calculate performance indicator statistical feature values within each time slice, and time-align and match the performance indicator statistical feature values with the call relationship to generate performance measurement data.

[0017] Calculate the time sequence change characteristics of the performance measurement data in the sliding time window, fuse and encode the time sequence change characteristics and the call relationship, generate the dynamic feature vector corresponding to the service node, and construct the cross-service timing association graph based on the call relationship including:

[0018] Set a sliding time window to time-segment the performance measurement data, construct a time-phase space of the performance measurement data in each sliding time window, calculate the trajectory curvature and attractor features of the performance measurement data in the time-phase space as the time sequence change characteristics;

[0019] Identify the hierarchical depth and branch density of the service node in the call link from the call relationship, and calculate the attenuation coefficient of the call propagation path based on the hierarchical depth and branch density;

[0020] The timing change characteristics of the upstream service node in the call relationship are weighted and aggregated according to the call attenuation coefficient, and the timing change characteristics and the weighted and aggregated upstream timing characteristics are cross-hierarchical fusion encoded through a multi-head attention mechanism to generate a dynamic feature vector corresponding to the service node.

[0021] Take the service node as the graph node and assign the dynamic feature vector, calculate the cosine similarity between the dynamic feature vectors of adjacent service nodes in the call relationship, take the cosine similarity as the weight value of the graph edge, and construct a cross-service timing association graph.

[0022] Identify the abnormal service node from the cross-service timing association graph, match the dynamic feature vector corresponding to the abnormal service node with the historical fault record in the historical fault database, and obtain the historical propagation path and historical feature vector in the historical fault record, including:

[0023] Calculate the dynamic feature vector difference matrix of adjacent time windows in the cross-service timing association graph, perform decomposition on the dynamic feature vector difference matrix to obtain the timing fluctuation component and the topological propagation component, and construct the state transition sequence based on the timing fluctuation component and the topological propagation component.

[0024] Calculate the entropy value of the timing fluctuation component matrix and the topological propagation component matrix in the state transition sequence, generate an anomaly score by combining the entropy values of the timing fluctuation component and the topological propagation component, identify the abnormal service node based on the anomaly score, and extract the abnormal propagation link centered on the abnormal service node from the state transition sequence.

[0025] Extract the timing fluctuation frequency spectrum component from the dynamic feature vector of the abnormal service node, extract the propagation delay spectrum component from the abnormal propagation link, combine the timing fluctuation frequency spectrum component and the propagation delay spectrum component to construct a dual-spectrum feature, and retrieve a matching historical fault record in the historical fault database according to the dual-spectrum feature. Based on the historical fault record, obtain the historical propagation path and the historical feature vector.

[0026] Based on the historical propagation path, calculate the propagation convergence coefficient of each service node in the cross-service timing association graph, calculate the feature deviation degree of the dynamic feature vector and the historical feature vector of each service node, and fuse the propagation convergence coefficient and the feature deviation degree to obtain the causal strength score of each service node, including:

[0027] extracting a directed propagation edge between historical service nodes in a historical propagation path and a historical time delay distribution corresponding to the directed propagation edge, identifying a candidate propagation subgraph matching a topology structure of the historical propagation path in a cross-service time sequence correlation graph, and constructing an upstream propagation tree in a multi-hop backtracking direction of each target service node in the candidate propagation subgraph;

[0028] calculating a time delay evolution deviation between the historical time delay distribution of the directed propagation edge in the upstream propagation tree and an actual time delay distribution of a corresponding directed propagation edge in the cross-service time sequence correlation graph;

[0029] performing propagation attenuation weighting on the directed propagation edge in the upstream propagation tree based on the time delay evolution deviation and performing normalized aggregation, to obtain a propagation convergence coefficient of the target service node;

[0030] extracting a time sequence component and a topology component from a dynamic feature vector and a historical feature vector of each target service node, respectively, and calculating a time sequence deviation degree between the time sequence components and a topology deviation degree between the topology components;

[0031] performing weighted combination on the time sequence deviation degree and the topology deviation degree, to generate a feature deviation degree of each target service node;

[0032] calculating a topology centrality of the target service node in the cross-service time sequence correlation graph, and adaptively weighting and fusing the propagation convergence coefficient and the feature deviation degree based on the topology centrality, to obtain a causal strength score of each target service node.

[0033] identifying a fault influence degree based on the causal strength score, and adaptively adjusting a resource isolation strategy and a traffic scheduling strategy of the service node based on the fault influence degree, to realize intelligent diagnosis and recovery of system faults, including:

[0034] dividing the causal strength score into a plurality of score sequences according to a fixed time window, extracting a mean value change amount and a variance change amount from the score sequences, calculating a cumulative value of the mean value change amount to obtain a fault diffusion degree, calculating a cumulative value of the variance change amount to obtain a fault fluctuation degree, and taking a product of the fault diffusion degree and the fault fluctuation degree as the fault influence degree;

[0035] monitoring a resource usage change of the service node based on the fault influence degree, calculating a growth rate and a decay rate of the resource usage, marking a time period in which the growth rate exceeds a preset growth threshold as a resource isolation starting point, marking a time period in which the decay rate exceeds a preset decay threshold as a resource isolation termination point, and generating the resource isolation strategy of the service node according to the resource isolation starting point and the resource isolation termination point;

[0036] The resource isolation strategy is converted into a resource limit vector, a computing resource limit value and a storage resource limit value are extracted from the resource limit vector, the concurrent processing threshold of the service node is determined according to the computing resource limit value, the cache processing threshold of the service node is determined according to the storage resource limit value, the traffic scheduling strategy of the service node is updated according to the concurrent processing threshold and the cache processing threshold, and the intelligent diagnosis and recovery of the system failure are realized.

[0037] In a second aspect, the embodiment of the application provides a system abnormality intelligent diagnosis and recovery system fusing time sequence logs, comprising:

[0038] A log collection unit is configured to collect running logs from a plurality of service nodes of a distributed system and perform time sequence alignment according to timestamps, and extract calling relations and performance measurement data from the time sequence aligned running logs;

[0039] A feature extraction unit is configured to calculate time sequence variation features of the performance measurement data in a sliding time window, fuse and encode the time sequence variation features and the calling relations, generate a dynamic feature vector corresponding to the service node, and construct the dynamic feature vector into a cross-service time sequence correlation graph based on the calling relations;

[0040] A fault matching unit is configured to identify an abnormal service node from the cross-service time sequence correlation graph, match a dynamic feature vector corresponding to the abnormal service node with historical fault records in a historical fault database, and obtain a historical propagation path and a historical feature vector in the historical fault records;

[0041] A cause-effect analysis unit is configured to calculate a propagation convergence coefficient of each service node in the cross-service time sequence correlation graph based on the historical propagation path, calculate a feature deviation degree of the dynamic feature vector of each service node and the historical feature vector, and fuse the propagation convergence coefficient and the feature deviation degree to obtain a cause-effect intensity score of each service node;

[0042] A fault recovery unit is configured to identify a fault influence degree according to the cause-effect intensity score, adaptively adjust a resource isolation strategy and a traffic scheduling strategy of the service node based on the fault influence degree, and realize intelligent diagnosis and recovery of the system failure.

[0043] In a third aspect, the embodiment of the application provides an electronic device, comprising:

[0044] A processor;

[0045] A memory for storing processor executable instructions;

[0046] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0047] In a fourth aspect, the present application provides a computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method described above.

[0048] In the embodiment, the effective integration of multi-source heterogeneous data is realized through the time sequence alignment and fusion analysis of the distributed system operation log, and the accuracy of fault diagnosis is improved. Based on the sliding time window and the construction method of dynamic feature vector, the dynamic change of system state can be captured in time, and the real-time performance of abnormal detection is improved. Through the experience learning and matching of historical fault data, combined with the calculation of propagation convergence coefficient and feature deviation degree, the propagation path and root cause position of the fault are accurately identified. The adaptive recovery strategy of fusion causal strength score realizes the precise control of resource isolation and traffic scheduling, and improves the efficiency of fault recovery. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 A flowchart of the system anomaly intelligent diagnosis and recovery method of the fusion time sequence log according to the embodiment of the present application is shown in

[0050] Figure 2 A causal strength calculation flowchart of the multi-source feature fusion according to the embodiment of the present application is shown in DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0052] The technical scheme of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0053] Figure 1 A flowchart of the system anomaly intelligent diagnosis and recovery method of the fusion time sequence log according to the embodiment of the present application is shown in Figure 1 As shown in the figure, the method comprises:

[0054] Collecting operation logs from multiple service nodes of a distributed system and performing time sequence alignment according to timestamps, and extracting call relationship and performance metric data from the time sequence aligned operation logs;

[0055] The time sequence change feature of the performance metric data is calculated in the sliding time window, the time sequence change feature is fused and encoded with the calling relationship, a dynamic feature vector corresponding to the service node is generated, and the dynamic feature vector is constructed into a cross-service time sequence association graph based on the calling relationship;

[0056] An abnormal service node is identified from the cross-service time sequence association graph, a dynamic feature vector corresponding to the abnormal service node is matched with a historical fault record in a historical fault database, and a historical propagation path and a historical feature vector in the historical fault record are obtained;

[0057] A propagation convergence coefficient of each service node in the cross-service time sequence association graph is calculated based on the historical propagation path, a feature deviation degree of the dynamic feature vector of each service node and the historical feature vector is calculated, and a causal strength score of each service node is obtained by fusing the propagation convergence coefficient and the feature deviation degree;

[0058] A fault influence degree is identified according to the causal strength score, and a resource isolation strategy and a traffic scheduling strategy of the service node are adaptively adjusted based on the fault influence degree, so as to realize intelligent diagnosis and recovery of system faults.

[0059] Running logs are collected from a plurality of service nodes of a distributed system and time sequence alignment is performed according to timestamps, and a calling relationship and performance metric data are extracted from the time sequence aligned running logs, including:

[0060] Raw running logs are collected from the plurality of service nodes, global clock synchronization calibration is performed on the raw running logs based on the timestamps, and the calibrated raw running logs are sorted according to the chronological order of the timestamps to construct time sequence aligned running logs;

[0061] Request tracking identifiers are extracted from the time sequence aligned running logs, log records spanning multiple service nodes are associated as complete calling links according to the request tracking identifiers, and a weighted calling relationship is constructed based on the calling order and calling frequency of the service nodes in the calling links;

[0062] Performance indicator data of each service node is extracted from the time sequence aligned running logs, the performance indicator data is time sequence sliced and grouped according to the timestamps, performance indicator statistical feature values in each time sequence slice are calculated, and the performance indicator statistical feature values are time sequence aligned and matched with the calling relationship to generate performance metric data.

[0063] The raw running log collection stage realizes acquisition of running logs of the plurality of service nodes by deploying a log collection component. The log collection component can be configured on each service node and supports real-time capture of log records generated by the service node. The collected raw logs include timestamps, request identifiers, service names, interface names, execution status codes, performance indicators and other information. After collection is completed, the log collection component transmits the raw logs to a central log processing module for subsequent processing.

[0064] The collected original operation logs are globally clock-synchronized and calibrated. Since the system clocks of different service nodes may be biased, leading to inconsistent log timestamps, calibration is needed. The calibration method is based on the clock synchronization algorithm of the network time protocol, and a reference clock service is selected as the standard time source to calculate the time deviation of each service node from the standard time source. For a log record generated by a service node, the corresponding time deviation value is added to the timestamp to obtain the calibrated unified timestamp. For example, if the time deviation of service node A is +2 seconds, the log record with the timestamp "2025-11-27 10:15:30" recorded on this node is updated to "2025-11-27 10:15:32" after calibration.

[0065] Time sequence alignment is to sort the calibrated original operation logs according to the chronological order of the timestamps. A quicksort algorithm is used to sort all log records, and the sorting key value is the calibrated timestamp. After sorting, a globally unified operation log sequence is obtained in the order of actual occurrence time, which is called the time sequence aligned operation log.

[0066] The call link construction process is implemented based on the request tracking identification technology. The request tracking identification is extracted from the time sequence aligned operation log, which is generated when the request enters the system and is passed throughout the request processing process, and is used to identify the processing process of the same request on different service nodes. The request tracking identification is extracted from the log record by regular expression pattern matching. Log records with the same request tracking identification are grouped, and these log records constitute a complete call link. Each call link contains the complete processing process of the request from the entrance to the exit, and records all the service nodes and their processing states passed by the request.

[0067] The weighted call relationship construction is implemented based on call link analysis. The call sequence of the service nodes in each call link is analyzed to identify the call relationship between the service nodes. For each identified call relationship, the call direction and call frequency are recorded. The call frequency is used as the weight value, indicating the closeness of the call relationship. After analyzing all call links, the call relationships and their weight values are merged to construct the weighted call relationship graph of the entire system. The nodes of the graph represent the service nodes, the edges represent the call relationships, and the weights of the edges represent the call frequencies. For example, if the frequency of service A calling service B is 100 times, the call relationship weight from A to B is 100.

[0068] Performance indicator extraction is to extract the performance data of each service node from the time-aligned operation logs. The extracted performance indicators include response time, throughput, error rate, resource occupancy rate, etc. For the response time indicator, the difference between the start time and the end time of request processing is calculated; for the throughput indicator, the number of processed requests per unit time is counted; for the error rate indicator, the ratio of error requests to total requests is obtained; for the resource occupancy rate indicator, resource monitoring related log records are parsed.

[0069] Time slice grouping is to group the extracted performance indicator data according to the time dimension. A fixed time window size, such as 5 minutes, is set, and the performance indicator data is grouped according to the time window to obtain a series of time slices. Each time slice contains the performance indicator data of each service node in the time window.

[0070] Performance indicator statistical feature value calculation is to perform statistical analysis on the performance indicator data in each time slice. The calculation methods include calculating the average value, median, maximum value, minimum value, standard deviation, percentile, etc. of the performance indicator data. These statistical feature values comprehensively reflect the performance status of each service node in a specific time window.

[0071] Performance metric data generation is achieved by time-aligning and matching the performance indicator statistical feature values with the call relationship. For each time slice, the performance indicator statistical feature values in the time window are associated with the corresponding call relationship to generate performance metric data containing time, service node, performance indicator statistical feature value and call relationship. These performance metric data comprehensively reflect the running state and performance of the system in each time window.

[0072] The present application realizes comprehensive monitoring and abnormal diagnosis of distributed system running state by fusing the time sequence logs of multiple service nodes. It overcomes the limitations of traditional log analysis methods in multi-service environment, realizes time sequence alignment of cross-service node logs and accurate construction of call relationship. Through time slice and performance indicator statistical feature analysis, performance abnormalities and service call abnormalities in the system can be found in time, and root cause positioning can be performed combined with the call relationship graph, improving the accuracy and efficiency of abnormal diagnosis. The time sequence characteristics of performance metric data help to find the performance trend changes and potential risks of the system, realize preventive maintenance, and improve the reliability and stability of the system.

[0073] In the sliding time window, the time sequence change characteristics of the performance metric data are calculated, the time sequence change characteristics are fused and encoded with the call relationship, and the dynamic feature vector corresponding to the service node is generated. The dynamic feature vector is constructed into a cross-service time sequence correlation graph based on the call relationship, which includes:

[0074] The performance metric data is time-sequentially segmented by setting a sliding time window, a time-sequential phase space of the performance metric data is constructed in each sliding time window, and a trajectory curvature and attractor feature of the performance metric data are calculated in the time-sequential phase space as time-sequential change features;

[0075] The hierarchical depth and branch density of the service node in the call link are identified from the call relationship, and an attenuation coefficient of the call propagation path is calculated based on the hierarchical depth and the branch density;

[0076] The time-sequential change features of the upstream service nodes in the call relationship are weighted and aggregated according to the call attenuation coefficient, the time-sequential change features and the weighted and aggregated upstream time-sequential features are cross-hierarchically fused and encoded through a multi-head attention mechanism, and a dynamic feature vector corresponding to the service node is generated;

[0077] The service node is taken as a graph node and is given a dynamic feature vector, a cosine similarity between the dynamic feature vectors of adjacent service nodes in the call relationship is calculated, the cosine similarity is taken as a weight value of a graph edge, and a cross-service time-sequential association graph is constructed.

[0078] The time-sequential phase space of the performance metric data is constructed by a delay embedding method. For the performance metric data of each service node, a suitable embedding dimension and delay parameter are selected to convert a one-dimensional time series into a trajectory in a multi-dimensional phase space. The embedding dimension is usually selected to be 3 to 5 dimensions, and the delay parameter is determined according to the autocorrelation characteristics of the time-sequential data, and is generally 1 to 3 times the sampling interval. For the response time index, assuming that the data sequence collected in a certain sliding window is [120ms, 135ms, 125ms, 180ms, 210ms], through the delay embedding method, the embedding dimension is selected to be 3, and the delay parameter is 1, the sequence can be converted into a point set in the phase space: (120ms, 135ms, 125ms), (135ms, 125ms, 180ms), (125ms, 180ms, 210ms). These points form a trajectory representing the dynamic behavior of the system in the phase space.

[0079] The trajectory curvature of the performance metric data in the time-sequential phase space is calculated by analyzing the change rate between adjacent points. In the constructed phase space, the angle change between adjacent points on the trajectory is calculated to obtain the local curvature of the trajectory. The greater the curvature value, the more intense the change of the performance index, which may indicate the abnormality of the system behavior. The convergence region of the phase space trajectory is analyzed to identify the attractor feature. The attractor feature includes the type, size and stability of the attractor, which is used to describe the long-term behavior pattern of the system. Under normal operating conditions, the system performance index usually forms a stable point attractor or a periodic attractor; while under abnormal conditions, a chaotic attractor or trajectory divergence phenomenon may occur. The trajectory curvature and the attractor feature are combined to form the time-sequential change feature describing the dynamic change of the performance of the service node.

[0080] Further, the service node is identified in the hierarchical depth and branch density in the call link, the hierarchical depth refers to the position depth of the service node in the call link, and the hierarchical depth is calculated from the entry service node, and the hierarchical depth is increased by one after each service call. The branch density indicates the extent of the service node calling downstream services, and is calculated as the number of different service nodes directly called by the node divided by the total number of service nodes. The hierarchical depth and branch density of the service node together reflect the structural importance of the node in the entire call network.

[0081] The attenuation coefficient calculation of the call propagation path is based on the hierarchical depth and branch density. The attenuation coefficient is used to quantify the influence strength between the service nodes in the call relationship, considering the information loss and influence attenuation in the service call process. The attenuation coefficient value is inversely proportional to the hierarchical depth and proportional to the branch density. In specific calculation, the hierarchical depth is standardized to the [0, 1] interval, and the branch density is standardized to the [0, 1] interval. The standardized hierarchical depth is converted into a basic attenuation value using an exponential function, and the basic attenuation value is adjusted according to the standardized branch density to obtain the final call attenuation coefficient.

[0082] The weighted aggregation of the timing change characteristics of the upstream service nodes is a feature fusion process using the call relationship. For each service node, identify all upstream calling nodes, obtain the timing change characteristics of these upstream nodes, and weight according to the call attenuation coefficient of each upstream node to the current node to obtain the weighted aggregated upstream timing characteristics. The weighting method uses the dot product operation of the call attenuation coefficient and the timing change characteristics. When a service node has multiple upstream calling nodes, sum all the weighted upstream timing characteristics to obtain the final weighted aggregation result.

[0083] The cross-level fusion encoding of the multi-head attention mechanism is a process of deep fusion of the service node's own timing change characteristics and the weighted aggregated upstream timing characteristics. The multi-head attention mechanism first projects the input features into query, key and value spaces, then calculates attention in different representation subspaces in parallel, and finally concatenates and linearly transforms the attention results of each subspace to obtain the final output. Through the multi-head attention mechanism, the important association between the service node's own characteristics and the upstream characteristics can be adaptively learned, and cross-level feature fusion can be realized. The fused features are processed through a fully connected layer and a nonlinear activation function to generate a feature vector representing the dynamic behavior of the service node, i.e., a dynamic feature vector.

[0084] The construction of the cross-service timing correlation graph is a process of organizing service nodes and the calling relationship therebetween into a graph structure. The service nodes are taken as nodes of the graph, and the dynamic feature vector generated in the previous step is taken as the node feature of each node. For each pair of adjacent service nodes in the calling relationship, the cosine similarity between the dynamic feature vectors thereof is calculated as the weight value of the edge connecting the two nodes. The cosine similarity reflects the similarity degree of the performance change patterns of the two service nodes, and the higher the similarity is, the more collaborative the performance changes of the two nodes are. In this way, a weighted undirected graph, i.e., the cross-service timing correlation graph, is constructed, which not only contains the service calling structure information, but also fuses the performance dynamic change features of the nodes and the performance collaborative relationship therebetween.

[0085] The present application realizes accurate characterization of the dynamic characteristics of system performance by using the sliding time window and phase space analysis technology, and breaks through the limitations of traditional static feature analysis. The cross-service feature fusion mechanism based on the calling relationship solves the problem of difficult tracking of abnormal propagation paths, and improves the accuracy of root cause positioning. The multi-head attention mechanism realizes adaptive fusion of features between different service levels, and enhances the ability to identify abnormal patterns in complex calling links. The cross-service timing correlation graph fuses the service performance change and topology structure information, provides a unified analysis framework for abnormal diagnosis, significantly improves the intelligent level of distributed system abnormal diagnosis, and provides a reliable basis for subsequent recovery strategy formulation.

[0086] From the cross-service timing correlation graph, the dynamic feature vector corresponding to the abnormal service node is matched with the historical fault records in the historical fault database to obtain the historical propagation path and the historical feature vector in the historical fault records, including:

[0087] The dynamic feature vector difference matrix of adjacent time windows in the cross-service timing correlation graph is calculated, the dynamic feature vector difference matrix is decomposed to obtain the timing fluctuation component and the topology propagation component, and the state transition sequence is constructed based on the timing fluctuation component and the topology propagation component.

[0088] The entropy values of the timing fluctuation component matrix and the topology propagation component matrix in the state transition sequence are calculated, the abnormal score is generated by combining the timing fluctuation component entropy value and the topology propagation component entropy value, the abnormal service node is identified based on the abnormal score, and the abnormal propagation link centered on the abnormal service node is extracted from the state transition sequence.

[0089] The timing fluctuation frequency spectrum component is extracted from the dynamic feature vector of the abnormal service node, the propagation delay spectrum component is extracted from the abnormal propagation link, the double-spectrum feature is constructed by combining the timing fluctuation frequency spectrum component and the propagation delay spectrum component, the matching historical fault record is searched in the historical fault database according to the double-spectrum feature, and the historical propagation path and the historical feature vector are obtained based on the historical fault record.

[0090] The dynamic feature vectors of the same service node in two adjacent time windows are subjected to element-by-element subtraction to obtain a difference vector describing the performance change trend of the service node. This operation is performed on all service nodes, and all difference vectors are arranged in order of service node identification to form a dynamic feature vector difference matrix. For example, if the dynamic feature vector of a service node in time window T1 is [0.72, 0.35, 0.91, 0.48] and the dynamic feature vector of the service node in T2 is [0.75, 0.29, 1.25, 0.51], the calculated difference vector is [0.03, -0.06, 0.34, 0.03].

[0091] The dynamic feature vector difference matrix is subjected to decomposition to decompose it into a time series fluctuation component and a topology propagation component. The non-negative matrix decomposition technique is used, the latent dimension of decomposition is set to 2, and the decomposition result is solved by an iterative optimization algorithm. The time series fluctuation component describes the change pattern of the performance indicators of the service node itself, and the topology propagation component reflects the performance change propagation effect caused by the service call relationship. The multiplication update rule is used in the decomposition process to ensure the non-negativity of the result and minimize the decomposition error. The maximum number of iterations is set to 100 and the convergence threshold is set to 0.001 in the iteration process.

[0092] The time series fluctuation component and the topology propagation component obtained by decomposition are used to construct a state transition sequence. The decomposition results of consecutive multiple time windows are arranged in time sequence to form a transition sequence representing the change of system state over time. The sequence includes a time series fluctuation component matrix sequence and a topology propagation component matrix sequence, describing the spatio-temporal evolution process of the system state. For data of 12 consecutive 30-second sliding windows within 10 minutes, 11 sets of dynamic feature vector difference matrices and their decomposition results can be obtained to construct a state transition sequence containing 11 time points.

[0093] The entropy values of the time series fluctuation component matrix and the topology propagation component matrix in the state transition sequence are calculated. For the time series fluctuation component matrix, the Shannon entropy of the row vector corresponding to each service node is calculated to measure the uncertainty of the performance change of the node. For the topology propagation component matrix, the Shannon entropy of the row vector corresponding to each service node is calculated to measure the uncertainty of the node in the call relationship propagation. When calculating the entropy value, the row vector of the matrix is normalized so that the sum of the elements of the row vector is 1, and the weighted sum with logarithm is calculated for the normalized element value to obtain the entropy value. The higher the entropy value, the greater the uncertainty and the higher the abnormality possibility.

[0094] The combined time fluctuation component entropy value and the topology propagation component entropy value generate an anomaly score. The two entropy values of each service node are combined by weighted summation. The weight coefficients are adjusted according to the historical anomaly detection effect, and the time fluctuation component entropy value is usually weighted by 0.6, and the topology propagation component entropy value is usually weighted by 0.4. The anomaly score obtained after combination reflects the overall abnormality degree of the service node, and the score value is between 0 and 1, and the value closer to 1 indicates a greater abnormality possibility.

[0095] An abnormal service node is identified based on the anomaly score. The anomaly score threshold is set to 0.8, and the service node whose anomaly score exceeds the threshold is identified as an abnormal service node. If the anomaly scores of multiple nodes all exceed the threshold, the nodes are ranked from high to low according to the scores, and the nodes ranked at the top are selected as the main abnormal service nodes. If it is detected that the anomaly score of service node A is 0.92, the anomaly score of node B is 0.88, and the anomaly score of node C is 0.75, nodes A and B are identified as abnormal service nodes, and C is not considered abnormal.

[0096] An abnormal propagation link centered on the abnormal service node is extracted from the state transition sequence. The change trend of the topology propagation component in the state transition sequence is analyzed to identify the propagation path of the anomaly. The values between the abnormal service node and other nodes in the topology propagation component matrix are checked, and the larger the value, the stronger the propagation correlation. According to the propagation intensity, the service nodes with propagation intensity exceeding a set threshold are selected to form an abnormal propagation link. The threshold is usually set to 50% of the maximum value of the topology propagation component.

[0097] The time fluctuation frequency spectrum component is extracted from the dynamic feature vector of the abnormal service node. Fast Fourier transform is performed on the dynamic feature vector sequence of the abnormal service node in the state transition sequence to convert the time domain feature to the frequency domain feature. The amplitude part of the transform result is taken as the time fluctuation frequency spectrum component to describe the frequency characteristics of the performance change of the abnormal service node. Usually, only the spectrum information of the low to medium frequency part is retained, because the high frequency part often contains noise rather than effective signals.

[0098] The propagation delay spectrum component is extracted from the abnormal propagation link. Cross-correlation analysis is performed on the anomaly score time sequence of each service node in the abnormal propagation link to calculate the time delay characteristics of the abnormal propagation. The time delay distribution is subjected to Fourier transform to obtain the frequency spectrum representation of the propagation delay, i.e. the propagation delay spectrum component, which reflects the time mode characteristics of the abnormal propagation between service nodes.

[0099] The time fluctuation frequency spectrum component and the propagation delay spectrum component are combined to construct a dual-spectrum feature. The two normalized frequency spectrum components are spliced into a comprehensive feature vector by feature splicing. The feature combines the performance change characteristics and the abnormal propagation characteristics of the abnormal service node, effectively representing the current anomaly mode.

[0100] The double-frequency spectrum features are used to search for matching historical fault records in a historical fault database. Cosine similarity is used as a matching metric to calculate the similarity between the double-frequency spectrum features of the current anomaly and the double-frequency spectrum features of the historical fault records. The historical fault record with the highest similarity and exceeding a set threshold is selected as the matching result. If there are multiple records with similarity exceeding the threshold, the top several records with the highest similarity are returned. The threshold is usually set to 0.85 to ensure that the matching result has sufficient similarity.

[0101] Based on the matched historical fault record, a historical propagation path and a historical feature vector are obtained. The historical propagation path information, including the fault source node, the propagation order and range, etc., is extracted from the matching record. At the same time, the historical feature vector, including the dynamic feature change pattern of each service node in the abnormal development process, is extracted for subsequent fault root cause analysis and recovery strategy formulation.

[0102] The present application realizes accurate identification and positioning of abnormal service nodes through dynamic analysis of the cross-service time sequence correlation graph, and solves the technical problems of complex abnormal propagation and difficult root cause tracing in a distributed environment. The decomposition method based on time sequence fluctuation and topological propagation component decouples the time characteristics and spatial characteristics of the anomaly, improving the accuracy of anomaly detection. The double-frequency spectrum feature innovatively integrates performance fluctuation and propagation delay information, enhancing the ability to distinguish abnormal patterns. Through intelligent matching with historical faults, the reuse of similar fault experience is realized, greatly shortening the fault diagnosis time and reducing the professional threshold of operation and maintenance personnel, providing a reliable decision basis for automated fault recovery, and effectively improving system reliability and service continuity.

[0103] As shown in Figure 2 , a flow chart of causal strength calculation of multi-source feature fusion of the present embodiment is shown.

[0104] Based on the historical propagation path, the propagation convergence coefficient of each service node in the cross-service time sequence correlation graph is calculated, the feature deviation degree of the dynamic feature vector of each service node and the historical feature vector is calculated, and the causal strength score of each service node is obtained by fusing the propagation convergence coefficient and the feature deviation degree, including:

[0105] The directed propagation edges between historical service nodes and the historical delay distribution corresponding to the directed propagation edges are extracted from the historical propagation path, the candidate propagation subgraph matching the topological structure of the historical propagation path is identified in the cross-service time sequence correlation graph, and the upstream propagation tree is constructed by multi-hop backtracking of each target service node in the candidate propagation subgraph along the in-degree direction;

[0106] The delay evolution deviation between the historical delay distribution of the directed propagation edges in the upstream propagation tree and the actual delay distribution of the corresponding directed propagation edges in the cross-service time sequence correlation graph is calculated.

[0107] The propagation convergence coefficients of the target service nodes are obtained by performing propagation decay weighting and normalization aggregation on the directed propagation edges in the upstream propagation tree based on time delay evolution bias;

[0108] The time sequence components and the topology components are extracted from the dynamic feature vectors and the historical feature vectors of the target service nodes respectively, and the time sequence bias degrees between the time sequence components and the topology bias degrees between the topology components are calculated;

[0109] The time sequence bias degrees and the topology bias degrees are combined by weighting to generate the feature bias degrees of the target service nodes;

[0110] The topology centrality of the target service nodes in the cross-service time sequence correlation graph is calculated, and the propagation convergence coefficients and the feature bias degrees are adaptively weighted and fused according to the topology centrality to obtain the causal strength scores of the target service nodes.

[0111] The historical propagation paths are usually recorded in a fault database, and contain the propagation order and time information of the abnormality between the service nodes at the time of historical fault occurrence. For each historical propagation path, the service node pairs therein are extracted as directed propagation edges, and the abnormal propagation time differences between these service node pairs are extracted as historical time delay values. If there are multiple historical propagation records between two service nodes, the time delay values are summarized to obtain a time delay distribution. A typical historical time delay distribution may be the abnormal propagation time delay set [2.1 seconds, 2.3 seconds, 1.9 seconds, 2.2 seconds] between service node pairs A and B, indicating that the abnormality usually takes about 2 seconds to propagate from node A to node B.

[0112] In the cross-service time sequence correlation graph, candidate propagation subgraphs that match the topology structure of the historical propagation paths are identified. A subgraph matching algorithm is used to search for subgraphs with similar topology structures in the current cross-service time sequence correlation graph, taking the topology structure of the historical propagation path as a template. The matching metric includes node similarity and edge similarity, the node similarity is calculated based on the functional type and interface features of the service nodes, and the edge similarity is calculated based on the frequency and response time features of the service call relationship. The subgraphs with a matching degree exceeding a set threshold are marked as candidate propagation subgraphs. The subgraph matching algorithm uses a heuristic search strategy to start expanding from high-similarity nodes and gradually build matching subgraphs, and the algorithm complexity is controlled at a polynomial level of the number of service nodes.

[0113] An upstream propagation tree is constructed by multi-hop backtracking in the in-degree direction of each target service node in the candidate propagation subgraph. The target service node is usually an abnormal node or a key business node in the candidate propagation subgraph. For each target node, a breadth-first traversal is performed in the reverse direction of service call to construct a directed tree structure with the target node as the root and the upstream service nodes as branches. The traversal depth is usually limited to 3 to 5 hops to avoid constructing a too large propagation tree. The actual call relationship and delay between nodes are recorded during the construction process. If a target service node has multiple upstream service nodes, the upstream propagation tree will have a branch structure.

[0114] The delay evolution deviation between the historical delay distribution of the directed propagation edge in the upstream propagation tree and the actual delay distribution of the corresponding directed propagation edge in the cross-service timing correlation graph is calculated. For each directed edge in the upstream propagation tree, the actual delay distribution in the current cross-service timing correlation graph is extracted, usually based on the service call logs in the recent period of time. The chi-square distance is used to calculate the difference between the historical delay distribution and the actual delay distribution, and the delay evolution deviation is obtained. In the chi-square distance calculation, the two delay distributions are first divided into the same delay intervals, the frequencies in each interval are counted, and then the weighted sum of the frequency differences is calculated. The delay evolution deviation reflects the degree of change of the current service call relationship relative to the historical fault, and the greater the deviation, the greater the difference between the current propagation mode and the historical propagation mode.

[0115] Based on the delay evolution deviation, the directed propagation edges in the upstream propagation tree are weighted and normalized aggregated to obtain the propagation convergence coefficient of the target service node. The smaller the delay evolution deviation, the closer the current propagation mode to the historical fault mode, and the higher the propagation weight. An exponential decay function is used to convert the delay evolution deviation into the propagation weight, with the maximum weight being 1 when the deviation is 0, and the weight exponentially decaying as the deviation increases. For each path in the upstream propagation tree, the cumulative propagation weight is calculated step by step from the leaf node to the root node, considering the decay factor of the path length. Finally, the cumulative weights of all paths are normalized and aggregated to obtain the propagation convergence coefficient of the target service node, which reflects the possibility of the target node as a potential fault source.

[0116] The timing component and the topology component are extracted from the dynamic feature vector and the historical feature vector of each target service node, respectively. The dynamic feature vector contains performance indicators, log event statistics and other information of the service node, and the historical feature vector comes from the feature data in the historical fault record. The feature vector is decomposed into a timing principal component and a topology principal component by principal component analysis method, the timing principal component mainly reflects the time characteristics of the node's own performance change, and the topology principal component mainly reflects the structural characteristics of the node in the service call network. Usually, the first few principal components with the largest explained variance are selected as representative components.

[0117] The time deviation degree between the time sequence components and the topology deviation degree between the topology components are calculated. The cosine distance is used to calculate the similarity between the current time sequence component and the historical time sequence component of the target service node, and the similarity is converted into the time deviation degree. The topology deviation degree between the current topology component and the historical topology component is calculated by the same method. The deviation degree value range is between 0 and 1, and the value closer to 0 indicates that the current feature is more similar to the historical feature, and the closer to 1 indicates that the difference is greater.

[0118] The time deviation degree and the topology deviation degree are combined by weighting to generate the feature deviation degree of each target service node. The weighting coefficient is adjusted according to the specific application scene, and the time deviation degree weight is usually 0.7 and the topology deviation degree weight is 0.3. The comprehensive feature deviation degree is obtained by weighted summation of the two parts of the deviation degree, which reflects the overall difference degree between the current state and the historical fault state of the target node.

[0119] The topology centrality of the target service node in the cross-service time sequence association graph is calculated, the propagation convergence coefficient and the feature deviation degree are adaptively weighted and fused according to the topology centrality, and the causal strength score of each target service node is obtained. The topology centrality adopts the closeness centrality index, and the reciprocal of the average shortest path length of the node to all other nodes in the graph is calculated. The higher the centrality of the node, the greater the influence in the propagation process, and the weight of the propagation convergence coefficient should be correspondingly increased. The Sigmoid transformation result of the topology centrality is used as the weight of the propagation convergence coefficient, and the rest of the weight is allocated to the feature deviation degree. The inverse value of the propagation convergence coefficient is taken, and the feature deviation degree is weighted and averaged to obtain the causal strength score. The score value range is between 0 and 1, and the value closer to 1 indicates that the service node has a greater possibility as the current abnormal root cause.

[0120] The present application realizes the accurate positioning of the abnormal root cause by the organic combination of the propagation convergence coefficient and the feature deviation degree. The propagation convergence coefficient provides the causal reasoning from the perspective of the topology structure, and the feature deviation degree provides the similarity analysis from the perspective of the state feature, and the two complement each other to form a multi-dimensional root cause evaluation system. The adaptive weighting mechanism based on historical experience effectively balances the importance difference of different service nodes in the abnormal propagation network, and improves the accuracy of root cause analysis. It can effectively deal with the cascading failure scene in the complex micro-service architecture, shorten the fault positioning time, reduce the work burden of the operation and maintenance personnel, improve the service stability, provide reliable basis for subsequent intelligent recovery decision, and has significant value in guaranteeing business continuity.

[0121] According to the causal strength score, the fault influence degree is identified, the resource isolation strategy and the flow scheduling strategy of the service node are adaptively adjusted based on the fault influence degree, and the intelligent diagnosis and recovery of the system fault are realized, including:

[0122] The causal intensity score is divided into multiple score sequences according to a fixed time window, the mean value change and the variance change are extracted from the score sequence, the cumulative value of the mean value change is calculated to obtain the fault diffusion degree, the cumulative value of the variance change is calculated to obtain the fault fluctuation degree, and the product of the fault diffusion degree and the fault fluctuation degree is taken as the fault influence degree;

[0123] The resource usage change of the service node is monitored based on the fault influence degree, the growth rate and the decay rate of the resource usage are calculated, the time period in which the growth rate exceeds a preset growth threshold is marked as a resource isolation starting point, the time period in which the decay rate exceeds a preset decay threshold is marked as a resource isolation termination point, and a resource isolation strategy of the service node is generated according to the resource isolation starting point and the resource isolation termination point;

[0124] The resource isolation strategy is converted into a resource limit vector, the computing resource limit value and the storage resource limit value are extracted from the resource limit vector, the concurrent processing threshold of the service node is determined by the computing resource limit value, the cache processing threshold of the service node is determined by the storage resource limit value, the traffic scheduling strategy of the service node is updated according to the concurrent processing threshold and the cache processing threshold, and intelligent diagnosis and recovery of system faults are realized.

[0125] A sliding window method is adopted, the window size is set to 30 seconds, and the sliding step is 10 seconds. For each score sequence segment obtained by cutting, the statistical characteristics, including the mean value and the variance, are calculated. The mean value change between adjacent windows reflects the overall trend of the causal intensity score, and the variance change reflects the fluctuation degree change of the score. In actual application, the mean values of the causal intensity scores of a service node in three consecutive time windows are 0.35, 0.62 and 0.78, respectively, the mean value changes of adjacent windows are 0.27 and 0.16, and the variances are 0.05, 0.08 and 0.12, respectively, and the variance changes are 0.03 and 0.04.

[0126] The cumulative value of the mean value change is calculated to obtain the fault diffusion degree. The mean value changes of the time windows are accumulated in time sequence, and a time decay factor can be applied in the accumulation process to gradually weaken the influence of the changes of the earlier time windows on the cumulative result. The time decay factor adopts an exponential decay form, and the decay rate is set to 0.9. The fault diffusion degree reflects the persistence and cumulative effect of the growth of the causal intensity score, and the larger the value is, the more serious the fault influence range diffusion is.

[0127] The cumulative value of the variance change amount is calculated to obtain the fault fluctuation degree. Similar to the calculation method of the fault diffusion degree, the variance change amount is accumulated in time sequence, and a time decay factor is also applied. The fault fluctuation degree reflects the cumulative effect of the cause intensity score fluctuation, and the larger the value is, the higher the uncertainty of the fault impact is. For a certain service node, if the variance change amounts of 5 consecutive time windows are 0.03, 0.04, 0.02, 0.05, and 0.06, and the time decay factor is considered, the calculated fault fluctuation degree is 0.17.

[0128] The product of the fault diffusion degree and the fault fluctuation degree is taken as the fault impact degree. The multiplication of the two considers both the expansion of the fault impact range and the instability of the fault impact, and the obtained fault impact degree can comprehensively reflect the severity of the fault. The value of the fault impact degree is usually normalized to the range of 0 to 1, which is convenient for subsequent processing and decision-making. By setting a threshold value of the fault impact degree, the service node can be divided into three levels of high impact, medium impact, and low impact, which correspond to different resource isolation and traffic scheduling strategies.

[0129] The resource usage change of the service node is monitored based on the fault impact degree. The resource usage mainly includes indicators such as CPU usage rate, memory usage rate, disk I / O rate, and network bandwidth usage rate. Time series sampling is adopted to collect the values of each resource indicator once a second, forming a resource usage time series. The change rate of the resource usage in a short time window (such as 10 seconds) is calculated to obtain the growth rate and decay rate of the resource usage.

[0130] For each resource indicator, the linear regression method in the sliding window is used to estimate its change trend. The window size is set to 10 seconds, and the slope of the regression line is the change rate of the resource indicator. If the slope is positive, it is the growth rate; if the slope is negative, the absolute value is the decay rate. For the CPU usage rate, if the sampling values in 10 seconds are 45%, 48%, 52%, 55%, 59%, 62%, 65%, 68%, 70%, and 73% in turn, the calculated growth rate by linear regression is about 3.1% / second.

[0131] The time period in which the growth rate exceeds the preset growth threshold is marked as the resource isolation starting point. The preset growth threshold is determined based on historical statistical data, and is usually set to 3 times the growth rate of the resource usage under normal load. The growth thresholds of different resource indicators are different, such as the growth threshold of the CPU usage rate can be set to 2% / second, and the growth threshold of the memory usage rate can be set to 1% / second. When the growth rate of a certain resource indicator exceeds its corresponding growth threshold for 3 consecutive sampling periods, the current time point is marked as the isolation starting point of the resource.

[0132] The time period with a decay rate exceeding a preset decay threshold is marked as a resource isolation termination point. The preset decay threshold is usually set to 50% to 70% of the growth threshold, ensuring that the resource usage has decreased significantly before isolation is lifted. When the decay rate of the resource indicator exceeds the decay threshold for 5 consecutive sampling periods, the current time point is marked as the isolation termination point for the resource.

[0133] A resource isolation policy for the service node is generated according to the resource isolation start point and the resource isolation termination point. The resource isolation policy includes three dimensions: isolation time period, isolated resource type, and isolation degree. The isolation degree is determined according to the fault impact degree, and is usually divided into three levels: complete isolation, partial isolation, and monitoring without isolation. Complete isolation means temporarily removing the service node from the service cluster; partial isolation means limiting the resource usage upper limit of the service node; monitoring without isolation means only strengthening monitoring without taking actual isolation measures.

[0134] The resource isolation policy is converted into a resource limit vector. The resource limit vector is a multi-dimensional vector, and each dimension corresponds to a limit value of a resource type, such as CPU core limit, memory capacity limit, etc. According to the isolation degree, the limit proportion of each dimension is set. For the partial isolation level, the resource is usually limited to 60% to 80% of the normal configuration; for the complete isolation level, it is limited to the minimum runnable configuration, such as 20% to 40%. For a service node, if the normal configuration is 4-core CPU and 8GB memory, and the fault impact degree is evaluated as medium, the partial isolation policy is adopted, then the resource limit vector can be set to [2.4 cores, 5.6 GB], i.e. limited to 70% of the normal configuration.

[0135] The computing resource limit value and the storage resource limit value are extracted from the resource limit vector. The computing resource limit value mainly refers to the limit of processor resources such as CPU and GPU, and the storage resource limit value mainly refers to the limit of storage media such as memory and disk. The computing resource limit value directly affects the concurrent processing capability of the service node, and the storage resource limit value affects the data caching capability of the service node.

[0136] The concurrent processing threshold of the service node is determined by the computing resource limit value. The concurrent processing threshold refers to the maximum number of requests that the service node can handle simultaneously. The number of requests that a unit of computing resource can handle is obtained through performance testing, and then multiplied by the computing resource limit value to obtain the concurrent processing threshold. For example, if it is measured that each core CPU can handle an average of 50 requests simultaneously, and the computing resource limit value is 2.4 cores, then the concurrent processing threshold is set to 120 requests.

[0137] The cache processing threshold of the service node is determined according to the storage resource limit value. The cache processing threshold refers to the maximum data amount that can be cached by the service node. According to historical statistical data, the data amount that can be effectively cached per unit storage resource is determined, and then multiplied by the storage resource limit value to obtain the cache processing threshold. If about 100,000 records can be effectively cached per GB of memory, and the storage resource limit value is 5.6 GB, then the cache processing threshold is set to 560,000 records.

[0138] The traffic scheduling strategy of the service node is updated according to the concurrent processing threshold and the cache processing threshold, so as to realize intelligent diagnosis and recovery of system failure. The traffic scheduling strategy includes two parts of inbound traffic control and outbound traffic redirection. The inbound traffic control is based on the concurrent processing threshold, and when the current number of concurrent requests approaches 80% of the threshold, flow control is started; when the threshold is reached, new requests are rejected or forwarded to other healthy nodes. The outbound traffic redirection is based on service call chain analysis, and identifies downstream nodes that frequently interact with the failed node, and redirects calls to these nodes to other healthy instances to reduce failure propagation.

[0139] Through time sequence analysis of the causal intensity score, the precise quantification of the degree of failure influence is realized, which provides a scientific basis for subsequent resource isolation and traffic scheduling. Based on the adaptive adjustment mechanism of the degree of failure influence, a differentiated recovery strategy can be dynamically generated according to the severity of the failure, avoiding the rigid problem of the traditional fixed threshold method. The cooperative optimization of resource isolation and traffic scheduling not only guarantees the overall availability of the system, but also maximally reduces the impact on normal business. It provides a strong guarantee for the stable operation of the core business of the enterprise, and has wide application value in large-scale distributed systems.

[0140] In a second aspect of the embodiments of the present application, a system abnormality intelligent diagnosis and recovery system integrating time sequence logs is provided, and the system comprises:

[0141] A log collection unit is configured to collect running logs from a plurality of service nodes of a distributed system and perform time sequence alignment according to timestamps, extract call relationship and performance metric data from the time sequence aligned running logs;

[0142] A feature extraction unit is configured to calculate time sequence variation features of the performance metric data within a sliding time window, fuse and encode the time sequence variation features and the call relationship to generate a dynamic feature vector corresponding to the service node, and construct the dynamic feature vector into a cross-service time sequence correlation graph based on the call relationship;

[0143] A fault matching unit is configured to identify an abnormal service node from the cross-service time sequence correlation graph, match the dynamic feature vector corresponding to the abnormal service node with historical fault records in a historical fault database, and obtain a historical propagation path and a historical feature vector in the historical fault records;

[0144] A cause-effect analysis unit is configured to calculate a propagation convergence coefficient of each service node in the cross-service timing correlation graph based on a historical propagation path, calculate a feature deviation degree of a dynamic feature vector and a historical feature vector of each service node, and obtain a cause-effect strength score of each service node by fusing the propagation convergence coefficient and the feature deviation degree.

[0145] A fault recovery unit is configured to identify a fault influence degree according to the cause-effect strength score, adaptively adjust a resource isolation strategy and a traffic scheduling strategy of the service node based on the fault influence degree, and realize intelligent diagnosis and recovery of system faults.

[0146] In a third aspect, an electronic device is provided, including:

[0147] A processor;

[0148] A memory for storing processor-executable instructions;

[0149] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0150] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0151] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer-readable storage medium having stored thereon computer-readable program instructions that, when executed by a computer, cause the computer to carry out various aspects of the present application.

[0152] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for intelligent diagnosis and recovery of system anomalies by integrating time-series logs, characterized in that, include: Run logs are collected from multiple service nodes in a distributed system and aligned to a time sequence by timestamps. Call relationships and performance metrics are then extracted from the time-series aligned run logs. Within a sliding time window, the time-series variation characteristics of performance metrics are calculated, and the time-series variation characteristics are fused and encoded with the call relationship to generate dynamic feature vectors corresponding to service nodes. Based on the call relationship, the dynamic feature vectors are used to construct a cross-service time-series association graph. Abnormal service nodes are identified from the cross-service time sequence correlation graph. The dynamic feature vectors corresponding to the abnormal service nodes are matched with historical fault records in the historical fault database to obtain the historical propagation paths and historical feature vectors in the historical fault records. The directed propagation edges between historical service nodes and the historical latency distribution corresponding to the directed propagation edges are extracted from the historical propagation path. Candidate propagation subgraphs that match the topology of the historical propagation path are identified in the cross-service temporal association graph. The upstream propagation tree is constructed by multi-hop backtracking along the in-degree direction for each target service node in the candidate propagation subgraph. Calculate the latency evolution deviation between the historical latency distribution of directed propagation edges in the upstream propagation tree and the actual latency distribution of the corresponding directed propagation edges in the cross-service time sequence association graph; Based on the time delay evolution deviation, the directed propagation edges in the upstream propagation tree are weighted by propagation attenuation and normalized to obtain the propagation convergence coefficient of the target service node. Calculate the feature deviation between the dynamic feature vector and the historical feature vector of each service node, and fuse the propagation convergence coefficient and the feature deviation to obtain the causal strength score of each service node. The degree of fault impact is identified by causal strength score, and the resource isolation strategy and traffic scheduling strategy of service nodes are adaptively adjusted based on the degree of fault impact to achieve intelligent diagnosis and recovery of system faults; Within a sliding time window, the time-series variation characteristics of performance metrics are calculated. These characteristics are then fused and encoded with call relationships to generate dynamic feature vectors corresponding to service nodes. Based on the call relationships, these dynamic feature vectors are used to construct a cross-service time-series correlation graph, including: A sliding time window is set to segment the performance measurement data in time series. Within each sliding time window, a time series phase space of the performance measurement data is constructed. The trajectory curvature and attractor features of the performance measurement data are calculated in the time series phase space as time series variation features. Identify the level depth and branch density of service nodes in the call chain from the call relationship, and calculate the attenuation coefficient of the call propagation path based on the level depth and branch density; The temporal change features of upstream service nodes in the call relationship are weighted and aggregated according to the call attenuation coefficient. The temporal change features and the weighted aggregated upstream temporal features are then fused and encoded across levels through a multi-head attention mechanism to generate dynamic feature vectors corresponding to the service nodes. Using service nodes as graph nodes and assigning them dynamic feature vectors, the cosine similarity between the dynamic feature vectors of adjacent service nodes in the call relationship is calculated. The cosine similarity is then used as the weight value of the graph edge to construct a cross-service temporal association graph.

2. The method according to claim 1, characterized in that, Runtime logs are collected from multiple service nodes in a distributed system and aligned to a specific time sequence using timestamps. Call relationship and performance metric data are extracted from the time-aligned logs, including: Raw operation logs are collected from multiple service nodes. Global clock synchronization calibration is performed on the raw operation logs based on timestamps. The calibrated raw operation logs are then sorted according to the order of timestamps to construct time-aligned operation logs. Extract request tracing identifiers from time-aligned runtime logs, associate log records spanning multiple service nodes into a complete call chain based on the request tracing identifiers, and construct weighted call relationships based on the call order and call frequency of service nodes in the call chain; The performance metrics data of each service node are extracted from the time-aligned runtime logs. The performance metrics data are then grouped into time-series slices according to timestamps. The statistical feature values ​​of the performance metrics within each time-series slice are calculated. Finally, the statistical feature values ​​of the performance metrics are matched with the call relationships to generate performance measurement data.

3. The method according to claim 1, characterized in that, Abnormal service nodes are identified from the cross-service time-series correlation graph. The dynamic feature vectors corresponding to the abnormal service nodes are matched with historical fault records in the historical fault database to obtain the historical propagation paths and historical feature vectors from the historical fault records, including: Calculate the dynamic feature vector difference matrix of adjacent time windows in the cross-service temporal correlation graph, decompose the dynamic feature vector difference matrix to obtain the temporal fluctuation component and the topology propagation component, and construct the state transition sequence based on the temporal fluctuation component and the topology propagation component. The entropy values ​​of the temporal fluctuation component matrix and the topology propagation component matrix in the state transition sequence are calculated. Anomaly scores are generated by combining the entropy values ​​of the temporal fluctuation component and the topology propagation component. Anomaly service nodes are identified based on the anomaly scores, and anomaly propagation links centered on the anomaly service nodes are extracted from the state transition sequence. The temporal fluctuation spectrum component is extracted from the dynamic feature vector of the abnormal service node, and the propagation delay spectrum component is extracted from the abnormal propagation link. The temporal fluctuation spectrum component and the propagation delay spectrum component are combined to construct a dual-spectrum feature. Based on the dual-spectrum feature, historical fault records are retrieved and matched in the historical fault database. Based on the historical fault records, historical propagation paths and historical feature vectors are obtained.

4. The method according to claim 1, characterized in that, The feature deviation between the dynamic feature vector and the historical feature vector of each service node is calculated. The causal strength score of each service node is obtained by fusing the propagation convergence coefficient and the feature deviation, including: Temporal components and topological components are extracted from the dynamic feature vectors and historical feature vectors of each target service node, respectively. The temporal deviation between temporal components and the topological deviation between topological components are calculated. The temporal deviation and topology deviation are weighted and combined to generate the feature deviation of each target service node; Calculate the topological centrality of the target service node in the cross-service temporal correlation graph, and adaptively weight and fuse the propagation convergence coefficient and feature deviation degree based on the topological centrality to obtain the causal strength score of each target service node.

5. The method according to claim 1, characterized in that, The system identifies the degree of fault impact based on causality strength scores, and adaptively adjusts resource isolation and traffic scheduling strategies for service nodes based on this degree of impact to achieve intelligent fault diagnosis and recovery. The causal intensity score is divided into multiple score sequences according to a fixed time window. The mean change and variance change are extracted from the score sequences. The cumulative value of the mean change is calculated to obtain the fault diffusion degree, and the cumulative value of the variance change is calculated to obtain the fault volatility degree. The product of the fault diffusion degree and the fault volatility degree is used as the degree of fault impact. Based on the degree of impact of the fault, monitor the changes in resource usage of the service node, calculate the growth rate and decay rate of resource usage, mark the time period when the growth rate exceeds the preset growth threshold as the resource isolation start point, and mark the time period when the decay rate exceeds the preset decay threshold as the resource isolation end point, and generate the resource isolation strategy for the service node based on the resource isolation start point and resource isolation end point. The resource isolation strategy is converted into a resource constraint vector. Computational resource constraint values ​​and storage resource constraint values ​​are extracted from the resource constraint vector. The concurrent processing threshold of the service node is determined by the computational resource constraint value, and the cache processing threshold of the service node is determined by the storage resource constraint value. The traffic scheduling strategy of the service node is updated according to the concurrent processing threshold and the cache processing threshold, so as to realize intelligent diagnosis and recovery of system faults.

6. A system for intelligent diagnosis and recovery of system anomalies integrating time-series logs, used to implement the method of any one of claims 1-5, characterized in that, include: The log collection unit is used to collect runtime logs from multiple service nodes in the distributed system and align them according to timestamps. It then extracts call relationship and performance measurement data from the time-aligned runtime logs. The feature extraction unit is used to calculate the time-series variation features of performance metric data within a sliding time window, fuse and encode the time-series variation features with the call relationship, generate dynamic feature vectors corresponding to service nodes, and construct a cross-service time-series association graph based on the call relationship of the dynamic feature vectors. The fault matching unit is used to identify abnormal service nodes from the cross-service time sequence correlation graph, match the dynamic feature vectors corresponding to the abnormal service nodes with historical fault records in the historical fault database, and obtain the historical propagation paths and historical feature vectors in the historical fault records. The causal analysis unit is used to calculate the propagation convergence coefficient of each service node in the cross-service time-series correlation graph based on the historical propagation path, calculate the feature deviation degree between the dynamic feature vector and the historical feature vector of each service node, and fuse the propagation convergence coefficient and the feature deviation degree to obtain the causal strength score of each service node. The fault recovery unit is used to identify the degree of fault impact based on the causal strength score, and adaptively adjust the resource isolation strategy and traffic scheduling strategy of the service node based on the degree of fault impact to realize intelligent diagnosis and recovery of system faults.

7. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 5.

Images