A multi-cloud environment log intelligent aggregation and threat tracing analysis method and system

Abstract

The application relates to the technical field of cloud computing, and discloses a log intelligent aggregation and threat tracing analysis method and system in a multi-cloud environment, which comprises the following steps: collecting original logs in the multi-cloud environment, injecting unified service identification and cloud platform labels, analyzing heterogeneous log formats into structured data, sorting according to timestamps and completing missing log entries; constructing a service dependency graph, quantifying the call relationship weight between services, fusing log semantic features and statistical features, and generating a feature matrix after standardization. The application realizes the standardized aggregation of multi-cloud logs through unified service identification, solves the problem of heterogeneous log processing, significantly improves the accuracy of complex threat identification and the propagation path verification capability through the feature fusion and cascading integrated detection mechanism based on the service dependency graph, realizes the accurate positioning of cross-cloud attack paths through the service identification-based tracing analysis, and effectively supports the root cause diagnosis and security protection in the multi-cloud environment.

Description

Technical Field

[0001] This invention relates to the field of cloud computing, and more specifically, to a method and system for intelligent log aggregation and threat tracing analysis in a multi-cloud environment. Background Technology

[0002] With the rapid development of cloud computing technology, multi-cloud environments have become the mainstream deployment method for modern enterprise IT infrastructure. By integrating multiple public, private, and hybrid cloud services, they improve resource elasticity and business continuity. However, this environment also brings significant operational challenges, especially the dispersion and heterogeneity of log data, leading to low log aggregation efficiency and difficulty in tracing security threats.

[0003] Traditional log management tools (such as the ELK stack) often face problems such as incompatible data formats, insufficient real-time performance, and limited intelligent analysis capabilities when processing multi-cloud logs, making it difficult to effectively support threat detection and attack path reconstruction.

[0004] When enterprises use a hybrid platform, heterogeneous log formats and dynamic parsing engines are generated. Existing log management tools may miss key data and reduce the accuracy of threat detection. Summary of the Invention

[0005] This invention provides a method and system for intelligent log aggregation and threat tracing analysis in multi-cloud environments, solving the technical problems in related technologies where enterprises use mixed platforms, resulting in heterogeneous log formats and dynamic parsing engines. Existing log management tools may also lack key data and reduce the accuracy of threat detection.

[0006] This invention provides a method for intelligent log aggregation and threat tracing analysis in a multi-cloud environment, comprising the following steps:

[0007] S100, intelligent log aggregation: collects raw logs from multi-cloud environments, injects unified service identifiers and cloud platform tags, parses heterogeneous log formats into structured data, sorts them by timestamps, and completes missing log entries;

[0008] S200, Feature Fusion and Graph Construction: Construct a service dependency graph, quantify the weight of call relationships between services, fuse log semantic features and statistical features, and generate a feature matrix after standardization;

[0009] S300, Cascaded Integrated Threat Detection: Uses the LSTM-Transformer model to analyze the time-series and semantic features of logs, outputs a preliminary threat score, verifies the threat propagation path in the service dependency graph through a graph neural network, and generates a source tracing graph and a set of threat events;

[0010] S400, Service Identifier-Driven Source Tracing: Extracts cross-cloud attack paths from the source map, corrects path weights by combining multi-cloud labels, quantifies the impact of threats on each service node, locates the root cause service node, and outputs the impact matrix.

[0011] Furthermore, the steps involved in intelligent log aggregation are as follows:

[0012] S110, Multi-protocol Dynamic Acquisition: Uses a log type determination function to analyze the format type of each raw log, and calls the corresponding parser to process the log based on the log type, generating parsed structured data;

[0013] S120, Non-standard field completion: Detects missing standard fields in the parsed log, uses a rule engine to complete the missing fields from external documents, and adds the completed fields and corresponding values ​​to the parsed structured log;

[0014] S130, Unified Semantic Mapping: Converts the enhanced logs into a standardized structure format and assigns credibility weights to fields from different sources;

[0015] S140, Log Stream Aggregation: Arrange all standardized structures in timestamp order to generate an ordered log stream, and verify whether the success rate of log parsing meets the quality requirements.

[0016] Furthermore, the steps of feature fusion and graph construction are as follows:

[0017] S210, Multimodal Feature Extraction: Extracting three types of anti-interference features from each log entry:

[0018] Temporal characteristics: the frequency of occurrence of a specific event within a statistical time window;

[0019] Semantic features: Using the BERT model to generate embedding vectors for threat keywords;

[0020] Graph features: Calculate inter-service dependency weights based on the number of service calls;

[0021] S220, Feature Matrix Construction: Concatenate the feature vectors of all logs row by row to construct the initial feature matrix;

[0022] S230, Noise Sample Filtering: Use an isolated forest to calculate the anomaly score for each sample, filter out anomaly samples based on a threshold, and obtain the cleaned feature matrix;

[0023] S240, Feature Standardization: Standardize the filtered feature matrix to make features of different dimensions comparable.

[0024] Furthermore, the steps for cascading integrated threat detection are as follows:

[0025] S310, Temporal-Semantic Feature Modeling: Input the feature matrix output by S200 for temporal modeling, use LSTM to capture long-term temporal dependencies, generate hidden state sequences, enhance semantic features through Transformer to obtain semantic vectors, and predict the preliminary threat score for each log based on the semantic vectors;

[0026] S320, Candidate Threat Filtering: Based on the comparison between the initial threat score and the threshold, logs that are higher than the threshold are filtered out to obtain a set of candidate threats;

[0027] S330, Graph Structure Validation: Construct a service dependency subgraph based on the candidate threat log set, apply GAT for message passing, update node representations, and calculate the threat score of each node through MLP;

[0028] S340, Threat Determination and Source Graph Generation: Combine LSTM scores and GNN scores for dual verification to determine the final set of threat events. Construct a source graph based on the threat events, retaining relevant nodes and edges.

[0029] Furthermore, in S310, the semantic vector is passed through a linear transformation layer, and the output value is compressed to between 0 and 1 using the Sigmoid function to obtain the preliminary threat score of the log. The closer the preliminary threat score is to 1, the higher the probability of a threat.

[0030] Furthermore, in S340, the analysis results of the LSTM model and the GNN model are combined to make the final threat determination. For each log record, it will only be judged as a real threat event if its LSTM threat score exceeds 0.8 and the corresponding service node's GNN threat score exceeds 0.9.

[0031] Furthermore, the steps for service identifier-driven tracing are as follows:

[0032] S410, Attack Path Extraction: In the source map, search for the propagation path starting from the threat event;

[0033] S420, Cross-Cloud Threat Propagation Analysis: Identify cross-cloud attack paths and make judgments using cloud platform tags in S100;

[0034] S430, Threat Impact Quantification: Calculate the threat impact value of service nodes and generate an impact matrix;

[0035] S440, Root Cause Service Location: Locating the source of an attack based on the impact matrix.

[0036] Furthermore, the steps for attack path extraction are as follows:

[0037] The path is defined as a sequence of nodes in the source graph, where each sequence consists of multiple consecutive service nodes, and there are direct call relationships between the nodes;

[0038] Calculate the weight value for each path;

[0039] The weight threshold for each path is set to 0.6, and paths with a weight greater than this threshold are selected as valid attack paths.

[0040] Furthermore, the steps for quantifying threat impact are as follows:

[0041] Calculate the threat impact value of each service node, arrange the threat impact values ​​of all service nodes in order of node number to form an impact matrix, where each element of the matrix corresponds to the threat impact value of a service node, and the dimension of the matrix is ​​equal to the total number of nodes in the source graph.

[0042] This invention also proposes a log intelligent aggregation and threat tracing analysis system for multi-cloud environments, comprising:

[0043] Log intelligent aggregation module: collects heterogeneous raw logs from multiple cloud platforms, supports mainstream formats, injects a unified service identifier and cloud platform tag into each log, parses the log content into structured data, sorts log entries by timestamp, detects and completes missing entries, and generates a standardized log dataset.

[0044] Feature fusion and graph construction module: Constructs a directed graph based on service call relationships, where nodes represent services and edges represent call chains. Weights quantify call frequency and response latency. Semantic and statistical features are extracted from logs to form an initial feature vector. Features are Z-score standardized to eliminate dimensional differences. Log features and graph structure are fused to generate a unified feature matrix, providing a standard feature matrix and service dependency graph as input for threat detection.

[0045] Cascaded integrated threat detection module: Uses LSTM-Transformer model to analyze the temporal patterns and semantic associations of logs, outputs a preliminary threat score for each log, filters high-confidence candidate threat logs based on thresholds, verifies the threat propagation path on the service dependency graph through graph neural network, optimizes the judgment results by combining node threat scores, and outputs the final set of threat events and source map;

[0046] Service Identifier-Driven Origin Module: Identifies attack paths with service identifiers as nodes in the origin graph, marks the propagation links across cloud platforms, adjusts path weights based on cross-cloud labels, quantifies the propagation cost in multi-cloud environments, calculates the impact of threats on service nodes, generates a threat impact matrix, assesses the attack spread range, locates the source service node of the attack, and outputs the root cause ID and impact matrix.

[0047] The beneficial effects of this invention are as follows:

[0048] This invention achieves standardized aggregation of multi-cloud logs through unified service identifiers, solving the problem of heterogeneous log processing. Based on the feature fusion and cascading integration detection mechanism of service dependency graphs, it significantly improves the accuracy of complex threat identification and propagation path verification capabilities. Combined with the source tracing analysis of service identifiers, it achieves precise location of cross-cloud attack paths, effectively supporting root cause diagnosis and security protection in multi-cloud environments. Attached Figure Description

[0049] Figure 1 This is a flowchart of a method for intelligent log aggregation and threat tracing analysis in a multi-cloud environment proposed in this invention;

[0050] Figure 2 This is a schematic diagram of the structure of a multi-cloud environment log intelligent aggregation and threat tracing analysis system proposed in this invention.

[0051] In the diagram: 101, Log Intelligent Aggregation Module; 102, Feature Fusion and Graph Construction Module; 103, Cascaded Integrated Threat Detection Module; 104, Service Identifier-Driven Source Tracing Module. Detailed Implementation

[0052] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0053] like Figure 1 As shown, a method for intelligent log aggregation and threat tracing analysis in a multi-cloud environment includes the following steps:

[0054] S100, intelligent log aggregation: collects raw logs from multi-cloud environments, injects a unified service identifier (Service-ID) and a cloud platform tag (cloud-ID), parses heterogeneous log formats (text / JSON / binary) into structured data, sorts them by timestamp, and completes missing log entries;

[0055] In one embodiment of the present invention, the following steps are specifically included:

[0056] S110, Multi-protocol dynamic acquisition: Uses a log type determination function to analyze the format type (Protobuf / JSON / text) of each raw log, and calls the corresponding parser to process the log based on the log type, generating parsed structured data;

[0057] For each input raw log, the format type is determined. The determination process is completed by checking the feature identifiers of the log content: when the Protobuf format feature is detected, the type value 1 is returned; when the JSON format feature is detected, the type value 2 is returned; and when it is determined to be plain text format, the type value 3 is returned.

[0058] Log type determination function:

[0059] ;

[0060] in, This function determines the log type and returns 1, 2, or 3, representing different log formats. It is in plain text format. This is a JavaScript object representation format. It is a binary protocol format. This is the i-th original log entry;

[0061] Based on the identified log type, the system selects the appropriate parsing method: for Protobuf logs, a dedicated Protobuf decoder is used for parsing, and the corresponding protocol description file must be provided as the basis for parsing; for JSON logs, a JSON parser is used for direct parsing; for logs of ordinary text type, a finely tuned BERT model is used for semantic parsing to convert unstructured text into structured data.

[0062] Protocol offloading execution:

[0063] ;

[0064] in, For the i-th original log entry, For the Protobuf protocol description file, These are the fine-tuned BERT model parameters. For Protobuf decoder functions, This is a JSON parser function. This is the fine-tuned BERT model. The parsed structured log;

[0065] S120, Non-standard field completion: Detects missing standard fields in the parsed log, uses a rule engine to complete the missing fields from external documents, and adds the completed fields and corresponding values ​​to the parsed structured log;

[0066] The system examines each parsed structured log record, comparing it against a set of standard fields (including required fields such as timestamp and event-type) to identify any missing fields in the current log. This process is accomplished by iterating through the list of standard fields and checking if the fields exist in the current log.

[0067] Missing field detection:

[0068] ;

[0069] in, For field names, A function to retrieve all field names from a log. For a standard set of fields, For the set of missing fields;

[0070] For any missing fields discovered, the system will invoke the rules engine and refer to external documentation such as OpenStack to complete the field values. The rules engine will generate appropriate values ​​for the missing fields based on the context information of the existing fields and the preset completion rules.

[0071] Rule engine completion:

[0072] ;

[0073] in, For the external field manual (OpenStack documentation). For rule engine functions, The set of field values ​​to be completed;

[0074] The completed field-value pairs are added to the original parsed log, generating an enhanced log record with all fields complete. This process is achieved by merging the original parsed data with the newly added field-value pairs.

[0075] Field enhancement:

[0076] ;

[0077] in, For field values, For the enhanced log, The parsed structured log, For the set of missing fields, For the complete set of field values, For field names;

[0078] S130, Unified Semantic Mapping: Converts the enhanced logs into a standardized structure format and assigns credibility weights to fields from different sources;

[0079] The system uses a structure conversion function to convert the enhanced log records into a unified standard format. The converted structure contains a fixed set of fields, such as timestamps, service identifiers, event types, and other key information.

[0080] Structure conversion:

[0081] ;

[0082] in, As confidence weights, For structure conversion functions, It is a standardized structure;

[0083] Meanwhile, the system assigns different reliability weights to each field in the logs. Fields present in the original logs are assigned a full weight of 0 due to their high reliability; fields obtained through completion are assigned a lower weight of 0.7, considering their potential uncertainty. This differentiated weighting helps distinguish the reliability of data from different sources during subsequent analysis.

[0084] Confidence weight calculation:

[0085] ;

[0086] in, This represents the weight calculation function. This is the default weight value for the original field. For the set of missing fields;

[0087] S140, Log Stream Aggregation: Arrange all standardized structures in timestamp order to generate an ordered log stream, and verify whether the success rate of log parsing meets the quality requirements.

[0088] The system will sort all the converted standard structures in ascending order according to their timestamp fields, forming a complete time-series log stream. This sorting process ensures that log records are organized in the chronological order of events.

[0089] Log stream generation:

[0090] ;

[0091] in, For a standardized structure, Total log volume It means according to from Sort the extracted timestamps. This is a sorted log stream;

[0092] The system calculates the success rate of log parsing; specifically, it measures the percentage of successfully parsed log entries (i.e., log entries that generated non-empty structures) out of the total number of log entries. The system requires this success rate to exceed 98% to ensure the completeness and reliability of the log aggregation results. If the success rate falls below this threshold, it indicates a potential problem with the current parsing process, requiring optimization or manual intervention.

[0093] Success rate verification:

[0094] ;

[0095] in, It is an empty set. The cardinality of the set (number of elements). To improve the success rate of parsing, This is the success rate threshold;

[0096] S200, Feature Fusion and Graph Construction: Construct a service dependency graph, quantify the weight of inter-service call relationships (such as API call frequency), fuse log semantic features (keyword TF-IDF) and statistical features (anomaly indicators), and generate a feature matrix after standardization;

[0097] In one embodiment of the present invention, the following steps are specifically included:

[0098] S210, Multimodal Feature Extraction: Extracting three types of anti-interference features from each log entry:

[0099] Temporal characteristics: the frequency of occurrence of a specific event within a statistical time window;

[0100] Semantic features: Using the BERT model to generate embedding vectors for threat keywords;

[0101] Graph features: Calculate inter-service dependency weights based on the number of service calls;

[0102] Timing characteristics: For each log record, the system counts the occurrences of a specific type of event within a specified time range (e.g., 300 seconds) before and after it. Specifically, it iterates through all logs within that time window, incrementing the count by 1 when a log with the same event type as the target event is found.

[0103] This statistic reflects the density of target events within a short period of time, which helps in identifying abnormal event outbreaks.

[0104] Temporal characteristics (events) In the time window (frequency within)

[0105] ;

[0106] in, For log timestamp, Let j be the timestamp of the j-th log entry. The size of the time window (e.g., 300 seconds). For the target event type (e.g., Login Failure). For the event type of the j-th log entry, This is an indicator function; it returns 1 when the condition is true and 0 otherwise. For the event Frequency of occurrence within the time window;

[0107] Semantic features: The system uses a pre-trained BERT model to process threat keywords in logs. Keywords (such as "SQL injection") are concatenated with special classification labels, input into the BERT model, and a 768-dimensional feature vector is extracted.

[0108] This vector encodes the semantic information of the keywords, enabling the system to understand the semantic relationships between different threat keywords.

[0109] Semantic features (keywords) BERT embedding):

[0110] ;

[0111] in, For threat keywords (such as SQL injection). Embed the keyword vector. This is a text concatenation operation. For BERT encoder functions, Classify and label BERT;

[0112] Graph characteristics: Calculate the dependency strength between services by analyzing service call relationships. Specifically, this involves counting the number of times service A calls service B in historical logs, dividing this number by the total number of times service A calls all adjacent services, and obtaining a weight value between 0 and 1.

[0113] This weight reflects the degree of interaction between the two services.

[0114] Graph features (service dependency weights):

[0115] ;

[0116] in, For service identifier (from) ), For service All adjacent services, Serving the historical logs Call service Number of times, For any adjacent service of service u, For service To Service Dependency weights Serving the historical logs Call service The number of times;

[0117] The system combines these three types of features into a comprehensive feature vector, which contains one temporal feature value, 768 semantic feature values, and one graph feature value. This comprehensive feature vector fully describes multiple aspects of the log's characteristics.

[0118] ;

[0119] in, ;

[0120] S220, Feature Matrix Construction: Concatenate the feature vectors of all logs row by row to construct the initial feature matrix;

[0121] The system arranges the 770-dimensional feature vectors of each log in the order of the logs to construct a large feature matrix;

[0122] Each row of the matrix corresponds to a feature vector of a log entry, with the number of rows equal to the total number of log entries and a fixed number of columns (770). This matrix structure enables the system to process the feature information of all log entries uniformly.

[0123] ;

[0124] in, Let i be the feature vector of the i-th log entry. This is the initial feature matrix;

[0125] S230, Noise Sample Filtering: Use an isolated forest to calculate the anomaly score for each sample, filter out anomaly samples based on a threshold, and obtain the cleaned feature matrix;

[0126] The system employs the Isolation Forest algorithm to identify and filter outomaly samples. For each feature vector, an anomalousness score is calculated: 100 decision trees are used, each randomly selecting 256 samples for construction. The anomalousness of a sample is assessed by the ease with which it becomes isolated. The anomalousness score is compared to a threshold of 0.65, retaining normal samples with scores below the threshold. This process effectively removes noisy data with anomalous feature distributions, improving the accuracy of subsequent analysis.

[0127] Calculate the sample anomaly score:

[0128] ;

[0129] in, The parameters for an isolated forest are (number of trees = 100, sample size = 256). These are abnormal scores. This is the function for the Isolation Forest algorithm;

[0130] Filtered noise sample:

[0131] ;

[0132] in, Abnormal threshold (empirical value) ), This is the cleaned feature matrix;

[0133] Output: The cleaned feature matrix ( )

[0134] S240, Feature Standardization: Standardize the filtered feature matrix to make features of different dimensions comparable;

[0135] The specific steps are as follows: Calculate the mean for each feature dimension: sum the values ​​of all samples in that dimension and divide by the total number of samples. Simultaneously calculate the standard deviation: sum the squared differences between all samples and the mean, divide by the number of samples, and take the square root.

[0136] Calculate the mean and standard deviation of each feature dimension:

[0137] ;

[0138] ;

[0139] in, Indexing for feature dimensions ( ), This represents the number of samples after cleaning. Let be the j-th dimension feature value of the i-th sample. Let be the mean of the j-th feature. Let be the standard deviation of the j-th feature;

[0140] For each feature value, a standardization transformation is performed: the original value is subtracted from the mean of that dimension, and then divided by the standard deviation. After this processing, the feature values ​​of each dimension are transformed into a standard distribution with a mean of 0 and a standard deviation of 1, eliminating the dimensional differences between different features.

[0141] Standardized eigenvalues:

[0142] ;

[0143] in, These are the standardized eigenvalues;

[0144] In the final standardized feature matrix, each element reflects the relative position of the original feature value in its dimension, which facilitates subsequent model processing.

[0145] Output the normalized matrix:

[0146] ;

[0147] in, The standardized feature matrix;

[0148] S300, Cascaded Integrated Threat Detection: Uses the LSTM-Transformer model to analyze the time-series and semantic features of logs, outputs a preliminary threat score, verifies the threat propagation path in the service dependency graph through a graph neural network (GNN), and generates a source map and a set of threat events;

[0149] In one embodiment of the present invention, the following steps are specifically included:

[0150] S310, Temporal-Semantic Feature Modeling (LSTM-Transformer): Input the feature matrix output by S200 for temporal modeling, use LSTM to capture long-term temporal dependencies, generate hidden state sequences, enhance semantic features through Transformer to obtain semantic vectors, and predict the initial threat score for each log based on the semantic vectors.

[0151] The system receives a log feature matrix that has undergone feature extraction and normalization as input. For each log record, the system uses a Long Short-Term Memory (LSTM) network to analyze its temporal features. The LSTM network generates new hidden states by processing the feature vector at the current time step and the hidden states and memory cell states from the previous time step. This process is repeated for log records at all time steps, ultimately resulting in a complete sequence of hidden states, where each hidden state has a dimension of 128.

[0152] Next, the system inputs the hidden state sequence generated by LSTM into the Transformer model for semantic feature enhancement. The Transformer model contains 8 attention heads and 4 layers, and analyzes the semantic relationships between log records through a self-attention mechanism, outputting an enhanced semantic vector sequence.

[0153] The system predicts a threat score based on the semantic vector of each log entry. Specifically, the semantic vector is passed through a linear transformation layer, and the output value is compressed to between 0 and 1 using the sigmoid function, yielding a preliminary threat score for that log entry. A score closer to 1 indicates a higher probability of a threat.

[0154] LSTM time series modeling (capturing long-term dependencies):

[0155] ;

[0156] ;

[0157] in, For time steps eigenvectors ( ), For time steps The hidden state, For time steps The state of the cells, This is the hidden state from the previous time step. This represents the cell state at the previous time step. LSTM parameters (hidden layer size) =128), The total length of the sequence. For the hidden layer dimension;

[0158] Transformer semantic enhancement (extracting keyword associations):

[0159] ;

[0160] in, For Transformer parameters (number of heads = 8, number of layers = 4). The hidden state sequence output by the LSTM. For semantic enhancement functions, The semantic vector matrix output by the Transformer

[0161] Threat score prediction:

[0162] ;

[0163] in, For the first Semantic vector of a log entry ( The OK), It is the Sigmoid activation function. For the initial threat score, , As a threat prediction bias term, used for linear transformation, This is the transpose of the weight vector;

[0164] S320, Candidate Threat Filtering: Based on the comparison between the initial threat score and the threshold, logs that are higher than the threshold are filtered out to obtain a set of candidate threats;

[0165] The system filters log entries based on a preliminary threat score. A threat threshold of 0.8 is set, and all log entries with a threat score higher than this threshold are filtered out to form a candidate threat set. This filtering process effectively reduces the number of logs requiring further analysis and improves subsequent processing efficiency.

[0166] ;

[0167] in, Threshold for LSTM-Transformer ( ), For candidate threat log set ( );

[0168] S330, Graph Structure Validation (GNN): Construct a service dependency subgraph based on the candidate threat log set, apply GAT for message passing, update node representations, and calculate the threat score of each node through MLP;

[0169] Based on the selected set of candidate threat logs, the system constructs a service dependency subgraph. Relevant service identifiers are extracted from each candidate threat log, and these services are used as nodes in the subgraph. Edges corresponding to service call relationships are added between these nodes to form a complete dependency subgraph structure.

[0170] Next, the system uses a 3-layer Graph Attention Network (GAT) to analyze the subgraph. In each layer, each node collects information from its neighboring nodes and aggregates it based on attention weights. The attention weights are calculated based on the original weights of the edges, reflecting the importance of interactions between different services. After processing with the ReLU activation function, a new representation of the node is obtained.

[0171] The system uses a multilayer perceptron (MLP) to process the final representation of each node, outputting a node threat score between 0 and 1. This score reflects the likelihood that the service node is under threat from a graph structure perspective.

[0172] Subgraph construction:

[0173] ;

[0174] ;

[0175] in, For the set of nodes in the subgraph, For the set of edges of the subgraph, For log Corresponding service identifier, For candidate threat log set

[0176] GAT Message Passing (3-layer Graph Attention Network):

[0177] ;

[0178] in, For the first Layer nodes The expression, For the l-th layer node The expression, For nodes The neighborhood group, Attention weights (based on edge weights) calculate), For the first Layer weight matrix, To modify the activation function of the linear unit, For the layer index of the GAT network, For neighboring nodes;

[0179] Node threat score:

[0180] ;

[0181] in, This represents the nodes of the last layer. For multilayer perceptron parameters, The number of GNN layers ( =3), For node threat score, , For multilayer perceptron functions;

[0182] S340, Threat Determination and Source Graph Generation: Combine LSTM scores and GNN scores for dual verification to determine the final set of threat events, construct a source graph based on the threat events, and retain relevant nodes and edges;

[0183] The system combines the analysis results of the LSTM and GNN models to make the final threat determination. For each log record, it is only considered a real threat event if its LSTM threat score exceeds 0.8 and the corresponding service node's GNN threat score exceeds 0.9. This dual verification mechanism can effectively reduce the false positive rate.

[0184] Final Threat Assessment:

[0185] ;

[0186] in, For the i-th log entry, The threat score predicted by the LSTM model. The node threat score predicted by the GNN model. For log Belonging service node LSTM score threshold ( =0.8), Validation threshold for GNN ( =0.9), For logical AND operator;

[0187] The system constructs a source graph based on confirmed threat events. All service nodes related to the threat events are added to the graph, and edges corresponding to the original dependencies are added between these nodes. This source graph preserves the critical path of threat propagation, aiding in subsequent analysis and tracing. The weight of each edge remains unchanged from the weight values ​​in the original service call relationships, representing the tightness of interaction between services.

[0188] Source tracing graph construction:

[0189] ;

[0190] in, For the constructed source map, For the set of nodes in the source graph, For the edge set of the origin graph, A collection of threat events, Let the set of edges of the original graph be . The first one in the figure The and the first Each node Let be the directed edge from node u to node v;

[0191] S400, Service Identifier-Driven Source Tracing: Extracts cross-cloud attack paths from the source map, corrects path weights by combining multi-cloud labels, quantifies the impact of threats on each service node, locates the root cause service node, and outputs the impact matrix.

[0192] In one embodiment of the present invention, the following steps are specifically included:

[0193] S410, Attack Path Extraction: In the source map, search for the propagation path starting from the threat event;

[0194] The path is defined as a sequence of nodes in the source graph, where each sequence consists of multiple consecutive service nodes with direct call relationships between them.

[0195] Path definition:

[0196] ;

[0197] in, For the first in the path 1 node For the first node to the second node The path representation of each node. This represents the path length (number of nodes). For the first in the path The path from the nth node to the nth node Directed edges of nodes, For the set of nodes in the source graph, Let the set of edges of the origin graph be . Indicates a line from arrive The path;

[0198] Calculate the weight value for each path. Obtain the edge weight values ​​(representing the strength of the service call relationship) between each pair of adjacent nodes in the path and the threat score of the starting node (generated by a graph neural network), and multiply them to obtain the local weight of that path segment. Calculate the average of all local weights as the overall weight of the entire path; this weight reflects the probability of an attack path.

[0199] Path weight calculation:

[0200] ;

[0201] in, For nodes arrive edge weights, For nodes Threat score, For path length, This is the index of the node in the path. For path The overall weight value, This is the path length normalization factor;

[0202] A path weight threshold of 0.6 is set, and paths with a weight greater than this threshold are selected as valid attack paths. This filtering process can eliminate attack paths with low probability and retain the most likely attack propagation paths.

[0203] Path filtering:

[0204] ;

[0205] in, The path weight threshold is η=0.6. A set of effective attack paths, This represents the overall weight value for path p. For paths that meet the weight threshold conditions, Indicates "makes", used in set definition;

[0206] S420, Cross-Cloud Threat Propagation Analysis: Identify cross-cloud attack paths and use the cloud-ID in S100 for judgment;

[0207] For each pair of adjacent service nodes in the path, their cloud platform identifiers (cloud-IDs) are compared to determine whether it is a cross-cloud call. If the cloud-IDs of the two nodes are different, it is determined to be a cross-cloud edge; otherwise, it is determined to be a same-cloud edge.

[0208] Cross-cloud edge determination:

[0209] ;

[0210] in, This is the identifier of the cloud platform to which node u belongs. This is the identifier of the cloud platform to which node v belongs. This is a function to determine cross-cloud edges; it returns 1 if the edge is in a different cloud, and 0 if it is in the same cloud. This represents a directed edge from node u to node v;

[0211] When correcting path weights, the number of cross-cloud edges in the path is counted. The corrected weight value is calculated based on the original path weight and the number of cross-cloud edges: the original weight is multiplied by an exponential function with a decay coefficient (0.3), where the exponent is the negative of the number of cross-cloud edges. This calculation method makes the weight decay more significant for paths with more cross-cloud edges, reflecting the difficulty of cross-cloud attack propagation.

[0212] Cross-cloud path correction:

[0213] ;

[0214] ;

[0215] in, The number of cloud edges crossed by the path. Trans-cloud attenuation factor ( ), The corrected path weights, This represents a complete path. It is a natural exponential function;

[0216] S430, Threat Impact Quantification: Calculate the threat impact value of service nodes and generate an impact matrix;

[0217] Calculate the threat impact value for each service node. Obtain the node's graph neural network threat score as the base impact value. Consider the node's position in all attack paths: for each path containing the node, divide the path's adjusted weight by the shortest distance from the node to the path's starting point to obtain the path's contribution to the node's impact. Multiply the impact contributions of all paths by the impact propagation attenuation factor (0.7), sum them, and add this sum to the base impact value to obtain the node's final threat impact value. This calculation process reflects the attenuation characteristic of threat impact with propagation distance.

[0218] Node impact value:

[0219] ;

[0220] in, To influence the propagation attenuation factor ( ), For nodes To the starting point of the path The shortest distance, The influence value of node v. Let the threat score be that of node v. For all paths containing node v, This is the starting node of the path;

[0221] Arrange the threat impact values ​​of all service nodes in order of node number to form an impact matrix. Each element of the matrix corresponds to the threat impact value of a service node, and the dimension of the matrix is ​​equal to the total number of nodes in the source graph.

[0222] Threat Impact Matrix:

[0223] ;

[0224] in, For nodes The impact value, For nodes The impact value, This represents the total number of nodes in the source graph. For the set of nodes in the source graph, Threat impact matrix;

[0225] S440, Root Cause Service Location: Locating the source of an attack based on the impact matrix;

[0226] When filtering the root cause candidate node set, the maximum impact value among all nodes is identified, and this value is multiplied by the root cause determination coefficient (set to 0.9) to serve as the filtering threshold. Nodes with impact values ​​greater than this threshold are selected as candidate root cause nodes. This filtering method ensures that only nodes with impact values ​​close to the maximum value are considered as potential attack sources.

[0227] Root cause candidate set:

[0228] ;

[0229] in, The root cause determination coefficient ( ), For the root cause candidate node set, For nodes The impact value, For nodes The impact value, For the set of nodes in the source graph, The maximum influence value among all nodes;

[0230] When selecting the final root cause node from the candidate root cause nodes, a comprehensive score is calculated for each candidate node: the node's graph neural network threat score is multiplied by its out-degree in the source graph. The out-degree represents the number of service calls initiated by the node. The node with the highest comprehensive score is selected as the final root cause node. This selection method considers both the threat level and the scope of the node's impact.

[0231] Ultimate Root Cause Selection:

[0232] ;

[0233] in, For nodes exist The degree of exit in the middle, The final selected root cause node, For nodes Threat score, argmax is the set of root cause candidate nodes, and argmax is the parameter that returns the maximum value of the objective function.

[0234] Based on the attack source output by S440, the following actions were taken:

[0235] 1) Root cause analysis report generation

[0236] Root Cause Extraction Service Detailed information:

[0237] Related logs (aggregation results of S100);

[0238] Attack type (S340 threat event tag);

[0239] Cross-cloud path (S420 corrected path) );

[0240] Output format: JSON report (including service ID, impact value) Key Evidence Log ID).

[0241] 2) Automatic response strategy execution

[0242] Isolation: Disabling when calling cloud platform APIs External access permissions.

[0243] repair:

[0244] If there is a configuration error: Push the Ansible repair script to CMDB.

[0245] It should be noted that Ansible is an automation tool used for batch configuration management, application deployment, etc. Here, it refers to automatically fixing vulnerabilities or misconfigurations using Ansible Playbook scripts.

[0246] CMDB stands for Configuration Management Database, a database that stores IT infrastructure configuration items (such as servers and service relationships). Here, it refers to pushing the fix script to the server node associated with the CMDB.

[0247] If it is a vulnerability attack: trigger WAF rule updates (such as Cloud Flare).

[0248] It should be noted that WAF rules refer to security policies within a Web Application Firewall, used to proactively defend against attacks targeting web services.

[0249] Alert: Send root cause analysis results to the SOC platform via Syslog.

[0250] 3) Dynamic updates to the knowledge base: Adding new entries to the threat knowledge base table;

[0251] like Figure 2 As shown, a multi-cloud environment log intelligent aggregation and threat tracing analysis system includes the following modules:

[0252] Module 1: Log Intelligent Aggregation Module 101

[0253] Raw log collection: Collects heterogeneous raw logs from multiple cloud platforms, supporting mainstream formats (JSON, text, Syslog).

[0254] Identifier Injection and Resolution: Inject a unified service identifier (Service-ID) and cloud platform tag (cloud_id) into each log entry, and parse the log content into structured data (timestamp, operation type, error code, etc.).

[0255] Time-series sorting and completion: Sort log entries by timestamp, detect and complete missing entries (e.g., through interpolation) to ensure the log sequence is complete and continuous.

[0256] Output structured logs: Generate standardized log datasets to provide a unified input for subsequent analysis.

[0257] Module 2: Feature Fusion and Graph Construction Module 102

[0258] Service dependency graph construction: A directed graph is constructed based on service call relationships, where nodes represent services, edges represent call chains, and weights quantify call frequency and response latency.

[0259] Multi-dimensional feature extraction: Extract semantic features (such as operation keywords) and statistical features (such as log frequency and anomaly count) from the logs to form an initial feature vector.

[0260] Feature standardization and fusion: Features are Z-score standardized to eliminate dimensional differences, and log features and graph structures are fused to generate a unified feature matrix.

[0261] Output Feature Matrix and Dependency Graph: Provides a standard feature matrix and service dependency graph as input for threat detection.

[0262] Module 3: Cascaded Integrated Threat Detection Module 103

[0263] Temporal-semantic modeling: The LSTM-Transformer model is used to analyze the temporal patterns and semantic associations of logs and output a preliminary threat score for each log.

[0264] Candidate threat screening: High-confidence candidate threat logs are filtered based on thresholds to reduce false alarms.

[0265] Graph structure verification: Threat propagation paths are verified on the service dependency graph using a graph neural network (GNN), and the judgment results are optimized by combining node threat scores.

[0266] Generate source graph and event set: Output the final set of threat events and source graph (including threat nodes and propagation edges).

[0267] Module 4: Service Identifier-Driven Origin Tracing Module 104

[0268] Cross-cloud attack path extraction: Identify attack paths with service identifiers as nodes in the source map and mark the propagation links across cloud platforms (cloud_id changes).

[0269] Path weight adjustment: Adjust path weights based on cross-cloud labels (e.g., increase cross-cloud hop penalty) to quantify the propagation cost in multi-cloud environments.

[0270] Threat impact quantification: Calculate the impact of threats on service nodes, generate a threat impact matrix, and assess the scope of attack spread.

[0271] Root cause identification and output: Locate the service node that is the source of the attack, output the root cause ID and the impact matrix, and support rapid response decision-making.

[0272] The embodiments of the present invention have been described above, but the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention, all of which are within the protection scope of the present invention.

Claims

1. A method for log intelligent aggregation and threat tracing analysis of multi-cloud environment, characterized in that, Includes the following steps: S100, intelligent log aggregation: collects raw logs from multi-cloud environments, injects unified service identifiers and cloud platform tags, parses heterogeneous log formats into structured data, sorts them by timestamps, and completes missing log entries. Specifically: S110, Multi-protocol Dynamic Acquisition: Uses a log type determination function to analyze the format type of each raw log, and calls the corresponding parser to process the log based on the log type, generating parsed structured data; S120, Non-standard field completion: Detects missing standard fields in the parsed log, uses a rule engine to complete the missing fields from external documents, and adds the completed fields and corresponding values ​​to the parsed structured log; S130, Unified Semantic Mapping: Converts the enhanced logs into a standardized structure format and assigns credibility weights to fields from different sources; S140, Log Stream Aggregation: Arrange all standardized structures in timestamp order to generate an ordered log stream, and verify whether the success rate of log parsing meets the quality requirements. S200, Feature Fusion and Graph Construction: Construct a service dependency graph, quantify the weight of call relationships between services, fuse log semantic features and statistical features, and generate a feature matrix after standardization; Specifically: S210, Multimodal Feature Extraction: Extracting three types of anti-interference features from each log entry: Time series Feature: The frequency of occurrence of a specific event within a statistical time window; Semantic features: Using the BERT model to generate embedding vectors for threat keywords; Graph features: Calculate inter-service dependency weights based on the number of service calls; S220, Feature Matrix Construction: Concatenate the feature vectors of all logs row by row to construct the initial feature matrix; S230, Noise Sample Filtering: Use an isolated forest to calculate the anomaly score for each sample, filter out anomaly samples based on a threshold, and obtain the cleaned feature matrix; S240, Feature Standardization: The filtered feature matrix is ​​standardized to make features of different dimensions comparable. S300, Cascaded Integrated Threat Detection: Uses the LSTM-Transformer model to analyze the time-series and semantic features of logs, outputs a preliminary threat score, verifies the threat propagation path in the service dependency graph through a graph neural network, and generates a source tracing graph and a set of threat events; Specifically: S310, Temporal-Semantic Feature Modeling: Input the feature matrix output by S200 for temporal modeling, use LSTM to capture long-term temporal dependencies, generate hidden state sequences, enhance semantic features through Transformer to obtain semantic vectors, and predict the preliminary threat score for each log based on the semantic vectors; S320, Candidate Threat Filtering: Based on the comparison between the initial threat score and the threshold, logs that are higher than the threshold are filtered out to obtain a set of candidate threats; S330, Graph Structure Validation: Construct a service dependency subgraph based on the candidate threat log set, apply GAT for message passing, update node representations, and calculate the threat score of each node through MLP; S340, Threat Determination and Source Graph Generation: Combine LSTM scores and GNN scores for dual verification to determine the final set of threat events, construct a source graph based on the threat events, and retain relevant nodes and edges; S400, Service Identifier-Driven Source Tracing: Extracts cross-cloud attack paths from the source map, corrects path weights by combining multi-cloud labels, quantifies the impact of threats on each service node, locates the root cause service node, and outputs the impact matrix. Specifically: S410, Attack Path Extraction: In the source map, search for the propagation path starting from the threat event; S420, Cross-Cloud Threat Propagation Analysis: Identify cross-cloud attack paths and make judgments using cloud platform tags in S100; S430, Threat Impact Quantification: Calculate the threat impact value of service nodes and generate an impact matrix; S440, Root Cause Service Location: Locating the source of an attack based on the impact matrix.

2. The method as claimed in claim 1, wherein, In S310, the semantic vector is passed through a linear transformation layer, and the output value is compressed to between 0 and 1 using the Sigmoid function to obtain the preliminary threat score of the log. The closer the preliminary threat score is to 1, the higher the probability of a threat.

3. The method as claimed in claim 1, wherein, In S340, the analysis results of the LSTM model and the GNN model are combined to make the final threat determination. For each log record, it will only be judged as a real threat event if its LSTM threat score exceeds 0.8 and the corresponding service node's GNN threat score exceeds 0.

9.

4. The method of claim 1, wherein, The steps for extracting the attack path are as follows: The path is defined as a sequence of nodes in the source graph, where each sequence consists of multiple consecutive service nodes, and there are direct call relationships between the nodes; Calculate the weight value for each path; The weight threshold for each path is set to 0.6, and paths with a weight greater than this threshold are selected as valid attack paths.

5. The method for intelligent log aggregation and threat tracing analysis in a multi-cloud environment according to claim 1, characterized in that, The steps for quantifying the impact of threats are as follows: Calculate the threat impact value of each service node, arrange the threat impact values ​​of all service nodes in order of node number to form an impact matrix, where each element of the matrix corresponds to the threat impact value of a service node, and the dimension of the matrix is ​​equal to the total number of nodes in the source graph.

6. A system for intelligent aggregation of logs and threat forensics analysis in a multi-cloud environment, the system comprising: The analysis is performed using the steps of the intelligent log aggregation and threat tracing analysis method for a multi-cloud environment as described in any one of claims 1-5, including: Log intelligent aggregation module: collects raw logs from multi-cloud environments, injects unified service identifiers and cloud platform tags, parses heterogeneous log formats into structured data, sorts them by timestamps, and completes missing log entries; Feature fusion and graph construction module: Constructs a service dependency graph, quantifies the weight of call relationships between services, fuses log semantic features and statistical features, and generates a feature matrix after standardization; Cascaded integrated threat detection module: Uses LSTM-Transformer model to analyze log time-series and semantic features, outputs preliminary threat score, verifies threat propagation path in service dependency graph through graph neural network, and generates source tracing graph and threat event set; Service Identifier-Driven Source Tracing Module: Extracts cross-cloud attack paths from the source map, corrects path weights by combining multi-cloud labels, quantifies the impact of threats on each service node, locates the root cause service node, and outputs the impact matrix.

Images