Anomaly Detection Method and System for Streaming Traceability Graph Based on Iterative Prediction and Correction

Abstract

This solution provides an anomaly detection method and system for streaming traceability graphs based on iterative prediction correction, including the steps: S1: Collect streaming data; S2: Sample and encode the encoding information of each node in the streaming traceability graph based on the streaming data; S3: Predict abnormal nodes based on the encoding information. This solution uses an adjacency list to cache streaming events and a sliding time window mechanism to update the adjacency list. It uses a Gaussian mixture model to predict the predicted node state of a node and then corrects it with the current node state. Based on the event frequency as the weight of neighbor nodes, weighted aggregation is used to obtain the encoding information of the current node. Based on the encoding information, abnormal points are judged, and real-time detection is implemented in a streaming manner through a streaming graph neural network, solving the problems of system scalability and detection accuracy.

Description

Technical Field

[0001] This application relates to the field of security detection, and particularly to an anomaly detection method and system for a streaming traceability graph based on iterative prediction correction. Background Art

[0002] P-EDR (Provenance-Based EDR) is a new generation of endpoint detection and response system based on attack traceability technology. Its core lies in achieving in-depth analysis and precise response to attack behaviors by constructing an event causality graph (traceability graph). Its unique advantage compared with traditional endpoint detection and response systems (Endpoint Detection and Response) is that it can reconstruct the dependency relationship between logs and alarms during the detection and investigation steps to achieve causal analysis. In the field of computer security, causal analysis is a complex process used to identify and understand the causal relationships between security events (such as network attacks or system anomalies), which includes analyzing the root causes of events, how they trigger other events, and how these events and behaviors interact and affect the security and stability of the entire system. Further, the traceability graph, which is the core technology in P-EDR, can form a visual attack path by recording the causal dependencies of terminal behavior events (such as file operations, process calls, network communications, etc.) and thus achieve traceability analysis. Traceability analysis refers to the process of tracing the sources and histories of data, files, processes, and operations in a computer system or network, including but not limited to performing traceability analysis in distributed or cluster systems. This analysis is crucial for understanding the background and impact of security events (such as malware attacks or data breaches).

[0003] The P-EDR system can effectively enhance the capabilities of traditional EDR systems and become a core defense tool against advanced network attacks. Especially when effectively solving complex scenarios such as fileless attacks and living-off-the-land attacks, P-EDR has demonstrated great advantages. Although P-EDR has been widely applied in the industry, problems such as "limited computing resources vs. complex computing tasks", "limited adaptability vs. diverse attack scenarios", and "difficulty in interpreting results vs. alert fatigue issues" still greatly limit its effectiveness. Specifically, during the process of real-time traceability analysis, since the dynamic traceability graph is collected at high speed in the form of a data stream of quadruples, and the current P-EDR system cannot handle detecting long-term slow attacks and covert attacks with limited memory overhead, and lacks interpretability for fine-grained attack detection, all of which limit its effectiveness in quickly locating anomalies in the actual production process. Summary of the Invention

[0004] The embodiments of the present application provide an anomaly detection method and system for a streaming traceability graph based on iterative prediction correction, which are implemented by a streaming graph neural network in a stream processing manner to solve the problems of system scalability and detection accuracy.

[0005] In a first aspect, the embodiments of the present application provide an anomaly detection method for a streaming traceability graph based on iterative prediction correction, including the following steps:

[0006] S1: Collect streaming data:

[0007] Collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include quadruples including a subject, an object, an operation, and a timestamp;

[0008] S2: Sample and encode the encoding information of each node of the streaming traceability graph based on the streaming data:

[0009] S21: Set an adjacency list to cache the streaming events obtained in real time and slide the time window of the adjacency list at intervals of a set time period;

[0010] S22: Whenever the time of the streaming event exceeds the time of the time window, retrieve the streaming traceability subgraph in the adjacency list of the current time window, use a message passing function in the streaming traceability subgraph to update the node state of the corresponding node by passing information, predict the predicted node state of each node based on the Gaussian mixture model, and fuse the predicted node state and the current node state to obtain the corrected node state of each node; Assign weights to each node based on the occurrence probability of each streaming event, and perform weighted aggregation encoding on the node state of each node to obtain the encoding information of each node;

[0011] S3: Predict abnormal nodes based on the encoding information:

[0012] Use the encoding information of each node as the input of the decoder to predict the anomaly value of each node. If the anomaly value exceeds the threshold, determine that the current node is an abnormal node.

[0013] In a second aspect, the embodiments of the present application provide an anomaly detection system for a streaming traceability graph based on iterative prediction correction, including:

[0014] A streaming data collection unit: used to collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include quadruples including a subject, an object, an operation, and a timestamp;

[0015] A sampling embedding unit is used to set an adjacency list to cache the real-time acquired streaming events and slide the time window of the adjacency list at intervals of a set time period. The main body of the streaming event serves as a node of the streaming traceability graph, and the linked list of each node records the objects and operations associated with the current streaming event. Whenever the time of the streaming event exceeds the time of the time window, the streaming traceability sub-graph in the adjacency list of the current time window is retrieved, information is passed in the streaming traceability sub-graph using a message passing function and the node status of the corresponding node is updated. The predicted node status of each node is predicted based on the Gaussian mixture model, and the corrected node status of each node is obtained by fusing the predicted node status and the current node status. Weights are assigned to each node based on the occurrence probability of each streaming event, and the node status of each node is weighted and aggregated and encoded to obtain the encoded information of each node. And key-value pairs are constructed with the encoded information and the update time as values and the current node as the key.

[0016] An anomaly detection unit is used to predict anomaly nodes based on the encoded information: taking the encoded information of each node as the input of a decoder to predict the anomaly value of each node. If the anomaly value exceeds the threshold, the current node is determined to be an anomaly node.

[0017] The main contributions and innovations of the present invention are as follows:

[0018] 1. The anomaly detection method for the streaming traceability graph based on iterative prediction and correction provided by this solution uses a Gaussian model to predict the node status, and corrects the node status of each node based on the predicted node status to ensure that the entity status can still be accurately updated in a high-throughput streaming environment.

[0019] 2. The anomaly detection method for the streaming traceability graph based on iterative prediction and correction provided by this solution introduces the event frequency into the aggregation embedding of the encoded information to take into account the influence of historical data on the encoded information.

[0020] 3. After obtaining the anomaly value of each node using the decoder, the anomaly detection method for the streaming traceability graph based on iterative prediction and correction provided by this solution can determine the survival time of the node based on the anomaly value, which can not only avoid the context loss caused by premature recycling, but also prevent data redundancy and storage pressure caused by staying in memory all the time.

[0021] 4. The anomaly detection method for the streaming traceability graph based on iterative prediction and correction provided by this solution uses an adjacency list to cache the streaming events, and adopts a sliding time window mechanism to update the adjacency list to reduce the memory occupancy consumption.

[0022] The details of one or more embodiments of this application are set forth in the following drawings and description to make the other features, objects, and advantages of this application more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0024] Figure 1 is a schematic flowchart of an anomaly detection method for a streaming traceability graph based on iterative prediction correction according to an embodiment of the present application.

[0025] Figure 2 is a schematic framework diagram of an anomaly detection method for a streaming traceability graph based on iterative prediction correction according to an embodiment of the present application.

[0026] Figure 3 is a logical diagram of streaming anomaly detection for an anomaly detection method for a streaming traceability graph based on iterative prediction correction according to an embodiment of the present application.

[0027] Figure 4 is a schematic framework diagram of an anomaly detection system for a streaming traceability graph based on iterative prediction correction according to an embodiment of the present application.

[0028] Figure 5 is a schematic hardware structure diagram of an electronic device according to an embodiment of the present application. Detailed Embodiments

[0029] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0030] It should be noted that: in other embodiments, the steps of the corresponding methods are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0031] Embodiment 1

[0032] As Figure 1 shown, this solution provides an anomaly detection method for a streaming traceability graph based on iterative prediction correction, including the following steps:

[0033] S1: Collect streaming data:

[0034] Collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a quadruple consisting of a subject, an object, an operation, and a timestamp;

[0035] S2: Sample and encode the encoding information of each node of the streaming traceability graph based on the streaming data:

[0036] S21: Set up an adjacency list to cache the streaming events obtained in real time and slide the time window of the adjacency list at regular intervals;

[0037] S22: Whenever the time of the streaming event exceeds the time of the time window, retrieve the streaming traceability subgraph in the adjacency list of the current time window, use the message passing function in the streaming traceability subgraph to update the node state of the corresponding node, predict the predicted node state of each node based on the Gaussian mixture model, and fuse the predicted node state and the current node state to obtain the corrected node state of each node; Assign weights to each node based on the occurrence probability of each streaming event, perform weighted aggregation encoding on the node state of each node to obtain the encoding information of each node, and construct key-value pairs with the encoding information and the update time as values and the current node as the key;

[0038] S3: Predict abnormal nodes based on the encoding information:

[0039] Take the encoding information of each node as the input of the decoder to predict the abnormal value of each node. If the abnormal value exceeds the threshold, it is determined that the current node is an abnormal node.

[0040] Figure 2 It is the overall framework of the abnormal detection method of the streaming traceability graph based on iterative prediction and correction in this solution. First, collect streaming data, standardize the streaming data into streaming events, and cache the streaming events in the form of an adjacency list, and use the sliding time window mechanism to slide and update the adjacency list. Based on the updated adjacency list, perform iterative prediction sampling and correction to update the encoding information of each node; Furthermore, input the encoding information into the decoder for decoding to predict the abnormal value, calculate the association or survival time of the abnormal node based on the abnormal value, and then reverse the nodes in the adjacency list based on the survival time.

[0041] In step S1 of collecting streaming data:

[0042] The streaming data is the system behavior log and is collected through underlying tools. Considering that the data formats of the streaming data collected by different underlying tools are not unified and will contain a lot of redundant information and duplicate information, therefore, in this solution, after the streaming data is collected in real time, the streaming data is preprocessed, and the preprocessed streaming data is standardized into streaming events. The means of preprocessing are selected from removing redundant information and duplicate logs in the streaming data. The redundant information includes irrelevant fields such as reserved fields and thread information, and the duplicate logs refer to the same logs with different timestamps.

[0043] In some embodiments, the underlying tools are tools built based on eBPF, LTTng, Kernel Module, Event Tracing for Windows, etc.

[0044] In some embodiments, the streaming data includes four major categories of system behavior logs, including process event logs, file event logs, container event logs, and network event logs. After the subject, object, operation, and timestamp in the system behavior logs are extracted respectively, they are used as streaming events. Of course, the specific system behavior logs can also be extended according to actual needs. Exemplarily, the streaming event corresponding to the streaming data of a certain process event log is: (ProcessA, / bin / bash, "execve", 2024-06-01 10:00:00).

[0045] The system behavior logs supported by this solution are shown in Table 1 below:

[0046] Table 1 Supported System Behavior Logs

[0047] 。

[0048] In step S21: This solution selects an adjacency list to cache the obtained streaming events. The adjacency list is a combination of multiple linked lists. The subject of the streaming event is used as the node of the streaming traceability graph. A linked list is constructed for each node in the adjacency list, and the linked list of each node records the object and operation associated with the current streaming event. This graph data structure can clearly represent the relationship between nodes and provide a basis for subsequent graph analysis and anomaly detection.

[0049] Exemplarily, for node 1 in the streaming traceability graph, there are two edges, corresponding to node 2 and node 4 respectively. Then, there are also two connected points in the linked list of the current node 1, which are node 2 and node 4 respectively.

[0050] It should be noted that the time of the streaming event is stored on the edge of the streaming traceability graph corresponding to the adjacency list, and the event frequency of the streaming event is stored in the event frequency database. The event frequency in the event frequency database is obtained based on offline data statistics. Generally, when the number of streaming events is large, it means that the value representing the event frequency tends to be stable. Whenever a new streaming event is obtained, the streaming event is added to the head or tail of the linked list of the adjacency list of the subject node based on the subject and object of the streaming event. In other words, the adjacency list is updated according to the streaming events obtained in real time.

[0051] The reason for adopting the adjacency list in this solution is that the adjacency list only stores the information directly associated with the node, and can quickly find all events related to a certain node, which is convenient for subsequent message passing and anomaly detection. In addition, since streaming events are generated at an extremely high speed, if all streaming events are stored in full, it will cause memory overflow. Therefore, this solution uses a sliding window mechanism to retain the streaming events in the recent time period, so as to control the memory occupancy while ensuring real-time performance.

[0052] This solution sets the time window of the adjacency list to slide every set time period, and updates the streaming events when the time window of the adjacency list does not slide, so as to achieve memory optimization and real-time guarantee through the constraint of the time dimension. Further, the length of the time window is determined according to the attack characteristics. Among them, the short window is suitable for rapidly erupting attacks (such as DDos), and the long window is suitable for APT attack chain analysis. Every set time period, the time window of the linked list of the adjacency list moves forward, and the length of the time window does not change. During the movement of the time window, the old events outside the time window range will be deleted, and only the streaming events within the time period of the recent time window will be retained.

[0053] Since this solution adopts a sliding window mechanism, it can retain only the events in the recent time period, greatly reducing the memory occupancy, avoiding the problem of memory overflow, and focusing on the latest attack signs, reducing the interference of historical noise.

[0054] In step S22, this solution uses predictive and corrective sampling to address the problem of time discontinuity in the streaming traceability graph. The Gaussian Mixture Model (GMM) is used to predict the node state and dynamically correct it in combination with actual data to ensure the accuracy of real-time analysis. In the actual analysis process, streaming events may be inconsistent in timestamp order and actual occurrence order due to network latency, processing batch division, etc.; and multiple operations of the same subject and object may also be overwritten by subsequent events, resulting in the loss of historical states and thus the problem of time discontinuity. Once the problem of time discontinuity occurs, it will affect the accuracy of anomaly detection. Therefore, this solution uses the GMM model to predict the node state of the events that have not been processed in time, and fuses the predicted value and the actual value to achieve the correction of the node state.

[0055] It should be noted that in this solution, predictions are not made immediately for each streaming event obtained. Instead, after a period of time, that is, when the time of the streaming event exceeds the event length of the time window, a batch update is performed using the previously stored state in the adjacency list and the newly obtained state.

[0056] Furthermore, in this solution, by using a message passing function to pass information for each streaming event, the correlation information between nodes can be effectively propagated. Since streaming events are continuously generated, the relationships between various entities (nodes) in the system are also dynamically changing. The role of the message passing function is to convert the dependency relationship of the quadruple streaming event (such as process A calling file B) into a feature representation (embedding vector) of the node, so as to reflect the dynamic changes of the system in real time to support real-time anomaly detection.

[0057] Furthermore, in the step of "using the message passing function to pass information and update the node states of the corresponding nodes", whenever the sampling embedding condition is met, the message passing function is used to pass the information of the streaming event to update the node states of the subject and object of the corresponding streaming event. In some embodiments, the sampling embedding condition is that the streaming event time of the received streaming event just triggers the sliding operation of the time window.

[0058] In addition, since after events in the same batch enter the current time window of the adjacency list, events such as <a, b, connect, t1> and <a, b, connect, t2> with only different time information will appear, and the event at time t2 will overwrite the event at time t1, resulting in discontinuity of the events. Therefore, this solution introduces a prediction and correction mechanism to alleviate the discontinuity of the events.

[0059] Specifically, this solution selects a Gaussian mixture model for predicting node states. The Gaussian mixture model (GMM) is a probability model that can model complex data distributions. Through this model, the historical state information of the node can be used to predict the future state, and then correct the current node state to improve the accuracy of subsequent anomaly detection.

[0060] Furthermore, collect the historical node states of the nodes, where the historical node states are the feature vectors of the nodes in a previous period of time, such as the resource usage of the nodes, the interaction frequency with other nodes, etc. These feature vectors constitute the dataset for training the Gaussian mixture model, and a Gaussian mixture model is constructed based on the historical node states. The formula for constructing the Gaussian mixture model is as follows:

[0061] ;

[0062] where ωRepresents the number of components, usually set to 2, corresponding to two types of events (positive event: "there is interaction"; negative event "no interaction"). α j Represents the proportion of different event types, initialized to a uniform distribution, and updated through statistical distribution subsequently. Are the mean and covariance. The mean represents the average impact of event type h on the change of the memory state of node i, and the covariance represents the volatility of the change of event type h on the memory state of node i.

[0063] Furthermore, in the step of "predicting the predicted node state of each node based on the Gaussian mixture model", the historical node states of each node before the current moment are input into the Gaussian mixture model for prediction to obtain the predicted node state.

[0064] In the step of "fusing the predicted node state and the current node state to obtain the corrected node state of each node", the predicted node state is weighted and added to the current node state to obtain the corrected node state of each node.

[0065] The corresponding formula is as follows:

[0066] Si’’=(1 - x)*Si’+x*Si;

[0067] Where Si’ is the predicted node state, Si is the current node state, Si’’ is the corrected node state, and x is a value between 0 and 1, determined according to the data situation of the streaming traceability subgraph. When the data situation in the streaming traceability subgraph is low noise and stable data, the value of x is close to 1, representing more trust in the current node state; when the data situation in the traceability graph is high noise and unstable data, the value of x is close to 0, representing more trust in the predicted node state.

[0068] It should be noted that the data situation of the streaming traceability subgraph refers to that the states of nodes do not change significantly in different time periods. If there is no significant change, the data is considered stable.

[0069] Furthermore, in the step of "assigning weights to each node based on the occurrence probability of each streaming event", the event frequency in the event frequency database is obtained as the occurrence probability of the current streaming event, and the difference between 1 and the occurrence probability is taken as the weight of the node involved in the current streaming event. The event frequency database is a database that statistically calculates the occurrence probability of streaming events based on historical data, records the possible occurrence probability of each streaming event, and thus can predict the probability of new streaming events based on the frequency of historical streaming events of the same type.

[0070] In the step of "performing weighted aggregation encoding on the node status of each node to obtain the encoding information of each node", all associated neighbor nodes and operations corresponding to the objects are obtained from the adjacency list of the node corresponding to the main body of the streaming event, the information of the neighbor nodes is encoded to obtain neighbor messages, and the neighbor information of all neighbor nodes is aggregated according to the corresponding weights to obtain the encoding information of the current main body node.

[0071] In other words, for the feature information of each node, it not only includes its own information but also the information of its surrounding nodes. Therefore, this solution combines the neighbor nodes of the main body node for weighted aggregation to obtain the embedding vector of the current main body node. The aggregation formula is as follows:

[0072] ;

[0073] where Node i represents the neighbor information of the i-th neighbor node, represents the embedding vector of the a-th main body node, represents the occurrence probability of the streaming event between the a-th main body node and the i-th neighbor node.

[0074] In some embodiments, the key-value pairs constructed in this solution use the encoding information and the update time as values, and the current node as the key, and the representation form is: <node i: <status encoding, update time>>.

[0075] In step S3 of predicting abnormal nodes based on the encoding information, a decoder is used to perform abnormal judgment based on the encoding information of each node. In some embodiments, the decoder determines whether the encoding information of the current node matches according to the historical encoding information of the node. If the encoding information of the current node differs more from the historical encoding information, the abnormal value of the current node is greater. In some embodiments, the decoder is MLP, LSTM, etc.

[0076] Furthermore, in order to eliminate expired data to avoid releasing the available memory occupancy overhead, step S3 of this solution further includes: determining the survival time of each node based on the abnormal value and time window of the nodes that have not been determined as abnormal nodes, and eliminating the nodes whose survival time in the key-value pair is less than the set threshold.

[0077] The calculation formula for the survival time is as follows:

[0078] .

[0079] In addition, since the sending of attacks often includes the impact on the information of multiple entities, this solution associates abnormal nodes through the label propagation rule to determine whether there is propagation of abnormal nodes. Furthermore, step S3 of this solution further includes:

[0080] All abnormal nodes are used as initialization labels, and each initialization label is propagated around to find other abnormal nodes on the propagation path. If the distance between the next abnormal node and the current abnormal node on the propagation path exceeds a predetermined distance threshold, the label will no longer be propagated.

[0081] As described above, this solution provides a complete set of abnormal detection methods for the streaming traceability graph. To more clearly show the streaming abnormal detection process of the abnormal detection method for the streaming traceability graph, the following uses Figure 3 as the flowchart of the streaming abnormal detection for introduction. It can be seen from Figure 3 that when the time window of the adjacency list does not slide, the collected streaming events are cached into the adjacency list. At the same time, the time window of the adjacency list slides every set time period, and the streaming events in the current time window are encoded during the sliding of the time window; it is judged whether the event time exceeds the time window. If not, the traceability subgraph corresponding to the adjacency list of the time window is obtained. If so, the streaming traceability subgraph corresponding to the adjacency list of the corresponding time window is queried and the streaming traceability subgraph is corrected and sampled; the event frequency database is queried, and the weight of each node is constructed based on the event frequency. Furthermore, the node state of each node is weighted and aggregated and encoded to obtain the encoding information of each node. At the same time, the key-value pair is updated based on the encoding information; it is judged whether the current node is an abnormal point based on the encoding information of each node. If it is an abnormal point, an alarm is generated and the alarm is associated. And regardless of whether it is an abnormal node, the survival time of the node is calculated, and the corresponding node is deleted from the key-value pair based on the survival time.

[0082] Embodiment 2

[0083] Based on the same concept, referring to Figure 4 this application also proposes an abnormal detection system for a streaming traceability graph based on iterative prediction and correction, including:

[0084] Streaming data acquisition unit: used to collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a quadruple consisting of a subject, an object, an operation, and a timestamp;

[0085] Sampling embedding unit, which is used to set an adjacency list to cache the real-time acquired streaming events and slide the time window of the adjacency list at regular intervals. The main body of the streaming events serves as the nodes of the streaming traceability graph, and the linked list of each node records the objects and operations associated with the current streaming event. Information is passed to each streaming event using a message passing function to update the node state of the corresponding node. The predicted node state of each node is predicted based on the Gaussian mixture model, and the corrected node state of each node is obtained by fusing the predicted node state and the current node state. The weight of each node is assigned based on the occurrence probability of each streaming event, and the node states of each node are weighted and aggregated encoded to obtain the encoded information of each node. A key-value pair is constructed with the encoded information and the update time as the value and the current node as the key.

[0086] Anomaly detection unit, which is used to predict anomaly nodes based on the encoded information: The encoded information of each node is used as the input of the decoder to predict the anomaly value of each node. If the anomaly value exceeds the threshold, the current node is determined to be an anomaly node.

[0087] It should be noted that the anomaly detection system is a complete system as a whole. The streaming data sampling unit, the sampling embedding unit, and the anomaly detection unit are only schematic illustrations based on functional modules. The specific technical means of this anomaly detection system are introduced in Example 1.

[0088] Example 3

[0089] This embodiment also provides an electronic device. Refer to Figure 5 , which includes a memory 404 and a processor 402. A computer program is stored in the memory 404, and the processor 402 is configured to run the computer program to execute the steps in any of the above embodiments of the anomaly detection method for the streaming traceability graph based on iterative prediction and correction.

[0090] Specifically, the above processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured as one or more integrated circuits implementing the embodiments of the present application.

[0091] Among them, the memory 404 may include a large-capacity memory 404 for data or instructions. The processor 402 reads and executes the computer program instructions stored in the memory 404 to implement any of the above embodiments of the anomaly detection method for the streaming traceability graph based on iterative prediction and correction.

[0092] Optionally, the above electronic device may further include a transmission device 406 and an input / output device 408. The transmission device 406 is connected to the above-mentioned processor 402, and the input / output device 408 is connected to the above-mentioned processor 402. The transmission device 406 can be used to receive or send data via a network.

[0093] The input / output device 408 is used to input or output information. In this embodiment, the input information may be streaming data, etc., and the output information may be outliers, etc.

[0094] Optionally, in this embodiment, the above-mentioned processor 402 may be configured to perform the following steps by a computer program:

[0095] S1: Collect streaming data:

[0096] Collect streaming data in real time and normalize the streaming data into streaming events, where the streaming events include a quadruple consisting of a subject, an object, an operation, and a timestamp;

[0097] S2: Sample and encode the encoding information of each node of the streaming traceability graph based on the streaming data:

[0098] S21: Set an adjacency list to cache the streaming events obtained in real time and slide the time window of the adjacency list every set time period. The subject of the streaming event serves as a node of the streaming traceability graph, and the linked list of each node records the objects and operations associated with the current streaming event;

[0099] S22: Whenever the time of the streaming event exceeds the time of the time window, retrieve the streaming traceability subgraph in the adjacency list of the current time window, use a message passing function to pass information in the streaming traceability subgraph and update the node state of the corresponding node, predict the predicted node state of each node based on the Gaussian mixture model, and fuse the predicted node state and the current node state to obtain the corrected node state of each node; Assign weights to each node based on the occurrence probability of each streaming event, perform weighted aggregation encoding on the node state of each node to obtain the encoding information of each node, and construct key-value pairs with the encoding information and the update time as values and the current node as the key;

[0100] S3: Predict abnormal nodes based on the encoding information:

[0101] Use the encoding information of each node as the input of the decoder to predict the outlier value of each node. If the outlier value exceeds the threshold, determine that the current node is an abnormal node.

[0102] It should be noted that the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and will not be elaborated here.

[0103] Generally, various embodiments can be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the present invention can be implemented in hardware, while other aspects can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although aspects of the present invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, the blocks, devices, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers, or other computing devices, or some combination thereof.

[0104] Embodiments of the present invention can be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. A computer software or program (also referred to as a program product), including software routines, applets, and / or macros, can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product can include one or more computer-executable components that are configured to perform the embodiments when the program runs. The one or more computer-executable components can be at least one software code or a part thereof. Additionally, in this regard, it should be noted that any block in the logical flow as shown in the figures can represent a program step, or interconnected logical circuits, blocks, and functions, or a combination of program steps and logical circuits, blocks, and functions. The software can be stored on physical media such as memory chips or storage blocks implemented within the processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs. The physical media is a non-transitory medium.

[0105] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as falling within the scope described in this specification.

[0106] The above embodiments only represent several implementation manners of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for detecting anomalies in a streaming traceability graph based on iterative prediction and correction, characterized in that: The following steps are involved: S1: Collect streaming data: Collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple of subject, object, operation and timestamp; S2: Encoding information of each node in the streaming traceability graph based on streaming data sampling encoding: S21: setting an adjacency table to cache streaming events acquired in real time and sliding a time window of the adjacency table at intervals of a set time period; S22: Whenever the time of a streaming event exceeds the time of a time window, the streaming traceability subgraph in the adjacency list of the current time window is retrieved, and the node state of the corresponding node is updated by using a message passing function in the streaming traceability subgraph to pass information, and the predicted node state of each node is predicted based on a Gaussian mixture model, and the predicted node state and the current node state are fused to obtain the corrected node state of each node; a weight is assigned to each node based on the occurrence probability of each streaming event, and the node state of each node is weighted aggregated and encoded to obtain the encoding information of each node; S3: Predict abnormal nodes based on encoded information: The encoded information of each node is used as the input of the decoder to predict the outlier value of each node. If the outlier value exceeds the threshold, the current node is judged as an outlier node.

2. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: The survival time of each node is determined based on the outlier value and time window of each node, and nodes whose survival time in the key-value pair is less than the set threshold are eliminated.

3. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: All abnormal nodes are used as initialization labels, and each initialization label is propagated in all directions to find other abnormal nodes on the propagation path. If the distance between the next abnormal node on the propagation path and the current abnormal node exceeds a predetermined distance threshold, the label will no longer be propagated.

4. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: The subject of the streaming event is the node of the streaming traceability graph. A linked list is constructed for each node in the adjacency list, and the linked list of each node records the object and operation associated with the current streaming event.

5. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: Whenever the sampling embedding condition is met, the message passing function is used to pass the information of the streaming event to update the node status of the subject and object of the corresponding streaming event.

6. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: Collect the historical node status of the node, and build a Gaussian mixture model based on the historical node status. The formula for building the Gaussian mixture model is as follows: ； in ω Indicates the quantity of components, α j Indicates the proportion of different event types. are the mean and covariance, where the mean represents the average impact of event type h on the memory state change of node j, and the covariance represents the volatility of event type h on the memory state change of node j.

7. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: The predicted node state and the current node state are weighted and added together to obtain the corrected node state of each node.

8. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: Based on the adjacency table of the node corresponding to the subject of the streaming event, the neighbor nodes and operations corresponding to all associated objects are obtained, the information of the neighbor nodes is encoded to obtain the neighbor message, and the neighbor information of all neighbor nodes is aggregated according to the corresponding weights to obtain the encoded information of the current subject node.

9. An anomaly detection system for streaming traceability graphs based on iterative prediction and correction, characterized in that: include: Streaming data collection unit: used to collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple including subject, object, operation and timestamp; A sampling embedding unit is used to set an adjacency table to cache streaming events acquired in real time and slide the time window of the adjacency table at set intervals, wherein the subject of the streaming event serves as a node of the streaming traceability graph, and the linked list of each node records the objects and operations associated with the current streaming event; whenever the time of the streaming event exceeds the time of the time window, the streaming traceability subgraph in the adjacency table of the current time window is called, and the message passing function is used in the streaming traceability subgraph to pass information and update the node state of the corresponding node, and the predicted node state of each node is predicted based on the Gaussian mixture model, and the predicted node state and the current node state are fused to obtain the corrected node state of each node; a weight is assigned to each node based on the probability of occurrence of each streaming event, and the node state of each node is weighted and aggregated to obtain the encoding information of each node, and a key-value pair is constructed with the encoding information and the update time as the value and the current node as the key; The anomaly detection unit is used to predict abnormal nodes based on the encoding information: the encoding information of each node is used as the input of the decoder to predict the abnormal value of each node, and if the abnormal value exceeds the threshold, the current node is judged to be an abnormal node.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes an anomaly detection method for a streaming provenance graph based on iterative prediction correction according to any one of claims 1 to 8.

Images