Streaming traceability graph anomaly detection method and system based on iterative prediction correction

Abstract

The invention provides an anomaly detection method and system for a streaming traceability graph based on iterative prediction correction. The anomaly detection method comprises the following steps: S1, collecting streaming data; S2, sampling and coding coding information of each node of the streaming traceability graph based on the streaming data; and S3, predicting an abnormal node based on coding information, adopting an adjacency list to cache a streaming event, adopting a sliding time window mechanism to update the adjacency list, adopting a Gaussian mixture model to predict a predicted node state of the node, and then correcting the predicted node state with a current node state. According to the method, the current node is detected, the coding information of the current node is obtained through weighted aggregation by taking the event frequency as the weight of the neighbor node, the abnormal point is judged based on the coding information, real-time detection is realized by adopting a stream processing mode through the stream graph neural network, and the problems of system scale and detection precision are solved.

Description

Technical Field

[0001] The present application relates to the field of security detection, and in particular to an anomaly detection method and system for a streaming traceability graph based on iterative prediction and correction. Background Art

[0002] P-EDR (Provenance-Based EDR) is a new generation of terminal detection and response system based on attack tracing technology. Its core lies in the in-depth analysis and precise response to attack behaviors by building an event causal relationship map (tracing map). Compared with the traditional terminal detection and response system (Endpoint Detection and Response), its unique advantage is that it can reconstruct the dependency between logs and alarms in the detection and investigation steps to achieve causal analysis. In the field of computer security, causal analysis is a complex process used to identify and understand the causal relationship between security events (such as network attacks or system anomalies), which includes analyzing the root causes of events, how they trigger other events, and how these events and behaviors interact and affect the security and stability of the entire system. Furthermore, the source traceability graph, which is the core technology in P-EDR, can form a visual attack path and realize source traceability analysis by recording the causal dependencies of terminal behavior events (such as file operations, process calls, network communications, etc.). Source traceability analysis refers to the process of tracking the source and history of data, files, processes and operations in a computer system or network, including but not limited to source traceability analysis in distributed or cluster systems. This analysis is crucial to understanding the context and impact of security incidents (such as malware attacks or data leaks).

[0003] The P-EDR system can effectively enhance the capabilities of traditional EDR systems and become a core defense tool for dealing with advanced network attacks. In particular, P-EDR has shown great advantages in effectively solving complex scenarios such as fileless attacks and off-site attacks. Although P-EDR has been widely used in the industry, problems such as "limited computing resources vs. complex computing tasks", "limited adaptability vs. diverse attack scenarios", and "difficult results to explain vs. alarm fatigue" still greatly limit its effectiveness. Specifically, in the process of real-time source tracing analysis, since the dynamic source tracing graph is collected at high speed in the form of a four-tuple data stream, the current P-EDR system cannot cope with the detection of long-term slow attacks and covert attacks under limited memory overhead, and lacks explainability for fine-grained attack detection, which limits its effectiveness in quickly locating anomalies in actual production processes. Summary of the invention

[0004] The embodiments of the present application provide an anomaly detection method and system for a streaming traceability graph based on iterative prediction and correction, which implements real-time detection through a streaming graph neural network using stream processing, thus solving the problems of system scale and detection accuracy.

[0005] In a first aspect, an embodiment of the present application provides an anomaly detection method for a streaming provenance graph based on iterative prediction and correction, comprising the following steps: S1: Collect streaming data: Collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple of subject, object, operation and timestamp; S2: Encoding information of each node in the streaming traceability graph based on streaming data sampling encoding: S21: setting an adjacency table to cache streaming events acquired in real time and sliding a time window of the adjacency table at intervals of a set time period; S22: Whenever the time of a streaming event exceeds the time of a time window, the streaming traceability subgraph in the adjacency list of the current time window is retrieved, and the node state of the corresponding node is updated by using a message passing function in the streaming traceability subgraph to pass information, and the predicted node state of each node is predicted based on a Gaussian mixture model, and the predicted node state and the current node state are fused to obtain the corrected node state of each node; a weight is assigned to each node based on the occurrence probability of each streaming event, and the node state of each node is weighted aggregated and encoded to obtain the encoding information of each node; S3: Predict abnormal nodes based on encoded information: The encoded information of each node is used as the input of the decoder to predict the outlier value of each node. If the outlier value exceeds the threshold, the current node is judged as an outlier node.

[0006] In a second aspect, an embodiment of the present application provides an anomaly detection system for a streaming provenance graph based on iterative prediction and correction, including: Streaming data collection unit: used to collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple including subject, object, operation and timestamp; A sampling embedding unit is used to set an adjacency table to cache streaming events acquired in real time and slide the time window of the adjacency table at set intervals, wherein the subject of the streaming event is used as a node of the streaming traceability graph, and the linked list of each node records the objects and operations associated with the current streaming event; whenever the time of the streaming event exceeds the time of the time window, the streaming traceability subgraph in the adjacency table of the current time window is called, and the message passing function is used in the streaming traceability subgraph to pass information and update the node state of the corresponding node, and the predicted node state of each node is predicted based on the Gaussian mixture model, and the predicted node state and the current node state are fused to obtain the corrected node state of each node; a weight is assigned to each node based on the probability of occurrence of each streaming event, and the node state of each node is weighted and aggregated to obtain the encoding information of each node, and a key-value pair is constructed with the encoding information and the update time as the value and the current node as the key; The anomaly detection unit is used to predict abnormal nodes based on the encoding information: the encoding information of each node is used as the input of the decoder to predict the abnormal value of each node, and if the abnormal value exceeds the threshold, the current node is judged to be an abnormal node.

[0007] The main contributions and innovations of the present invention are as follows: 1. The anomaly detection method of the streaming traceability graph based on iterative prediction and correction provided by this solution adopts a Gaussian model to predict the node state, and corrects the node state of each node based on the predicted node state to ensure that the entity state can still be accurately updated in a high-throughput streaming environment.

[0008] 2. The anomaly detection method based on iterative prediction and correction streaming traceability graph provided by this scheme introduces event frequency into the aggregate embedding of encoded information to take into account the impact of historical data on the encoded information.

[0009] 3. The anomaly detection method of the streaming provenance graph based on iterative prediction correction provided by this solution can determine the survival time of the node based on the anomaly value after obtaining the anomaly value of each node using the decoder, which can avoid the context loss caused by premature recycling and prevent data redundancy and storage pressure caused by always resident in memory.

[0010] 4. The anomaly detection method for streaming traceability graph based on iterative prediction and correction provided by this solution adopts an adjacency list to cache streaming events, and uses a sliding time window mechanism to update the adjacency list to reduce memory consumption.

[0011] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a flow chart of an anomaly detection method of a streaming traceability graph based on iterative prediction and correction according to an embodiment of the present application.

[0013] Figure 2 It is a schematic diagram of a framework of an anomaly detection method of a streaming provenance graph based on iterative prediction and correction according to an embodiment of the present application.

[0014] Figure 3 It is a logic diagram of streaming anomaly detection according to the anomaly detection method of streaming traceability graph based on iterative prediction and correction according to an embodiment of the present application.

[0015] Figure 4 It is a schematic diagram of the framework of an anomaly detection system of a streaming traceability graph based on iterative prediction and correction according to an embodiment of the present application.

[0016] Figure 5 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0018] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0019] Embodiment 1 like Figure 1 As shown, this scheme provides an anomaly detection method for a streaming traceability graph based on iterative prediction and correction, comprising the following steps: S1: Collect streaming data: Collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple of subject, object, operation and timestamp; S2: Encoding information of each node in the streaming traceability graph based on streaming data sampling encoding: S21: setting an adjacency table to cache streaming events acquired in real time and sliding a time window of the adjacency table at intervals of a set time period; S22: Whenever the time of a streaming event exceeds the time of a time window, the streaming traceability subgraph in the adjacency list of the current time window is retrieved, and the message passing function is used to pass information in the streaming traceability subgraph to update the node state of the corresponding node, and the predicted node state of each node is predicted based on the Gaussian mixture model, and the predicted node state and the current node state are fused to obtain the corrected node state of each node; a weight is assigned to each node based on the occurrence probability of each streaming event, and the node state of each node is weighted aggregated and encoded to obtain the encoding information of each node, and a key-value pair is constructed with the encoding information and the update time as the value and the current node as the key; S3: Predict abnormal nodes based on encoded information: The encoded information of each node is used as the input of the decoder to predict the outlier value of each node. If the outlier value exceeds the threshold, the current node is judged as an outlier node.

[0020] Figure 2 This is the overall framework of the anomaly detection method of the streaming traceability graph based on iterative prediction and correction in this scheme. First, the streaming data is collected and standardized into streaming events and cached in the form of an adjacency list. The adjacency list is updated using a sliding time window mechanism. Iterative prediction sampling and correction are performed based on the updated adjacency table to update the encoding information of each node. The encoded information is then input into the decoder for decoding to predict outliers. The association of abnormal nodes or the calculation of survival time are performed based on the outliers. The nodes in the adjacency list can then be reversely eliminated based on the survival time.

[0021] In step S1 of collecting streaming data: Streaming data is a system behavior log and is collected through underlying tools. Considering that the data format of streaming data collected by different underlying tools is not uniform and may contain a lot of redundant and repeated information, this solution collects streaming data in real time and preprocesses the streaming data, and standardizes the preprocessed streaming data into streaming events, wherein the preprocessing means is selected from removing redundant information and duplicate logs in the streaming data, wherein the redundant information includes reserved fields and irrelevant fields including thread information, and duplicate logs refer to the same logs with different timestamps.

[0022] In some embodiments, the underlying tool is a tool built on eBPF, LTTng, Kernel Module, Event Tracing for Windows, etc.

[0023] In some embodiments, the streaming data includes four major types of system behavior logs, including process event logs, file event logs, container event logs, and network event logs. The subject, object, operation, and timestamp in the system behavior log are extracted as streaming events. Of course, the specific system behavior log can also be expanded according to actual needs. For example, the streaming event corresponding to the streaming data of a process event log is: (ProcessA, / bin / bash, "execve", 2024-06-01 10:00:00).

[0024] The system behavior logs supported by this solution are shown in Table 1 below: Table 1 Supported system behavior logs .

[0025] In step S21: This solution selects an adjacency list to cache the acquired streaming events, wherein the adjacency list is a combination of multiple linked lists, the subject of the streaming event serves as a node of the streaming traceability graph, a linked list is constructed for each node in the adjacency list, and the linked list of each node records the objects and operations associated with the current streaming event. This graph data structure can clearly represent the relationship between nodes, providing a basis for subsequent graph analysis and anomaly detection.

[0026] For example, for node 1, there are two edges in the streaming traceability graph, corresponding to node 2 and node 4 respectively. Then, there are also two connected points in the linked list of the corresponding current node 1, namely node 2 and node 4.

[0027] It should be noted that the time of the streaming event is stored on the edge of the streaming traceability graph corresponding to the adjacency list, and the event frequency of the streaming event is stored in the event frequency database. The event frequency in the event frequency database is obtained based on offline data statistics. It is generally believed that when the number of streaming events is large, the numerical value of the event frequency tends to be stable. Whenever a new streaming event is obtained, the streaming event is added to the head or tail of the linked list of the adjacency list of the subject node based on the subject and object of the streaming event. In other words, the adjacency list is updated according to the streaming events obtained in real time.

[0028] The reason why this solution uses the adjacency list is that it only stores information directly related to the node, and can quickly find all events related to a node, which is convenient for subsequent message transmission and anomaly detection. In addition, since streaming events are generated at a very high speed, if the streaming events are stored in full, it will cause memory overflow. Therefore, this solution uses a sliding window mechanism to retain streaming events in the most recent time period, thereby ensuring real-time performance while controlling memory usage.

[0029] This solution sets a time window for sliding the adjacency table at intervals of a set time period, and updates streaming events when the time window of the adjacency table does not slide, achieving memory optimization and real-time guarantee through the constraints of the time dimension. Furthermore, the length of the time window is determined according to the characteristics of the attack, where a short window is suitable for fast-burst attacks (such as DDos), and a long window is suitable for APT attack chain analysis. The time window of the linked list of the adjacency table is moved forward at intervals of a set time period, and the length of the time window does not change. In the process of moving the time window, old events that exceed the time window range will be deleted, and only streaming events within the time period of the most recent time window will be retained.

[0030] Since this solution adopts a sliding window mechanism, it can only retain events in the most recent time period, greatly reducing memory usage and avoiding memory overflow problems. It can also focus on the latest signs of attack and reduce the interference of historical noise.

[0031] In step S22, this solution uses prediction and correction sampling to address the time discontinuity problem of the streaming traceability graph. The node status is predicted by the Gaussian mixture model (GMM) and dynamically corrected in combination with actual data to ensure the accuracy of real-time analysis. In the actual analysis process, streaming events may cause inconsistencies between the timestamp order and the actual occurrence order due to network delays, processing batch divisions, etc.; and multiple operations of the same subject and object may also be covered by subsequent events, resulting in the loss of historical status, and then the problem of time discontinuity. Once the problem of time discontinuity occurs, it will affect the accuracy of anomaly detection. Therefore, this solution uses the GMM model to predict the node status of events that are not processed in time, and fuses the predicted value and the actual value to achieve the correction of the node status.

[0032] It should be noted that this solution does not make a prediction immediately for each streaming event obtained, but after a period of time, that is, when the time of the streaming event exceeds the event length of the time window, a batch update is performed using the past stored states and the newly obtained states in the adjacency table.

[0033] Furthermore, this solution can effectively propagate the associated information between nodes by using the message passing function to pass information for each streaming event. As streaming events are constantly generated, the relationship between entities (nodes) in the system is also changing dynamically. The role of the message passing function is to convert the dependency of the four-tuple streaming event (such as process A calling file B) into the feature representation (embedded vector) of the node, thereby reflecting the dynamic changes of the system in real time to support real-time anomaly detection.

[0034] Further, in the step of "using a message passing function to pass information and update the node state of the corresponding node", whenever the sampling embedding condition is met, the message passing function is used to pass the information of the streaming event to update the node state of the subject and object of the corresponding streaming event. In some embodiments, the sampling embedding condition is that the streaming event time of the received streaming event just triggers the sliding operation of the time window.

[0035] In addition, since the same batch of events enters the current time window of the adjacency table,<a,b,connect,t1> and<a,b,connect,t2> For this kind of event that only has different time information, the event at time t2 will cover the event at time t1, thus causing event discontinuity. Therefore, this scheme introduces a prediction and correction mechanism to alleviate the discontinuity of events.

[0036] Specifically, this solution selects the Gaussian mixture model to predict the node status. The Gaussian mixture model (GMM) is a probabilistic model that can model complex data distribution. Through this model, the historical state information of the node can be used to predict the future state, and then the current node state can be corrected to improve the accuracy of subsequent anomaly detection.

[0037] Furthermore, the historical node states of the nodes are collected, where the historical node states are feature vectors of the nodes in the previous period of time, such as the resource usage of the nodes, the frequency of interaction with other nodes, etc. These feature vectors constitute the data set for training the Gaussian mixture model. The Gaussian mixture model is constructed based on the historical node states, where the formula for constructing the Gaussian mixture model is as follows: ; in ω Indicates the number of components, usually set to 2, corresponding to two event types (positive event: "interaction exists"; negative event: "no interaction"), α j Indicates the proportion of different event types, which is initialized as uniform distribution and subsequently updated through statistical distribution; are the mean and covariance, where the mean represents the average impact of event type h on the memory state change of node i, and the covariance represents the volatility of event type h on the memory state change of node i.

[0038] Furthermore, in the step of "predicting the predicted node state of each node based on the Gaussian mixture model", the historical node state of each node before the current moment is input into the Gaussian mixture model for prediction to obtain the predicted node state.

[0039] In the step of "fusing the predicted node state and the current node state to obtain the corrected node state of each node", the predicted node state and the current node state are weighted and added together to obtain the corrected node state of each node.

[0040] The corresponding formula is as follows: Si''=(1-x)*Si'+x*Si; Among them, Si' is the predicted node state, Si is the current node state, Si'' is the corrected node state, and x is a value between 0 and 1, which is determined according to the data situation of the streaming provenance subgraph. When the data situation in the streaming provenance subgraph is low noise and stable data, the value of x is close to 1, indicating that the current node state is more trusted; when the data situation in the provenance graph is: high noise and unstable data, the value of x is close to 0, indicating that the predicted node state is more trusted.

[0041] It should be noted that the data situation of the streaming traceability subgraph refers to the fact that the status of the nodes in different time periods does not change significantly. If there is no significant change, the data is considered stable.

[0042] Furthermore, in the step of "assigning the weight of each node based on the probability of occurrence of each streaming event", the event frequency in the event frequency database is obtained as the probability of occurrence of the current streaming event, and the difference between 1 and the probability of occurrence is taken as the weight of the node involved in the current streaming event. The event frequency database is a database that calculates the probability of occurrence of streaming events based on historical data, records the possible probability of occurrence of each streaming event, and can then predict the probability of new streaming events based on the frequency of historical streaming events of the same type.

[0043] In the step of "performing weighted aggregation encoding on the node status of each node to obtain the encoding information of each node", the neighbor nodes and operations corresponding to all associated objects are obtained from the adjacency table of the node corresponding to the subject of the streaming event, the information of the neighbor nodes is encoded to obtain the neighbor messages, and the neighbor information of all neighbor nodes is aggregated according to the corresponding weights to obtain the encoding information of the current subject node.

[0044] In other words, the feature information of each node not only contains its own information but also the information of the nodes around it. Therefore, this scheme combines the neighboring nodes of the main node for weighted aggregation to obtain the embedding vector of the current main node. The aggregation formula is as follows: ; Node i Represents the neighbor information of the i-th neighbor node, represents the embedding vector of the a-th subject node, It represents the probability of occurrence of a streaming event between the a-th main node and the i-th neighbor node.

[0045] In some embodiments, the key-value pair constructed by this solution uses the encoding information and the update time as the value and the current node as the key, and is expressed in the form of: <node i: <state encoding, update time>>.

[0046] In step S3, in predicting abnormal nodes based on coding information, a decoder is used to perform abnormal judgment based on the coding information of each node. In some embodiments, the decoder judges whether the coding information of the current node matches based on the historical coding information of the node. The greater the difference between the coding information of the current node and the historical coding information, the greater the abnormal value of the current node. In some embodiments, the decoder is MLP, LSTM, etc.

[0047] Furthermore, in order to remove expired data to avoid releasing available memory usage overhead, step S3 of this solution further includes: determining the survival time of each node based on the abnormal values ​​and time windows of nodes that are not determined to be abnormal nodes, and removing nodes whose survival time within the key-value pair is less than the set threshold.

[0048] The calculation formula for survival time is as follows: .

[0049] In addition, since the sending of attacks often includes the impact on multiple entity information, this solution associates abnormal nodes through label propagation rules to determine whether abnormal nodes are propagated. Further, step S3 of this solution further includes: All abnormal nodes are used as initialization labels, and each initialization label is propagated in all directions to find other abnormal nodes on the propagation path. If the distance between the next abnormal node on the propagation path and the current abnormal node exceeds a predetermined distance threshold, the label will no longer be propagated.

[0050] As mentioned above, this solution provides a complete set of anomaly detection methods for streaming traceability graphs. In order to more clearly demonstrate the streaming anomaly detection process of the anomaly detection method for streaming traceability graphs, the following is a Figure 3 The flow chart of streaming anomaly detection is introduced by Figure 3 It can be seen that when the time window of the adjacency table does not slide, the collected streaming events are cached in the adjacency table, and the time window of the adjacency table is slid at set intervals for each time period, and the streaming events in the current time window are encoded between the sliding of the time window; it is determined whether the event time exceeds the time window, if not, the traceability subgraph corresponding to the adjacency table of the time window is obtained, if so, the streaming traceability subgraph corresponding to the adjacency table of the corresponding time window is queried and the streaming traceability subgraph is corrected and sampled; the event frequency database is queried, and the weight of each node is constructed based on the event frequency, and then the node state of each node is weighted and aggregated to obtain the encoding information of each node, and the key-value pair is updated based on the encoding information; based on the encoding information of each node, it is determined whether the current node is an abnormal point, if it is an abnormal point, an alarm is generated and the alarm is associated, and the survival time of the node is calculated regardless of whether it is an abnormal node, and the corresponding node is deleted from the key-value pair based on the survival time.

[0051] Embodiment 2 Based on the same idea, refer to Figure 4 , the present application also proposes an anomaly detection system of a streaming traceability graph based on iterative prediction and correction, including: Streaming data collection unit: used to collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple including subject, object, operation and timestamp; A sampling embedding unit is used to set an adjacency table to cache streaming events acquired in real time and slide the time window of the adjacency table at set intervals, wherein the subject of the streaming event serves as a node of the streaming traceability graph, and the linked list of each node records the objects and operations associated with the current streaming event; a message passing function is used to pass information to each streaming event and update the node state of the corresponding node, and the predicted node state of each node is predicted based on a Gaussian mixture model, and the predicted node state and the current node state are fused to obtain the corrected node state of each node; a weight is assigned to each node based on the probability of occurrence of each streaming event, and the node state of each node is weighted and aggregated to obtain the encoding information of each node, and a key-value pair is constructed with the encoding information and the update time as the value and the current node as the key; The anomaly detection unit is used to predict abnormal nodes based on the encoding information: the encoding information of each node is used as the input of the decoder to predict the abnormal value of each node, and if the abnormal value exceeds the threshold, the current node is judged to be an abnormal node.

[0052] It should be noted that the anomaly detection system is a complete system as a whole. The streaming data sampling unit, the sampling embedding unit and the anomaly detection unit are only schematic illustrations of the division based on functional modules. The specific technical means of the anomaly detection system are introduced in Example 1.

[0053] Embodiment 3 This embodiment also provides an electronic device, referring to Figure 5 , including a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the anomaly detection method of the streaming traceability graph based on iterative prediction and correction.

[0054] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0055] The memory 404 may include a large-capacity memory 404 for data or instructions. The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the anomaly detection methods for streaming provenance graphs based on iterative prediction and correction in the above embodiments.

[0056] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402. The transmission device 406 may be used to receive or send data via a network.

[0057] The input / output device 408 is used to input or output information. In this embodiment, the input information may be streaming data, etc., and the output information may be abnormal points, etc.

[0058] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program: S1: Collect streaming data: Collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple of subject, object, operation and timestamp; S2: Encoding information of each node in the streaming traceability graph based on streaming data sampling encoding: S21: setting an adjacency table to cache the streaming events acquired in real time and sliding the time window of the adjacency table at each set time interval, wherein the subject of the streaming event is used as a node of the streaming traceability graph, and the linked list of each node records the object and operation associated with the current streaming event; S22: Whenever the time of a streaming event exceeds the time of a time window, call the streaming traceability subgraph in the adjacency list of the current time window, use the message passing function in the streaming traceability subgraph to pass information and update the node state of the corresponding node, predict the predicted node state of each node based on the Gaussian mixture model, and fuse the predicted node state and the current node state to obtain the corrected node state of each node; assign a weight to each node based on the probability of occurrence of each streaming event, perform weighted aggregation coding on the node state of each node to obtain the coding information of each node, and construct a key-value pair with the coding information and the update time as the value and the current node as the key; S3: Predict abnormal nodes based on encoded information: The encoded information of each node is used as the input of the decoder to predict the outlier value of each node. If the outlier value exceeds the threshold, the current node is judged as an outlier node.

[0059] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0060] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0061] Embodiments of the present invention can be implemented by computer software, which is executable by a data processor of a mobile device, such as in a processor entity, or implemented by hardware, or implemented by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros can be stored in any device readable data storage medium, and they include program instructions for performing specific tasks. Computer program products can include one or more computer executable components configured to perform embodiments when the program is running. One or more computer executable components can be at least one software code or a part thereof. In addition, at this point, it should be noted that any box of the logic flow in the figure can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored in physical media such as memory chips or storage blocks implemented in processors, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and data variants thereof, CDs. Physical media are non-transient media.

[0062] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for detecting anomalies in a streaming traceability graph based on iterative prediction and correction, characterized in that: The following steps are involved: S1: Collect streaming data: Collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple of subject, object, operation and timestamp; S2: Encoding information of each node in the streaming traceability graph based on streaming data sampling encoding: S21: setting an adjacency table to cache streaming events acquired in real time and sliding a time window of the adjacency table at intervals of a set time period; S22: Whenever the time of a streaming event exceeds the time of a time window, the streaming traceability subgraph in the adjacency list of the current time window is retrieved, and the node state of the corresponding node is updated by using a message passing function in the streaming traceability subgraph to pass information, and the predicted node state of each node is predicted based on a Gaussian mixture model, and the predicted node state and the current node state are fused to obtain the corrected node state of each node; a weight is assigned to each node based on the occurrence probability of each streaming event, and the node state of each node is weighted aggregated and encoded to obtain the encoding information of each node; S3: Predict abnormal nodes based on encoded information: The encoded information of each node is used as the input of the decoder to predict the outlier value of each node. If the outlier value exceeds the threshold, the current node is judged as an outlier node.

2. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: The survival time of each node is determined based on the outlier value and time window of each node, and nodes whose survival time in the key-value pair is less than the set threshold are eliminated.

3. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: All abnormal nodes are used as initialization labels, and each initialization label is propagated in all directions to find other abnormal nodes on the propagation path. If the distance between the next abnormal node on the propagation path and the current abnormal node exceeds a predetermined distance threshold, the label will no longer be propagated.

4. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: The subject of the streaming event is the node of the streaming traceability graph. A linked list is constructed for each node in the adjacency list, and the linked list of each node records the object and operation associated with the current streaming event.

5. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: Whenever the sampling embedding condition is met, the message passing function is used to pass the information of the streaming event to update the node status of the subject and object of the corresponding streaming event.

6. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: Collect the historical node status of the node, and build a Gaussian mixture model based on the historical node status. The formula for building the Gaussian mixture model is as follows: ； in ω Indicates the quantity of components, α j Indicates the proportion of different event types. are the mean and covariance, where the mean represents the average impact of event type h on the memory state change of node i, and the covariance represents the volatility of event type h on the memory state change of node i.

7. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: The predicted node state and the current node state are weighted and added together to obtain the corrected node state of each node.

8. The anomaly detection method of streaming traceability graph based on iterative prediction and correction according to claim 1 is characterized in that: Based on the adjacency table of the node corresponding to the subject of the streaming event, the neighbor nodes and operations corresponding to all associated objects are obtained, the information of the neighbor nodes is encoded to obtain the neighbor message, and the neighbor information of all neighbor nodes is aggregated according to the corresponding weights to obtain the encoded information of the current subject node.

9. An anomaly detection system for streaming provenance graph based on iterative prediction and correction, characterized in that: include: Streaming data collection unit: used to collect streaming data in real time and standardize the streaming data into streaming events, where the streaming events include a four-tuple including subject, object, operation and timestamp; A sampling embedding unit is used to set an adjacency table to cache streaming events acquired in real time and slide the time window of the adjacency table at set intervals, wherein the subject of the streaming event serves as a node of the streaming traceability graph, and the linked list of each node records the objects and operations associated with the current streaming event; whenever the time of the streaming event exceeds the time of the time window, the streaming traceability subgraph in the adjacency table of the current time window is called, and the message passing function is used in the streaming traceability subgraph to pass information and update the node state of the corresponding node, and the predicted node state of each node is predicted based on the Gaussian mixture model, and the predicted node state and the current node state are fused to obtain the corrected node state of each node; a weight is assigned to each node based on the probability of occurrence of each streaming event, and the node state of each node is weighted and aggregated to obtain the encoding information of each node, and a key-value pair is constructed with the encoding information and the update time as the value and the current node as the key; The anomaly detection unit is used to predict abnormal nodes based on the encoding information: the encoding information of each node is used as the input of the decoder to predict the abnormal value of each node, and if the abnormal value exceeds the threshold, the current node is judged to be an abnormal node.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes a program code for controlling a process to execute a process, and the process includes an anomaly detection method for a streaming provenance graph based on iterative prediction correction according to any one of claims 1 to 8.

Images