Data processing method and data processing device

Abstract

The present application provides a data processing method and a data processing device. The method comprises: propagating a label of a starting point of an event to a label of an end point of the event. A label of a node can indicate historical state information of a target path which has been detected in log data and uses the node as an end point, so that by means of label propagation, an abnormal path having an abnormal historical state is found, and the abnormal path is output in a local streaming provenance graph mode. According to the method, a complete streaming provenance graph is not required to be constructed, the occupation of computing resources and memories can be reduced, abnormal behaviors (for example, abnormal paths) can be detected in real time, and abnormal paths can be output so as to trace abnormal behaviors.

Description

Data processing method and data processing device

[0001] This application claims priority to Chinese Patent Application No. 202410045734.5, filed on January 11, 2024, entitled "Data Processing Method and Data Processing Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of information security technology, and in particular to a data processing method and a data processing apparatus. Background Technology

[0003] Anomaly detection is an important component of building a comprehensive and in-depth intrusion detection system, and it is of great value in protecting information system security.

[0004] Currently, the relevant technology can obtain offline data by collecting offline data streams and control flows in the system after the fact; then, a complete origin graph can be constructed using the offline data; and finally, critical paths can be searched in the origin graph to identify anomalies.

[0005] However, this approach requires constructing a complete origin graph before detecting anomalous behavior, resulting in high memory and computing resource consumption.

[0006] Summary of the Invention

[0007] This application provides a data processing method and a data processing apparatus. By using tag propagation, the historical state information of the target path indicated by the tag at the starting point of an event is propagated to the tag at the end point of the event. When the historical state information indicated by the tag at the end point meets the alarm conditions, abnormal paths with abnormal historical state information can be identified to identify attack behavior. This process does not require the complete construction of the origin graph, which can reduce the consumption of memory and computing resources.

[0008] In one possible implementation, this application provides a data processing method. The method includes: determining a target event to be detected based on real-time collected log data, the target event including a start point, an end point, and a relationship between the start point and the end point; determining a second label for the end point based on a first label of the start point; wherein, the label of each node in the target event is used to indicate historical state information of a target path ending at that node, wherein the target path is one of at least one paths ending at that node, wherein the at least one path is a path composed of at least one event to be detected determined based on the log data; wherein the nodes in the target event include the start point and the end point of the target event; determining second path information related to the second label based on first path information associated with the first label; and outputting a local streaming origin graph related to the end point of the target event based on the second path information when the second label meets an alarm condition.

[0009] The log data in question is log data collected in real time from the device under test.

[0010] Log data can be converted into a data stream, which may include multiple events, each of which is a target event to be detected.

[0011] The historical status information of the target path can be at least one of the following: anomaly level information, historical information between the node and the network (e.g., the node has network data inflow), historical information between the node and the file (e.g., information about the node reading sensitive files), or historical information about the node calling or being called by certain processes.

[0012] For ease of explanation, we will take the historical state information as an anomaly level information as an example. When the historical state information is other types of information, the implementation principle of the method in this application is the same, and will not be repeated here.

[0013] Both the start and end points of an event can have their own labels, initially empty and containing no data. However, as more abnormal events are detected, and at least one path terminates at a corresponding node, the information indicated by the node's label can be updated or assigned as historical state information of the target path terminating at that node. For example, the target path could be the path with the highest degree of abnormality among at least one path terminating at that node, and the historical state information would be the degree of abnormality information of that target path. For instance, the information indicated by the node's label could be an anomaly score AS indicating the degree of abnormality, or a normality score RS indicating the normality of the path with the highest degree of abnormality, where RS + AS = 1.

[0014] Furthermore, based on differences in historical state information and application scenarios, the target path can be the path with the highest degree of anomaly among at least one path ending at this node, or a path with a relatively high degree of anomaly among at least one path ending at this node, or a path with better semantics among at least one path ending at this node, or a path with network connectivity among at least one path ending at this node, etc. The specific path chosen as the target path can vary depending on the requirements of the application scenario and the specific content of the historical state information; therefore, no restrictions are imposed here.

[0015] Since the label of each node in the target event is used to indicate the historical state information of the target path with that node as the endpoint, when the node is the starting point of the target event, the target path is the first path information associated with the first label; when the node is the endpoint of the target event, the target path is the second path information associated with the second label.

[0016] Furthermore, the alarm conditions satisfied by the second label vary depending on the content of the historical status information of the target path indicated by the second label. These alarm conditions can be flexibly configured based on the application scenario and the specific content of the historical status information.

[0017] For example, if the historical status information is an anomaly level information, the alarm condition can be that the anomaly level exceeds a certain anomaly level threshold, which will trigger an alarm for the abnormal path (i.e., the second path information related to the second label).

[0018] For example, if the historical status information is related to at least one of the network or files, then the alarm condition can be the condition that the network has been connected and data has been sent out and sensitive files have been read. In this way, the path formed by the nodes that have been connected to the network and read sensitive files can be output as the second path information.

[0019] Therefore, the alarm conditions are related to the specific content of the historical status information and the scenario, and are not limited here.

[0020] The following implementation will be illustrated using the information stored in the tag as the normal score RS of the target path mentioned above.

[0021] In this embodiment, considering that a single event is insufficient to support accurate detection of attack behavior, this application can discover abnormal behavior at the path granularity. The application can determine abnormal paths (such as the second path information mentioned above) from real-time collected log data, and when the abnormal path meets the alarm conditions, directly output a local streaming origin graph based on the determined abnormal path. In this process, it is not necessary to construct a complete streaming origin graph, thereby reducing the memory and computing resource consumption of constructing the origin graph. Furthermore, this method can perform real-time detection of abnormal paths that meet alarm conditions in real-time log data, and output the streaming origin graph constructed from these abnormal paths in real-time, facilitating rapid real-time detection of attack behavior. Moreover, the historical state information of the target path indicated by the label of the event's starting point can be transmitted to the label of the event's ending point, thereby continuously determining the target path of each node through label propagation (explanation and definition are above), improving the accuracy of the determined abnormal paths. Furthermore, the determined abnormal paths can be output as a streaming origin graph, enabling the reconstruction of attack behavior paths, facilitating the analysis of attack behavior by operations and maintenance personnel.

[0022] In one possible implementation, the method further includes: determining the normality level of the target event; determining historical state information (e.g., abnormality level information) of a third path based on a first label of the starting point and the normality level of the target event; wherein the third path is a path composed of the first path information and the target event; and determining a second label of the endpoint based on the historical state information of the third path.

[0023] The normality of an event can be expressed by any information that can indicate the normality of the event. As the specific content of the historical state information differs, the information indicating the normality of the event can also differ. Because the historical state information is different, the definition of abnormality of the abnormal behavior to be detected can also differ.

[0024] For example, when the historical state information is anomaly level information, the normality of the event can be the frequency of the event (referring to normal events) mentioned in the implementation. When the historical state information is other information, the information characterizing the normality of the event can be other frequency information of the event, which is not limited here.

[0025] For example, the first label of the starting point (e.g., file A) in the target event is the normal score AS of the most abnormal path that has been detected according to the method of this application, ending at file A.

[0026] For example, the normality of the third path (e.g., normality score RS) can be determined by multiplying the normality of the first label and the target event, and then the normality of the third path can be used to determine its abnormality (e.g., abnormality score AS = 1 - RS).

[0027] Of course, when the historical state information is not anomaly information, the algorithm between the first label and the normality of the target event is not limited to the multiplication operation here, but can also be other operation methods, which are not restricted here.

[0028] Furthermore, the normality of the target event determined above can be a preset threshold or it can be read from, for example, the normal behavior model described below. There are no restrictions on this, and it can be related to the content of the historical state information. There are no restrictions on this either.

[0029] In this embodiment of the application, a streaming label propagation method is used to propagate local labels along the path, which facilitates fast, lightweight, and real-time detection of abnormal paths based on the label of the destination.

[0030] In one possible implementation, the method further includes: determining the normality level of the target event based on a pre-constructed information table; wherein the information table includes first information indicating the normality level of a preset event.

[0031] The information table can be a normal behavior model, which can include the normality of multiple preset events (e.g., the frequency of events). When a target event is detected using real-time log data, the normality of the target event can be determined through the normal behavior model.

[0032] In one possible implementation, the method further includes: when the target event is the preset event in the information table, determining the normality of the target event based on first information in the information table indicating the normality of the target event.

[0033] If the target event belongs to the preset events in the information table, it means that the target event is a known event, and the normality of the target event can be determined by using the first information.

[0034] The first piece of information could be, for example, the degree of normality of a preset event in the normal behavior model (e.g., event e). i Frequency of occurrence M ei (or other information that can indicate the normality of the event).

[0035] Then, after determining the target event e, which belongs to the known events... i When the frequency M is within the normal range, eiAs the target event e i The degree of normality, or, M can be considered normal. ei *0.7+0.3 is taken as the target event e i The degree of normality, or other strategies, are not limited here.

[0036] In this embodiment of the application, the normality of the target event being detected can be obtained from a pre-built information table, thereby facilitating the rapid detection of abnormal behavior.

[0037] In one possible implementation, the method further includes: when the target event does not belong to the preset event in the information table, configuring a preset normality threshold (e.g., a frequency threshold, e.g., 0.1, not limited) as the normality of the target event.

[0038] In this embodiment of the application, M is an event that has not appeared in the information table. ei The value (UNSEEN_EVENT_SCORE) should not be too small to avoid insufficient information collected from the information table, and it also needs to have a certain degree of differentiation from the events (also described as known events) in the information table. In practice, the M value for unseen events can be... ei The value is set to 0.1 to indicate the normality of the event.

[0039] In one possible implementation, determining the historical state information of the third path based on the first label of the starting point and the normality of the target event includes: when the first label of the starting point is empty data, if the normality of the target event meets a first preset condition, then determining the historical state information of the third path based on the normality of the target event.

[0040] When the starting point label is empty, the normality of the target event can express the historical state information of the target path (e.g., the degree of abnormality), where the target path is the path of the target event. However, it is only necessary to determine the ending point label of the target event when the target event is an abnormal event. Therefore, when determining the normality of the target event (e.g., M of event 1), e1 If the first preset condition is met, the target event can be determined to be an abnormal event.

[0041] For example, if the target event is event 1, its frequency of occurrence in the normal behavior model is M. e1 Since the label of the starting point of event 1 is empty, therefore, M e1 Given the historical state information (e.g., normal score RS1) of the target path (here, the path of event 1), then in the M... e1 If the value is less than the normal threshold (e.g., 0.35), then event 1 is considered an abnormal event.

[0042] Then we can base our judgment on the normality of the target event (e.g., M). e1 To determine the historical state information of the third path, the label of the starting point of event 1 is empty, so the third path is also the path of event 1. For example, if the historical state information is the degree of abnormality, then the historical state information is RS1 = M. e1 Alternatively, the historical state information is AS = 1 - M e1 Accordingly, in conjunction with the above implementation method, when determining the second label of the endpoint based on the historical state information of the third path, if the second label of the endpoint currently has no information, for example, its value is empty, then the second label of the endpoint can be initialized with the historical state information of the third path (e.g., M). e1 For example, if the second label of the endpoint has a lower degree of anomaly than the historical state information of the third path (e.g., degree of anomaly), then the second label of the endpoint is updated to the historical state information of the third path.

[0043] In this embodiment, when the label of the starting point of the target event is empty, it means that no event with that starting point as the ending point has been detected from the real-time log data. In this case, the normality level (e.g., M) of the target event can be directly determined. ei The normal score RS of the third path is used as the normal score. If the RS is less than the normal score threshold, it means that the normality of the event meets the first preset condition and the event is an abnormal time. The RS is used to determine the AS of the third path to determine its abnormality.

[0044] In one possible implementation, the historical state information includes anomaly degree information. The step of determining the historical state information of the third path based on the first label of the starting point and the normality of the target event includes: determining the normality of the target path indicated by the first label based on the first label of the starting point; and determining the anomaly degree information of the third path based on the normality of the target path and the normality of the target event.

[0045] In this embodiment, when the label of the starting point of the target event has data, it indicates that the method of this application has detected an event with that starting point as the endpoint from the real-time log data. Then, the normality of the target path indicated by the first label of the starting point (e.g., AS) can be compared with the normality of the currently detected target event (e.g., M). ei Operations such as multiplication are performed to obtain the normal score RS of the third path, and then the AS of the third path is determined using RS to determine its degree of abnormality.

[0046] In one possible implementation, determining the second label of the destination based on the historical state information of the third path includes: when the second label of the destination is empty data, initializing the second label of the destination to indicate the historical state information of the third path.

[0047] In this implementation, when the label of the endpoint of the target event is empty, the label of the endpoint of the target node is initialized to indicate the historical state information (e.g., anomaly level information) of the third path. For example, the label can store the normal score RS of the third path or the abnormal score AS of the third path; there is no limitation here.

[0048] In one possible implementation, determining the second path information related to the second tag based on the first path information related to the first tag includes: when it is determined that the second path information is empty data, initializing the second path information related to the second tag of the destination as the information of the third path.

[0049] When the second label is empty, the second path information associated with that second label is also empty.

[0050] Therefore, in this embodiment of the application, the third path information (here, the path of the current target event) can be directly used as the second path information related to the label of the endpoint of the target event.

[0051] In one possible implementation, determining the second label of the endpoint based on the historical state information (e.g., anomaly level) of the third path includes: when it is determined that the second label needs to be refreshed based on the historical state information of the third path and the historical state information of the target path indicated by the second label, updating the second label of the endpoint to indicate the historical state information of the third path.

[0052] In this embodiment, the second label of the endpoint is not empty data. Therefore, when determining whether the second label needs to be refreshed, the relationship between the historical status information of the third path and the historical status information of the target path indicated by the second label can be used to decide whether the second label needs to be refreshed.

[0053] For example, if the historical status information is anomaly level information, then if it is determined that the anomaly level of the third path is greater than the anomaly level of the target path indicated by the second label, then it is determined that the second label needs to be refreshed, and the second label of the endpoint can be updated to information indicating the anomaly level of the third path.

[0054] In this embodiment, the endpoint of the target event currently has a label, meaning its second label contains corresponding information. For example, the label of the node stores the RS of the most abnormal path terminating at that node. Then, it can be determined whether the abnormality level of the currently detected third path passing through the target event to reach the endpoint is greater than the abnormality level currently indicated by the second label of the target event's endpoint. If so, it indicates that the second label needs to be updated to reflect the abnormality level of the third path. This ensures that the label of the detected endpoint of the target event always maintains the latest historical state information.

[0055] In one possible implementation, determining the second path information related to the second tag based on the first path information related to the first tag includes: refreshing the second path information related to the second tag of the destination to the information of the third path.

[0056] In this embodiment, when the label of the endpoint of the target event is refreshed, the path information associated with that label also needs to be refreshed synchronously to the corresponding third path information. This ensures that the node's label and the path information associated with that label (e.g., the most abnormal path) remain synchronized.

[0057] In one possible implementation, the first tag and the first path information, the second tag and the second path information are all cached in memory.

[0058] In the above embodiments, the first label of the starting point of the target event and its first path information can be cached in memory, and the label of the ending point of the target event and its second path information are also cached in memory.

[0059] In this way, for a node in a detected event, this application can cache the information of the degree of abnormality of the most abnormal path ending at that node, as well as the information of the most abnormal path. Therefore, it is not necessary to fully construct the streaming origin graph formed by the paths corresponding to all events, but only to cache the relevant information of the abnormal paths in the streaming origin graph, which can reduce the caching and computing resource usage.

[0060] In one possible implementation, the method further includes: deleting the information of the first tag and the first path information cached in memory when at least one of the first tag at the starting point and the first path information associated with the first tag meets a second preset condition.

[0061] This implementation mainly describes the case of reducing the labels of nodes. The steps of this method can be performed before label propagation, for example, before the step of determining the second label of the endpoint based on the first label of the starting point.

[0062] In this embodiment, if at least one of the labels of the starting point of the currently detected event or its corresponding most abnormal path satisfies a second preset condition, the label information and path information cached for that starting point in memory can be deleted to reduce further propagation of the labels. This is because attack behavior does not often involve normal behavior; when the path is long but the alarm condition has not yet been met, clearing the labels of the corresponding nodes reduces the alarm rate for abnormal paths.

[0063] In one possible implementation, the method further includes: deleting the information of the first tag and the first path information cached in memory when the historical state information of the target path indicated by the first tag at the starting point meets the tag reduction condition (e.g., the degree of abnormality is less than a second preset abnormality threshold).

[0064] For example, in event e i If the RS (normal score) stored in the first label of a source node is greater than a preset normal score threshold, it indicates that the path associated with that first label is not abnormal enough; or if the AS indicated by that first label is less than a preset abnormal score threshold, it indicates that the path associated with that first label is not abnormal enough. Therefore, the label of the source node and the most abnormal path information indicated by the RS in that label can be deleted to reduce memory usage. This can prevent the propagation of labels with normal scores (RS) greater than the preset normal score threshold.

[0065] In one possible implementation, the method further includes: deleting the information of the first tag and the first path information cached in memory when the path length of the first path information associated with the first tag is greater than a preset length threshold.

[0066] Among them, event e i The path length (length) of the first path information related to the tag starting from the src (e.g., the most unusual path ending at the src) can indicate the number of tag propagation rounds.

[0067] If the number of propagation rounds of the label exceeds the preset number of rounds threshold, it means that the label has been propagated too many times without triggering an alarm. The label of the node and the historical status information (such as the most abnormal path information) indicated by the label can be deleted to prevent the label from being over-propagated.

[0068] In one possible implementation, the method further includes: deleting the information of the second tag and the second path information cached in memory when at least one of the second tag at the endpoint and the second path information associated with the second tag meets a third preset condition.

[0069] This implementation mainly describes the case of reducing the labels of nodes. The steps of this method can be executed after label propagation, for example, after the step of determining the second label of the endpoint based on the first label of the starting point.

[0070] In this embodiment, when at least one of the labels of the endpoint of the currently detected event or its corresponding most abnormal path meets a third preset condition, the label information and path information cached for that endpoint in memory can be deleted to reduce further propagation of the labels. This is because attack behavior does not involve many normal behaviors; when the path is long but the alarm condition has not yet been met, clearing the labels of the corresponding nodes reduces the alarm rate for abnormal paths.

[0071] In one possible implementation, the method further includes: deleting the information of the second tag and the second path information cached in memory when the historical state information of the target path indicated by the second tag at the endpoint meets the tag reduction condition (e.g., the degree of abnormality is less than a certain preset abnormality threshold, which may be the same as or different from the above-mentioned second preset abnormality threshold).

[0072] The principle behind this implementation is the same as that for reducing the first label at the starting point, so it will not be elaborated here.

[0073] In one possible implementation, the method further includes: deleting the information of the second tag and the second path information cached in memory when the path length of the second path information associated with the second tag is greater than a certain length threshold (which may be the same as or different from the preset length threshold).

[0074] Among them, event e i The path length (length) of the second path information associated with the label of the endpoint dst (e.g., the most unusual path whose endpoint is the endpoint of the path) can indicate the number of label propagation rounds.

[0075] If the number of propagation rounds of the label exceeds the preset number of rounds threshold, it means that the label has been propagated too many times without triggering an alarm. The label of the node and the historical status information (such as the most abnormal path information) indicated by the label can be deleted to prevent the label from being over-propagated.

[0076] In one possible implementation, the historical state information includes anomaly degree information; the step of outputting a local streaming origin graph related to the endpoint of the target event based on the second path information when the second tag meets the alarm conditions includes: when the anomaly degree information of the target path indicated by the second tag is greater than a third preset anomaly degree threshold, outputting a local streaming origin graph related to the endpoint of the target event based on the second path information.

[0077] For example, if the label of a node stores the normal score RS of the corresponding third path, then when determining whether an alarm needs to be triggered for the third path, it can be determined whether the normal score RS is less than the preset normal score alarm threshold. If the normal score RS stored in the label is less than the preset normal score alarm threshold, it means that the label of the target node meets the alarm conditions, and the second abnormal path in the streaming origin graph can be output.

[0078] For example, the preset alarm threshold can also be a preset abnormal score alarm threshold. When determining whether an alarm needs to be triggered for the third path, the normal score RS indicated by the second tag of the target node can be calculated according to Formula 5. Then, it can be determined whether the AS is greater than the preset abnormal score alarm threshold. If the AS is greater than the preset abnormal score alarm threshold, it means that the tag of the target node meets the alarm conditions, and the second abnormal path in the streaming origin graph can be output. The larger the preset abnormal score alarm threshold, the more difficult it is to trigger an alarm. The threshold in the alarm conditions can be flexibly set according to the need to trigger an alarm, such as a preset normal score alarm threshold or a preset abnormal score alarm threshold, etc., which indicate the alarm conditions.

[0079] In one possible implementation, the step of outputting a local streaming origin graph related to the endpoint of the target event based on the second path information includes: aggregating the second path information, and with the local streaming origin graph of the target node in the second path information to obtain an aggregated local streaming origin graph; wherein the target node is a node whose label satisfies the alarm condition; and outputting the aggregated local streaming origin graph as a local streaming origin graph related to the endpoint of the target event.

[0080] In one possible implementation, the method further includes updating the local streaming origin graph of each node in the aggregated local streaming origin graph to the aggregated local streaming origin graph.

[0081] In one possible implementation, determining the second label of the endpoint based on the first label of the starting point includes: determining the second label of the endpoint based on the first label of the starting point and a preset normalization coefficient.

[0082] This can be combined with any of the above implementation methods that use the first label of the starting point to determine the second label of the ending point. Each time the second label of the ending point is determined, a preset normalization coefficient (e.g., α) can be used.

[0083] In this embodiment of the application, considering that the longer the path, the higher the anomaly score, a normalization coefficient can be used to offset the impact of path length on the accuracy of the label.

[0084] In one possible implementation, this application provides a data processing apparatus. The apparatus may include: a first determining module, configured to determine a target event to be detected based on real-time collected log data, the target event including a start point, an end point, and a relationship between the start point and the end point; a second determining module, configured to determine a second label for the end point based on a first label of the start point; wherein, the label of each node in the target event is used to indicate historical state information of a target path ending at that node; wherein, the target path is one of at least one paths ending at that node, wherein the at least one path is a path composed of at least one event to be detected determined based on the log data; wherein, the nodes in the target event include the start point and the end point of the target event; a third determining module, configured to determine second path information related to the second label based on first path information related to the first label; and an alarm module, configured to output a local streaming origin graph related to the end point of the target event based on the second path information when the second label meets an alarm condition.

[0085] In one possible implementation, the apparatus further includes: a fourth determining module, configured to determine the normality level of the target event; and a second determining module, specifically configured to: determine historical state information of a third path based on a first label of the starting point and the normality level of the target event; wherein the third path is a path composed of the first path information and the target event; and determine a second label of the endpoint based on the historical state information of the third path.

[0086] In one possible implementation, the fourth determining module is specifically used to determine the normality level of the target event based on a pre-built information table; wherein the information table includes first information indicating the normality level of the preset event.

[0087] In one possible implementation, the fourth determining module is specifically used to determine the normality of the target event based on first information in the information table indicating the normality of the target event when the target event is the preset event in the information table.

[0088] In one possible implementation, the fourth determining module is specifically configured to set the preset normality threshold as the normality of the target event when the target event does not belong to the preset event in the information table.

[0089] In one possible implementation, the second determining module is specifically used to determine the historical state information of the third path based on the normality of the target event if the normality of the target event meets a first preset condition when the first label of the starting point is empty data.

[0090] In one possible implementation, the historical state information includes anomaly degree information, and the second determining module is specifically used to: determine the normality of the target path indicated by the first tag based on the first tag of the starting point; and determine the anomaly degree information of the third path based on the normality of the target path and the normality of the target event.

[0091] In one possible implementation, the second determining module is specifically configured to initialize the second label of the endpoint to indicate historical status information of the third path when the second label of the endpoint is empty data.

[0092] In one possible implementation, the third determining module is specifically used to initialize the second path information related to the second tag of the endpoint as the information of the third path when it is determined that the second path information is empty data.

[0093] In one possible implementation, the second determining module is specifically used to update the second label of the endpoint to information indicating the degree of abnormality of the third path when the second label of the endpoint is not empty data and the degree of abnormality of the third path is greater than the degree of abnormality of the target path indicated by the second label.

[0094] In one possible implementation, the third determining module is specifically used to determine, based on the historical state information of the third path and the historical state information of the target path indicated by the second tag, that when it is necessary to refresh the second tag, update the second tag of the destination to the historical state information indicating the third path.

[0095] In one possible implementation, the first tag and the first path information, the second tag and the second path information are all cached in memory.

[0096] In one possible implementation, the device further includes a deletion module, configured to delete the information of the first tag and the first path information cached in memory when at least one of the first tag at the starting point and the first path information associated with the first tag satisfies a second preset condition.

[0097] In one possible implementation, the deletion module is specifically used to delete the information of the first tag and the first path information cached in memory when the historical state information of the target path indicated by the first tag at the starting point meets the tag reduction condition.

[0098] In one possible implementation, the deletion module is specifically used to delete the information of the first tag and the first path information cached in memory when the path length of the first path information associated with the first tag is greater than a preset length threshold.

[0099] In one possible implementation, the historical state information includes anomaly degree information; the alarm module is specifically used to output a local streaming origin graph related to the endpoint of the target event based on the second path information when the anomaly degree information of the target path indicated by the second tag is greater than a third preset anomaly degree threshold.

[0100] In one possible implementation, the apparatus further includes: an aggregation module, configured to aggregate the second path information and the target node in the second path information with a local streaming origin graph to obtain an aggregated local streaming origin graph; wherein the target node is a node whose label satisfies the alarm condition; and an output module, configured to output the aggregated local streaming origin graph as a local streaming origin graph related to the endpoint of the target event.

[0101] In one possible implementation, the apparatus further includes an update module, configured to update the local streaming origin graph of each node in the aggregated local streaming origin graph to the aggregated local streaming origin graph.

[0102] The effects of the data processing apparatus in the above embodiments are similar to those of the data processing methods in the above embodiments, and will not be repeated here.

[0103] In one possible implementation, embodiments of this application provide a computing device cluster including at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, causing the computing device cluster to perform a data processing method according to the first aspect or any possible implementation thereof.

[0104] The effect of the computing device cluster in this embodiment is similar to that of the data processing methods in the above embodiments, and will not be described again here.

[0105] In one possible implementation, this application provides a computer program product containing instructions that, when executed by a computing device cluster, cause the computing device cluster to perform a data processing method in the first aspect or any possible implementation thereof.

[0106] The effect of the computer program product in this embodiment is similar to that of the data processing methods in the above embodiments, and will not be repeated here.

[0107] In one possible implementation, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device cluster, enable the computing device cluster to perform the data processing method described in any of the above embodiments.

[0108] The effect of the computer-readable storage medium in this embodiment is similar to that of the data processing methods in the above embodiments, and will not be repeated here. Attached Figure Description

[0109] Figure 1 is a schematic diagram of an exemplary system architecture;

[0110] Figure 2 is a schematic diagram illustrating an exemplary data processing procedure;

[0111] Figure 3a is an exemplary schematic diagram of the paths between nodes;

[0112] Figure 3b is an exemplary schematic diagram of the paths between nodes;

[0113] Figure 3c is an exemplary schematic diagram of the paths between nodes;

[0114] Figure 3d is an exemplary schematic diagram of the paths between nodes;

[0115] Figure 3e is an exemplary schematic diagram of the paths between nodes;

[0116] Figure 4a is a schematic diagram illustrating an exemplary data processing procedure;

[0117] Figure 4b is a schematic diagram illustrating an exemplary data processing procedure;

[0118] Figure 4c is a schematic diagram illustrating an exemplary data processing procedure;

[0119] Figure 5 is a schematic diagram of a localized loss origin map, as exemplarily shown;

[0120] Figure 6 is a block diagram of an exemplary data processing apparatus;

[0121] Figure 7 is a schematic diagram of the structure of an exemplary computing device;

[0122] Figure 8 is a schematic diagram of the structure of an exemplary computing device;

[0123] Figure 9 is a schematic diagram of the structure of a computing device cluster as an example. Detailed Implementation

[0124] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0125] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0126] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0127] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0128] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0129] To facilitate understanding of the technical solutions of the embodiments of this application, before describing the technical solutions of the embodiments of this application, the technical terms involved in the embodiments of this application will be briefly introduced first:

[0130] Event triples: In most systems, actions can be abstracted as events (s, p, o) consisting of a triple of subject, predictor, and object. Here, s represents the executor or subject of the action, o represents the object or recipient of the action, p represents the action or predicate, the relationship or attribute between objects, and the time when the event occurs.

[0131] A streaming provenance graph contains information about nodes and edges, but is not necessarily in graph form. A streaming provenance graph treats the subject in the above event triples as the source node (also called the starting point), the object as the target node (also called the ending point), and the predicate as the edge, thus constructing a provenance graph describing the system's behavior. The data stream formed by these event triples in chronological order is the streaming provenance graph. Streaming provenance graphs possess most of the characteristics of graphs and support customized graph computation algorithms.

[0132] A path is formed by connecting several event triplets in chronological order. This chronological order refers to the sequence of events occurring from the furthest point in time to the present.

[0133] Tag propagation algorithms are a class of customized graph computation algorithms particularly well-suited for streaming origin graph analysis. They utilize the label information of labeled nodes to predict the label information of unlabeled nodes. The tag propagation algorithm stores the results of local computations in the node's label and propagates that node's label along the direction of the event (i.e., the edges in the streaming origin graph) to the next node, thus completing the overall computation.

[0134] Anomaly Path: Due to the stealth of attacks, a single event is difficult to support accurate detection. Therefore, this application can discover abnormal behavior at the granularity of the path, i.e., anomaly path.

[0135] Anomaly detection is a crucial component of building a comprehensive and in-depth intrusion detection system, and it is of great value in protecting information system security. A good anomaly detection solution must first be able to accurately identify anomalies from a massive amount of normal behavior; failure to accurately identify anomalies or generating too many false alarms will significantly reduce the value of the detection solution. Secondly, it needs to provide sufficient information to help analysts understand, confirm, and handle attacks.

[0136] Traditional host-side (endpoint) anomaly detection solutions are mainly based on the dynamic and static analysis of individual malicious programs, or on the analysis of specific logs. However, as network attacks gradually become more advanced long-term threats (also known as advanced persistent threats, APTs), and attack behaviors become more integrated, it is often difficult to distinguish local behaviors from normal behaviors.

[0137] In the traditional approach, anomaly detection can be performed using a process whitelist (as trusted processes). Specifically, if a process is not in the process whitelist, or is not a descendant process of a trusted process in the whitelist, it is considered abnormal.

[0138] However, attackers can easily exploit methods such as memory injection to launch attacks while the original process is running. These attacks can perfectly bypass deterministic anomaly detection. For example, during memory injection, malicious code can be injected into a whitelisted process running in memory to bypass detection and achieve the attack.

[0139] In the traditional approach two, anomaly detection can be performed based on anomaly analysis of process call chains. Specifically, all process call chains exhibiting normal behavior can be collected and stored as a set as a normal behavior model. Any process call chain not present in the set will trigger an alert.

[0140] While such methods can detect attacks based on malicious processes, attackers can also bypass anomaly detection based on process call chains by using methods such as memory injection to launch attacks without creating new processes.

[0141] Traditional approach three involves anomaly detection based on the origin graph. In this approach, data flow and control flow within the system are collected post-incidentally to obtain offline data. This offline data is then used to construct a complete origin graph. Next, critical paths are searched within the origin graph to identify anomalous paths. For example, algorithms such as graph matching and similarity calculation can be used to capture anomalous behavior on a global scale (a subgraph composed of interactions between multiple system entities).

[0142] However, the traditional solution 3 is a post-hoc algorithm that identifies abnormal paths from a constructed complete origin graph. It cannot detect abnormal behavior in real time, nor can it reconstruct attack paths in real time. This makes it difficult for alarm analysts to maintain and understand. In addition, the algorithm for constructing the origin graph has a large computational overhead, so constructing a complete origin graph will consume a lot of computing resources. Furthermore, the process of constructing a complete origin graph will consume a large amount of cache space.

[0143] To address this, this application provides a data processing method that can collect log data in real time to determine events. Each event includes a start point and an end point, both of which can have their own labels. For example, the initial label may be empty (this is not limited and can also be non-empty). A node's label can be used to indicate the historical state information of a target path ending at that node. This target path can be one of at least one path ending at that node. The at least one path is a path formed by at least one event (also called the event to be detected) determined based on the log data. This application can use label propagation to determine the label of the end point of the event based on the label of the start point, and to determine the path related to the label of the end point (i.e., the path indicated by the label) based on the path associated with the start point's label. Then, when a label meets an alarm condition, the path associated with that label can be displayed as a local streaming origin graph. This method does not require a complete origin graph to be constructed beforehand. Instead, it only processes real-time collected log data to identify abnormal paths that meet alarm conditions and reach the corresponding nodes. Then, it reconstructs and outputs alarm-laden abnormal paths to obtain a local origin graph, thereby enabling the detection of abnormal behavior. This method can identify abnormal paths (e.g., paths related to node labels, such as target paths) through real-time analysis of log data, enabling real-time detection of abnormal behavior. It also allows for real-time reconstruction and output of alarm-laden abnormal paths to assist alarm personnel in analyzing and understanding attack-related abnormal behavior and taking appropriate action. Furthermore, since it does not require a complete origin graph to be constructed before detecting abnormal behavior, but only a local origin graph to be constructed for alarm-laden abnormal paths, it significantly reduces the computational resource consumption and cache space usage of the origin graph algorithm. The historical state information of the target path indicated by the label of the event's starting point can be passed to the label of the event's ending point, thus continuously determining the target path of each node through label propagation (explanation and definition above), improving the accuracy of the identified abnormal paths. Furthermore, it can output the identified abnormal paths as a streaming origin graph, enabling the reconstruction of attack paths and facilitating the analysis of attack behaviors by operations and maintenance personnel.

[0144] The historical status information of the target path can be at least one of the following: anomaly level information, historical information between the node and the network (e.g., the node has network data inflow), historical information between the node and the file (e.g., information about the node reading sensitive files), or historical information about the node calling or being called by certain processes.

[0145] For ease of explanation, we will take the historical state information as an anomaly level information as an example, and the target path as the path with the highest anomaly level among at least one path ending at this node as an example. When the historical state information is other types of information, the implementation principle of the method in this application is the same, and will not be repeated here.

[0146] Figure 1 is an exemplary schematic diagram of a system architecture of this application.

[0147] As shown in Figure 1, the system may include a streaming processing platform 100 and terminals 1 to n.

[0148] The streaming processing platform 100 can be implemented as software or hardware.

[0149] In one possible implementation, the streaming platform 100 can be deployed on a cloud-based platform, a cluster of compute nodes, a cluster of compute nodes and a cluster of storage nodes, or a cluster of compute nodes and a cluster of storage nodes. The compute cluster may include multiple compute nodes, and the storage cluster may include multiple storage nodes. The compute nodes are compute servers used for computation, and the storage nodes are storage servers used for storage.

[0150] In one possible implementation, the streaming platform 100 can also be deployed on an off-cloud device. This device can be, for example, at least one of a terminal device or an off-cloud server. The terminal device can be a personal computer, laptop, wearable device, etc. The off-cloud server can be a single server or a server cluster, etc.

[0151] Furthermore, the number of terminals shown in Figure 1 can be one or more, n≥1, and the specific number of n is not limited.

[0152] The terminal shown in Figure 1 is a device whose system is to be tested for the presence of any abnormal attacks. The streaming platform 100 shown in Figure 1 can be used for real-time detection of attack behaviors on any terminal.

[0153] Specifically, as shown in Figure 1, each terminal from terminal 1 to terminal n may be equipped with a data acquisition module 201.

[0154] The acquisition module 201 can be used to collect log data on the local terminal in real time and report the collected log data to the streaming processing platform 100 (e.g., a computing node deployed in the cloud) in real time.

[0155] The log data may include system control flow and data flow information.

[0156] For example, the log data may include system control classes and data flow information related to the entity's operations.

[0157] The entity may include, but is not limited to, at least one of the following: file entity, process entity, network entity.

[0158] In one possible implementation, the log data may include data on at least one of the following operations: file-related operations (e.g., opening a file, creating a new file, etc.), process-related operations, and network-related operations (e.g., Domain Name System (DNS) lookups, Transmission Control Protocol (TCP) connections, etc.).

[0159] As shown in Figure 1, the streaming processing platform 100 can read the log data reported by each terminal in real time, and perform abnormal behavior detection and reconstruction of the attack path of the abnormal behavior on the corresponding terminal based on the log data.

[0160] In one possible implementation, as shown in Figure 1, the streaming platform 100 may include, but is not limited to, at least one of the following modules: event frequency calculation module 101, event frequency storage module 102, abnormal path mining module 202, node data caching module 203, abnormal path alarm module 204, abnormal path aggregation module 205, etc.

[0161] Among them, the event frequency calculation module 101 and the event frequency storage module 102 are mainly used to build an offline normal behavior model.

[0162] The abnormal path mining module 202, the node data caching module 203, and the abnormal path alarm module 204 are mainly used to detect abnormal paths in real time based on label propagation according to the pre-built normal behavior model.

[0163] The abnormal path alarm module 204 can be used to output abnormal paths that have alarms.

[0164] The abnormal path aggregation module 205 is mainly used to aggregate the abnormal paths detected above that have alarms, and output the aggregated abnormal path (a local streaming origin graph) to complete the reconstruction of the attack path.

[0165] In one possible implementation, the event frequency calculation module 101 can be used to calculate the frequency of occurrence of each individual event involved in the offline log data. Here, the offline log data is a given set of relevant log data that includes only normal events.

[0166] This frequency indicates the normality of a single event occurring. In this embodiment, the single event involved in the offline log data is assumed to be a normal event, rather than the abnormal event that this application aims to detect. However, in practical applications, events involved in real-time collected log data can be divided into normal events and abnormal events, and this application aims to detect abnormal events.

[0167] The event frequency storage module 102 can be used to store the occurrence frequency of each individual event calculated by the event frequency calculation module 101 to obtain a normal behavior model, which can be stored in a database (hereinafter referred to as the baseline database).

[0168] The event frequency storage module 102 can store the occurrence frequency of each individual event in a matrix to obtain a normal behavior model (e.g., a matrix).

[0169] Before detecting and restoring abnormal paths, the streaming processing platform 100 of this application can obtain a normal behavior model through the event frequency calculation module 101 and the event frequency storage module 102. Then, the normal behavior model is used to detect and restore abnormal paths in the real-time collected log data.

[0170] In some embodiments, since events that are considered normal events can change, the offline log data used to construct the normal behavior model can be updated periodically. The periodically updated offline log data is log data related to the updated normal events.

[0171] As shown in Figure 1, the event frequency calculation module 101 and the event frequency storage module 102 can be run periodically so that the event frequency calculation module 101 can recalculate the frequency of each individual event related to the updated offline log data based on the updated offline log data (e.g., user input) and transmit the frequency of the event to the event frequency storage module 102.

[0172] The event frequency storage module 102 can update the single events and their occurrence frequencies in the pre-built normal behavior model based on the occurrence frequency of each single event periodically sent by the event frequency calculation module 101, so that the occurrence frequency of events in the normal behavior model of this application is consistent with the offline log data after the most recent update.

[0173] In one possible implementation, the event frequency calculation module 101 can calculate the frequency of a single event occurring within a certain time range from offline log data. This time range could be the most recent week, the most recent month, or a calendar year, and can be set according to requirements without limitation.

[0174] For example, if the single event could be process A opening file B, then the frequency of this single event occurring in the past week can be calculated from the offline log data.

[0175] In one possible implementation, the event frequency calculation module 101 can also calculate the frequency of a single event occurring on a group of hosts from offline log data.

[0176] The host can be a physical host, such as a terminal as shown in Figure 1. The host can also be a virtual instance (such as a virtual machine or container) running on a physical device; there are no restrictions here.

[0177] For example, if the single event could be that process A opens file B, then the frequency of this single event occurring on 10 hosts can be counted from offline log data.

[0178] In one possible implementation, the event frequency calculation module 101 can also calculate the frequency of a single event occurring on a group of hosts within a certain time range from offline log data.

[0179] Example 1

[0180] The following example 1 illustrates the process by which the streaming platform 100 constructs a normal behavior model.

[0181] This application can define a standard single event, event e i =(src i ,dst i ,rel i ), where, according to the event triple, src i Indicates event e i The main body (corresponding to the source node of the origin graph), dst i Indicates event e i The object (corresponding to the target node in the origin graph), rel i Indicates event e i The predicate (corresponding to the edge between the source node and the target node in the origin graph, i.e., the relation).

[0182] The event frequency calculation module 101 can statistically analyze offline log data and count the events e that meet certain conditions within a specific time range t on a group of hosts (the number of hosts is h). i Frequency of occurrence M ei .

[0183] Among them, event e i The above condition is that the source node of the event is src. i The target node of the event is dst i The relationship between the events is rel. i .

[0184] Where i is a positive integer, 1≤i≤n, and n is the maximum number of types of a single event appearing in the log data.

[0185] For example, event e1 is the first type of normal event: process A opened file B.

[0186] For example, event e2 is the second type of normal event: process 1 opened file 2.

[0187] For example, the event frequency calculation module 101 can use formulas 1, 2, and 3 to count each type of single event e in the offline log data. i Frequency of occurrence M ei .

[0188] In Formulas 1 and 2, j represents host j, 1≤j≤h, and the number of hosts to be counted is h.

[0189] In Formula 1, Freq(e i ) represents event e i The number of times an event occurs repeatedly in log data (also known as frequency), where the number is the event e. i The number of times (sum(src)) occurs on each host within a specific time range t. i ,dst i ,rel i The sum of (e,j,t). In the following text, Freq(e) will be used. i (abbreviated as event e) i Frequency of occurrence.

[0190] Optionally, to avoid a single event occurring on a single host... i The frequency is too high, thus affecting the understanding of event e. i Statistical frequency M ei The accuracy of the data is reduced, which in turn decreases the accuracy of the normal behavior model in the statistical analysis of this event. i The number of times sum(src) occurs on a host j within a specific time range t. i ,dst i ,rel i When the sum(src,j,t) is greater than or equal to 1, then sum(src) i ,dst i ,rel i If the number of iterations (j,t) is 1, then sum(src) = 1. i ,dst i ,rel i ,j,t)=0.

[0191] For example, event e i If process A opens file B, then on host 1, within the time range t, there are 3 events where process A opens file B. Therefore, sum(src) i ,dst i ,rel i ,1,t)=1, instead of taking the value 3.

[0192] In Formula 2, This means that event e was ignored. i dst i The event e i The number of times a data point repeats in the log data (also known as frequency). In Formula 2, "*" indicates ignored values ​​for dst. i The number of times is the event e. i 'The number of times this occurs on each host within a specific time range t, sum(src) i ,*,rel i The sum of (j,t). In the following text, Abbreviated as event e i ′ in the frequency of occurrence.

[0193] Similarly, alternatively, to avoid a single event occurring on a single host... i The frequency of ' is too high, thus affecting the understanding of event e. i Statistical frequency M ei The accuracy of the data is reduced, which in turn decreases the accuracy of the normal behavior model in the statistical analysis of this event. i The number of times (src) occurs on a host j within a specific time range t. i ,*,rel i When the sum(src,j,t) is greater than or equal to 1, then sum(src) i ,*,rel i If the number of iterations (j,t) is 1, then sum(src) = 1. i ,*,rel i ,j,t)=0.

[0194] To understand this, consider event e. i If process A opens file B, then event e i If process A opens a file, then Freq(e) i Given offline log data, calculate the number of times process A opens file B within a specific time range t on a set of hosts; then... This is a calculation of the number of times process A opened a file within a specific time range t on the aforementioned set of hosts, based on offline log data.

[0195] Referring again to Formula 3 above, the ratio between the absolute value of the calculation result of Formula 1 and the absolute value of the calculation result of Formula 2 is the frequency M of the event of process A opening file B on a group of hosts within a specific time range t, calculated for offline log data. ei .

[0196] Thus, according to Formulas 1, 2, and 3 above, we can calculate e for each type of single event that meets the conditions in the offline log data. i Frequency of occurrence M ei In the following text, M will be referred to as M. ei Abbreviated as event e i Frequency of occurrence.

[0197] Then, the event frequency calculation module 101 can calculate the frequency of each individual event e based on the offline log data. i The frequency of occurrence M ei The event frequency is transmitted to the event frequency storage module 102.

[0198] Finally, the event frequency storage module 102 can store each individual event e i The frequency of occurrence M ei It is stored as a matrix (also called a transition probability matrix), which can be stored in the baseline database as a normal behavior model.

[0199] Example 2

[0200] The following example, using Example 2, illustrates the process by which the streaming processing platform 100 processes offline log data to build a normal behavior model.

[0201] Please refer to Figure 2, which illustrates an exemplary process for constructing a normal behavior model. As shown in Figure 2, this process may include the following steps:

[0202] S101, determine the data type of each piece of data received in the log data.

[0203] In this embodiment, offline log data can be converted into an offline data stream. Similarly, in the subsequent embodiment for detecting abnormal paths using real-time log data, the real-time log data can also be converted into a real-time data stream.

[0204] A data stream may include multiple data entries arranged in chronological order. Each data entry may have a data type. The data type may be an entity type, which means that the data entry is a piece of information about an entity. Alternatively, the data type may be an event type, which means that the data entry is an event and contains event information about that event.

[0205] As mentioned above, the log data may include system control flow and data flow information related to entity-related operations.

[0206] The entity may include, but is not limited to, at least one of the following: file entity, process entity, network entity.

[0207] The types of events can include, but are not limited to, Event type and Entity type. The Entity type can be a file entity, a process entity, or a network entity.

[0208] S102a, when the data type of a received data is Entity, the entity information of that entity is written into the node information table as the node information of a node.

[0209] For example, if the data type of the received data is an entity type, such as the information of process A, then the data can carry relatively complete entity information about process A. In this case, a node A about process A can be created, and the entity information of process A can be stored in the node information table as the node information of node A.

[0210] The node information table can be a data table in a database, and there are no restrictions on the storage structure of the node information table.

[0211] S102b: When the data type of a received data is Event, extract the corresponding source node src and target node dst for that event.

[0212] For example, if the data type of a received piece of data is an event type, it means that the data is an event, such as: process A opens file B. Then, we can extract information from the source node src indicating process A, and information from the target node dst indicating file B.

[0213] However, the event information in this data entry is incomplete regarding the entity information of the source and target nodes. For example, the information about the source and target nodes in this event information may only include the node IDs. Therefore, the following step S103 needs to be executed.

[0214] S103, query the node information table to see if the source node src and the target node dst exist.

[0215] After S103, if at least one of the source node src and the target node dst is not found in the node information table, the process ends.

[0216] If the node information table does not contain both the source node (src) and the target node (dst), for example, if the node information table does not contain node information for process A, nor does it contain node information for file B; or if the node information table contains node information for process A, but not for file B; or if the node information table does not contain node information for process A, but does contain node information for file B, then it means that the complete entity information of the source node and target node in the current Event type cannot be obtained, and the process ends.

[0217] After S103, if the source node src and the target node dst exist in the node information table, S104 can be executed.

[0218] S104: Obtain the node information of the source node src and the target node dst respectively.

[0219] If a source node (e.g., process A) and a target node (e.g., file B) exist in the node information table, then the node information of the source node (e.g., process A) and the node information of the target node (e.g., file B) can be read from the node information table to obtain the entity information of the two entities in the event.

[0220] S105. Based on the node information of the source node src and the target node dst, as well as the event information of the above events, construct a single event.

[0221] The event information is the event description information of a data point (here, an event) of type Event that is currently received.

[0222] In addition, the complete entity information of the source node and the target node in the event was also obtained through the above S104.

[0223] In this way, the three types of information mentioned above can be used to construct a single event.

[0224] Optionally, in S106, the single event is generalized to obtain the generalized event e. i .

[0225] To ensure the normal behavior model (e.g., by all M) ei To ensure the transferability of the transition probability matrix (formed by the model) and avoid making the model too large, this application embodiment performs a generalization operation on the entity information corresponding to the source node (subject, also known as the starting point) and the target node (object, also known as the ending point) of the event triple. The generalization operation strategy may include, but is not limited to: abstracting nondeterministic information, removing nondeterministic information, abstracting or removing limiting information about a specific scenario, etc.

[0226] The following are generalization strategies for entity information for different types of entities:

[0227] Process-type entities: The attribute fields of process-type entities generally only retain: 1. the path to the corresponding executable file, and 2. command-line arguments.

[0228] File-related entities: mainly retain the file path, and generalize the path, removing user-related information. For example, / home / user / mediaplayer will be changed to / home / * / mediaplayer.

[0229] Network (socket) type entities: Each network connection entity contains two addresses: the source address of the source node and the destination address of the destination node, along with their respective port numbers. Generally, the external address can be retained while the internal address is removed. For example, for outward connections (such as requests sent to external servers), the source address and its port number can be removed while the destination address and its port number are retained; similarly, for inward connections (such as requests received from external servers), the destination address and its port number can be removed while the source address and its port number are retained.

[0230] The generalization configuration for entities can be customized. To broaden the applicability of the trained normal behavior model (across different hosts), fields related to host, user, and network environment need to be removed from the entity fields. For example, / home / zhenyuan / core.sh should be generalized to / home / * / core.sh, and the local IP and port of the network connection can be ignored. Specific generalization operations need to be analyzed in conjunction with the specific scenario.

[0231] S107, determine each event e in the event stream for a set of hosts within a specific time range t. i (e.g., the generalized event e) i The frequency of occurrence M ei .

[0232] For example, formulas 1 to 3 in Example 1 above can be used to generalize each event e. i Calculate its frequency M ei .

[0233] In Formula 1, how can two events be identified as the same event e? i Then, we can compare whether the source node information (e.g., entity information) of the two events is the same, whether the target node information (e.g., entity information) of the two events is the same, and whether the relationship (also called predicate) of the two events is the same. If the source node information, target node information, and relationship of the two events are all the same, then it means that they are the same single event e.i This allows for the control of a single event e. i Calculation of the frequency of occurrence.

[0234] In Formula 2, how can we determine that two events are the same event e? i Then, we can compare whether the node information (e.g., entity information) of the source nodes of the two events are the same, and whether the relationship (also called predicate) of the two events is the same. If the node information of the source nodes and the relationship of the two events are the same, then it means that they are the same event. i This allows for the tracking of event e. i Calculation of the frequency of occurrence.

[0235] Finally, using Formula 3 above, we can calculate the generalized event e for each event. i The frequency of occurrence M ei .

[0236] S108, based on each event e i Frequency of occurrence M ei Generate a normal behavior model, and optionally, write the normal behavior model into the baseline database.

[0237] Where i can be a positive integer, with no specific restrictions, and its specific value depends on the type of single event involved in the log data being processed.

[0238] In this step, multiple events can be e i The respective event occurrence frequency M ei Store it as a table to serve as the normal generative model.

[0239] Optionally, this normal behavior model can be written into the database.

[0240] In this embodiment of the application, when constructing a normal behavior model, each type of single event e in the offline log data can be statistically analyzed. i Frequency of occurrence M eiIn this process, events can first be extracted from log data, and the entity information of the events can be generalized. Then, the multiple generalized events obtained by processing the log data are compared to determine their frequency, thus obtaining the frequency of the corresponding event. Before calculating the frequency of events, the events extracted from the log data can be generalized first. This reduces the number of event types involved in the normal behavior model obtained in this application, thus avoiding the problem of an overly large normal behavior model due to excessive event frequencies stored in it. Furthermore, by generalizing the entity information of events, the single event corresponding to the frequency stored in the normal behavior model is an event generalized from the entity information, making the frequency statistics for that event applicable to other scenarios and ensuring the portability of the normal behavior model. Furthermore, in the process shown in Figure 2 above, by writing the entity information of the identified Entity type data into the node information table, when an Event type event is received subsequently, the complete node information (e.g., entity information) of the source node and the target node of the event can be obtained from the node information table, which facilitates the comparison of the generalized events (e.g., in Formula 1 above, it is necessary to compare the source node, the target node, and the relationship, while in Formula 2, it is not necessary to compare the target node).

[0241] In one possible implementation, offline log data can be updated periodically, so that the process shown in Figure 2 can be executed periodically to periodically update the occurrence frequency M of normal events in the normal behavior model. ei .

[0242] Example 3

[0243] The following example, using Example 3, describes how the streaming platform 100 uses offline log data to generate the frequency M of each individual event involved in the log data. ei The process.

[0244] Step 1 involves collecting log data from the test machine over a 3-hour period, analyzing each event in the log data, and updating the statistical data for each event.

[0245] S10, when analyzing the event / var / spool / cron / root->FILE_READ->cron, combine the above formula 1 to update the event frequency statistics (1) Freq( / var / spool / cron / root,FILE_READ,cron)+=1, and the above formula 2 to update the event frequency statistics (2) Freq( / var / spool / cron / root,FILE_READ)+=1;

[0246] Among them, the event e i The file at path / var / spool / cron / root is read by the cron process, where the file at path / var / spool / cron / root is the source node and the cron process is the target node.

[0247] Among them, the event e i The line '' indicates that the file at path / var / spool / cron / root is read, where the source node is the file at path / var / spool / cron / root, the relation rel indicates the file read operation, and the target node is omitted.

[0248] S20, when analyzing the event cron->PROCESS_FORK->bash (an event in which a process creates a child process), combine the above formula 1 to update the event frequency statistics (1) Freq(cron,PROCESS_FORK,bash)+=1, and the above formula 2 to update the event frequency statistics (2) Freq(cron,PROCESS_FORK)+=1;

[0249] S30, when analyzing the event bash->PROCESS_FORK->bash (an event in which a process creates a child process), combine the above formula 1 to update the event frequency statistics (1) Freq(bash,PROCESS_FORK,bash)+=1, and the above formula 2 to update the event frequency statistics (2) Freq(bash,PROCESS_FORK)+=1;

[0250] By analogy, all events in the log data can be processed, and the frequency of each event can be counted according to Formula 1 and Formula 2.

[0251] Step 2: Iterate through the frequency statistics (1) and frequency statistics (2) of each event in Step 1 above, and calculate the occurrence frequency of each single event according to Formula 3 above.

[0252] For events in S10, their occurrence frequency M( / var / spool / cron / root,FILE_READ,cron) = Freq( / var / spool / cron / root,FILE_READ,cron) / Freq( / var / spool / cron / root,FILE_READ);

[0253] For events in S20, their occurrence frequency M(cron,PROCESS_FORK,bash) = Freq(cron,PROCESS_FORK,bash) / Freq(cron,PROCESS_FORK);

[0254] For the events in S30 above, their occurrence frequency M(bash,PROCESS_FORK,bash) = Freq(bash,PROCESS_FORK,bash) / Freq(bash,PROCESS_FORK);

[0255] By analogy, the frequency M of all events can be calculated. ei In order to construct a normal behavior model.

[0256] Example 4

[0257] The following describes the implementation principle of the streaming processing platform 100 of this application, which uses a label propagation method to calculate the abnormal score of a path.

[0258] Formula 3 above is used to calculate the frequency of a single event (identified as a normal event). The result of Formula 3 reflects the normality of a single event. The streaming platform 100 can then use the result of Formula 3 to further calculate the normality score of any path using the principle of Formula 4, reflecting the normality of that path, and to calculate the abnormality score of that path using Formula 5, reflecting the abnormality of that path. Here, the path is the most abnormal path calculated for a single received event during the real-time detection of abnormal behavior in log data, leading to the target node of that single event.

[0259] Where RS is the normal fraction of path P, length is the length of path P (specifically, the number of individual events constituting path P), and M... ei The single event e calculated using Formula 3 above i The frequency of occurrence M ei Formula 4 represents the M for each individual event constituting path P. ei Perform cumulative multiplication to obtain the normal score for path P.

[0260] For easier understanding, please refer to the path diagram shown in Figure 3a.

[0261] For example, if the log data shows events 1 and 2 occurring sequentially in time, where event 1: file A is read by process B; event 2: process B creates child process C, then the source node of event 1 is node A (representing file A) as shown in Figure 3a, and its target node is node B (representing process B) as shown in Figure 3a. Path 1: from node A to node B, can represent event 1.

[0262] Similarly, node C shown in Figure 3a can be used as the target node of event 2 to represent process C in event 2. Path 2: from node B to node C can represent event 2.

[0263] For example, the frequency of occurrence of event 1 stored in the normal behavior model is M. e1 The frequency of event 2 is M. e2 .

[0264] Path P is path 1 + path 2, which is the path from node A through node B to node C.

[0265] The length of path P is 2 (including events 1 and 2 mentioned above). Therefore, according to formula 4, the normal score of path P is RS(P) = M. e1 *M e2 .

[0266] Formula 4 reflects the cumulative product of the normality of multiple events, where event e is the event. i The more normal, the better. The closer it is to 1, the closer RS ​​is to 1.

[0267] Intuitively, the anomaly level of a path is the cumulative product of the anomaly levels of the individual events within that path. Event e i The more abnormal, The smaller the value, the greater the abnormality of the path, and thus the smaller RS(P). Therefore, to intuitively reflect the abnormality of the path, the abnormality score AS(P) of path P can be calculated using Formula 5 to express the abnormality of path P:

[0268] AS(P) = 1 - RS(P), Formula 5;

[0269] In one possible implementation, the streaming platform 100 may use the normal score RS(P) of path P as the label of the target node (also called the endpoint, such as node C shown in Figure 3a) of path P.

[0270] In one possible implementation, referring to the principle of Formula 4, when the streaming platform 100 calculates the label of the target node in a single event, it can compare the label of the source node in the single event with the frequency M of the single event. eiPerform the product to obtain the label of the target node.

[0271] Continuing with Figure 3a as an example, after detecting event 1, for example, if node A has no label, that is, no event with node A as the target node was detected (e.g., an abnormal event), then the RS(1) of path 1 can be calculated as M according to the above formula 4. e1 , of which M e1 Let M be the frequency of occurrence of event 1 mentioned above. Then, this application can take the value M. e1 RS(1) is used as the label of the target node (node ​​B) in path 1.

[0272] Then, the streaming processing platform 100 of this application detects event 2 (which occurs after event 1) in the log data. It can then obtain the label of the source node in event 2, i.e., the label of node B, and obtain the frequency M of event 2 from the normal behavior model. e2 Therefore, when determining the label of the target node (node ​​C) in event 2, we can refer to the principle of formula 4 and compare the label of the source node (node ​​B) of event 2 with the frequency M of event 2. e2 Multiplying them together achieves the effect of cumulatively multiplying the normal fractions of path 1 and path 2 on path P, thus obtaining the normal fraction RS(P) = M of path P (the complete path formed by path 1 and path 2 as shown in Figure 3a). e1 *M e2 And the normal score RS(P) = M e1 *M e2 This serves as the label for the target node (node ​​C) in event 2. Thus, by considering the label of the source node (node ​​B) in event 2 and the frequency M of event 2 occurring, we can... e2 We perform calculations to obtain the normal score RS(P) for path P. When this normal score RS(P) is less than the normal score RS in the label of the target node (node ​​C) in event 2, we can use the label propagated from the source node (node ​​B) to update the normal score of the most abnormal path (e.g., path 1 + path 2) of the target node in the current event 2.

[0273] Of course, in some embodiments, the information stored in the label of the target node of the event can also be other information that can indicate the anomaly score of the corresponding most abnormal path, such as the label storing the anomaly score (e.g., AS(P) calculated by formula 5), ​​etc., which is not limited here.

[0274] In one possible implementation, for any path, where the arbitrary path is the most abnormal path to the target node in a single event received during real-time detection of abnormal behavior in log data, this application can calculate the anomaly score of this arbitrary path using the principle of Formula 4 and in combination with Formula 5. However, the longer the abnormal path, the higher its anomaly score. Therefore, to eliminate the interference of path length on the anomaly score calculated by this application and improve the accuracy of the anomaly score calculated for abnormal paths, this application can pre-obtain a normalization coefficient α based on a sampling method, where α > 1. This normalization coefficient α can be used to eliminate the negative impact of path length on the anomaly score of the path calculated by this application.

[0275] In one possible implementation, Formula 6 below can be used as a possible way to calculate the normal fraction of path P.

[0276] The parameters in Formula 6 are similar to those in Formula 4, and will not be repeated here. The difference is that a normalization coefficient α is added, which is a pre-calculated constant.

[0277] Continuing with Figure 3a as an example, when calculating the normal fraction RS(1) of path 1 shown in Figure 3a, according to formula 6, RS(1) = M e1 *α. Among them, the normal score RS(1) can be used as the label of node B.

[0278] Then, upon receiving event 2, RS(P) = RS(1) * M can be calculated. e2 *α, where path P in Figure 3a is path 1 + path 2.

[0279] Similarly, for subsequent received events, the normal score RS of the corresponding path is calculated according to the principle of Formula 6, and then the abnormal score of the corresponding path is obtained by using Formula 5 above.

[0280] In other words, when the streaming processing platform 100 of this application calculates the normal score of the most abnormal path ending at the target node in each real-time detected single event, it can multiply the normal score of the most abnormal path by a normalization coefficient α according to Formula 6, and then use Formula 5 to obtain the abnormal score of the most abnormal path.

[0281] Example 5

[0282] The following describes the process by which the streaming processing platform 100 of this application detects abnormal paths using a tag propagation method.

[0283] Figure 4a is a schematic diagram illustrating an exemplary detection process.

[0284] Before introducing the process, the following technical terms are explained: In the embodiments of this application, the source node and target node of a single event can each have their own label and abnormal path information associated with that label. This abnormal path information can be cached in memory. A node's label can indicate the abnormal score or normal score of a path, where the target node is the most abnormal path for that node (i.e., the path with the highest abnormal score). Optionally, the target node's label can be cached in memory using a label storage table; no specific limitation is imposed.

[0285] Specifically, in this embodiment, the label of the source node of a single event can be named a first label, the anomaly score indicated by the first label can be named a first anomaly score, and the path associated with the first label can be named a first anomaly path; and the label of the target node of the single event can be named a second label, the anomaly score indicated by the second label can be named a second anomaly score, and the path associated with the second label can be named a second anomaly path; and the frequency M of the single event occurring using the label of the source node can be... ei The calculated abnormal score of the path is named the third abnormal score, and the path with the third abnormal score is named the third path. The third path is the first abnormal path related to the first label of the source node of the single event, and the path after connecting the path of the single event.

[0286] As shown in Figure 4a, the process may include the following steps:

[0287] S201, determine event e in the real-time collected log data. i .

[0288] Taking Figure 1 as an example, terminals 1 to n can report the real-time collected log data to the streaming processing platform 100.

[0289] The anomaly path mining module 202 (or other modules not shown in the streaming processing platform 100, etc., are not limited) can determine event e in the real-time collected log data. i .

[0290] In one possible implementation, the anomaly path mining module 202 can determine event e from the received real-time log data according to the processing flow and principle shown in Figure 2 of Example 2 (S101 to S106). i Among them, the event e i For the generalized single event, please refer to the relevant introduction in Example 2 for the specific implementation, which will not be repeated here.

[0291] In this way, the anomaly path mining module 202 can detect individual events in the real-time event stream according to the chronological order of events in the log data, and process each detected individual event e i The following process is followed to detect abnormal paths (hereinafter referred to as abnormal paths). Optionally, the abnormal path can be output, and the abnormal path can be reconstructed to facilitate the detection of problems by operation and maintenance personnel.

[0292] S203, based on event e i The first tag of the source node's src attribute determines the event e. i The target node dst's second label and the second abnormal path information associated with that second label.

[0293] In this process, by propagating the label of the source node src, the label of the target node dst can be obtained using the label of the source node src, thereby determining the abnormal path related to the label of the target node dst.

[0294] The following is based on the current event e i The implementation process of S203 will be explained by considering whether the source node src and the target node dst have labels and their sizes, in conjunction with Examples 5.1 and 5.2.

[0295] Example 5.1

[0296] Figure 4b is an exemplary schematic diagram illustrating the process of initializing the label and path of the target node dst.

[0297] In the process shown in Figure 4b, the label obtained by initializing the target node is the second label in S203, and the path obtained by initializing it is the second abnormal path information in S203.

[0298] As shown in Figure 4b, the process executed by the streaming platform 100 may include the following steps:

[0299] S1001, Determine event e i The target node does not have a second label.

[0300] For example, the tag storage table in memory can be queried. If no tag information for the target node is found, it means that the target node does not have a tag, which further explains the current event e. i If the path (e.g., the path from node A to node B) is the first path detected that can reach node B, then node B needs to be assigned an initial label.

[0301] It should be understood that the source node and target node of an event do not have their own labels initially, or the node is initially assigned a label, but the initial value of the label is empty. Only when a path that can reach the node is detected will the node be assigned a label indicating the historical state information (such as the degree of abnormality) of the corresponding path or its label be updated.

[0302] S1002, based on the first tag of the source node src, determine the third anomaly score of the third path.

[0303] In S1002, the normal behavior model may be optionally incorporated to determine the third abnormal score.

[0304] In other embodiments, the occurrence frequency M of the current event can also be calculated in real time. ei Or, other information that can indicate the normality of the event.

[0305] S1004, initialize the second label of the target node dst with information indicating the third abnormal score.

[0306] S1005, initialize the second abnormal path information related to the second label of the target node dst as the information of the third path.

[0307] Optionally, S1003, in event e i If the source node does not have a first label, event e can be determined when the third anomaly score is greater than the first threshold. i S1004 to S1005 will only be executed if it is an abnormal event.

[0308] For example, the third path is for the single event e. i The first abnormal path associated with the first label of the source node, and the single event e i The path after the connection.

[0309] With event e i Taking event 1, which involves moving from node A to node B, as an example, process A opens file B. Here, process A is the source node, and file B is the target node.

[0310] Referring to Figure 1, the anomaly path mining module 202 can first query the event e from the event frequency storage module 102. i Frequency of occurrence M ei For example, event e i The frequency of occurrence is M e1 .

[0311] Based on event e i Whether the source node has a label makes the implementation of S1002 fall into two categories.

[0312] Please refer to Figure 3b(1), Case 1: Node A has no label and Node B also has no label.

[0313] Therefore, when calculating the outlier score for path AB (an example of a third path), M can be used. e1 Calculate the normal score RS = M for path AB using formula 4 above (here, length = 1). e1 .

[0314] This application also presets a label threshold (an example of the first threshold mentioned above). In this example, the label threshold is a threshold for the normal score indicated by the label, such as 0.35, or other data.

[0315] The size of the label threshold affects the number of labels initialized during real-time anomaly detection in this application. When the label threshold is the threshold indicating the normal score, a larger label threshold results in more initialized labels, but also higher computational and memory overhead. However, a small label threshold may lead to missed detections of abnormal events. Therefore, this application reasonably sets it to a value around 0.35, for example, a value within the range of 0.2 to 0.45. The specific value is not limited to this range and can be flexibly set according to the required detection accuracy of abnormal behavior and the computational and memory overhead requirements of the system.

[0316] In other embodiments, the label threshold can also be a threshold for the anomaly score indicated by the label. In this case, the calculated AS is compared with the label threshold. If AS is greater than or equal to the label threshold, then the event e is considered to have occurred. i For abnormal events, conversely, event e i That is a normal event.

[0317] The normal score RS = M is obtained for path AB. e1 Afterwards, due to the event e i Since the source node lacks label information, its calculated RS needs to be compared with a label threshold (e.g., 0.35). If the RS is greater than or equal to 0.35, then the event e is considered valid. i If it's a normal event, the processing for that event ends, and you can continue monitoring for the next single event in the real-time log data. Conversely, if the RS is less than 0.35, it indicates that event e... i If it is an abnormal event, then the event e needs to be handled. i The target node in the dataset is node B, which is assigned a label and a related abnormal path.

[0318] As shown in Figure 3b(2), a label can be assigned to node B, with Tag_B being M. e1In addition, the most abnormal path information related to the label of node B can be initialized as: event 1 recorded in chronological order from early to late. The path corresponding to this most abnormal path information is: A->B. There may be information indicating the relationship between the two nodes in this path.

[0319] In this embodiment, the label of the node is the normal score of the corresponding path as an example. In other embodiments, the label may also be information indicating the normal score or information indicating the abnormal score of the path. There is no limitation here.

[0320] Please refer to Figure 3c(1), Case 2: Node A has a label, and Node B does not have a label.

[0321] As shown in Figure 3c(1), the method of this application has previously detected event 3 consisting of path MA, and has already initialized the label of node A of the event, for example, Tag_A is M. e3 , of which M e3 This represents the frequency of event 3 in the normal behavior model.

[0322] Therefore, the currently detected event e i The source node, i.e., node A, has a label, but node B does not have a label.

[0323] Then, according to Formula 4 above, the RS of the path MAB (an example of a third path) from path MA to path AB can be calculated as the label of the source node (node ​​A) and the frequency of occurrence of event e1. e1 The product of, i.e., RS = M e3 *M e1 Since the source node, i.e., node A, has a label, the label of node B can be directly initialized as shown in Figure 3c(2), where Tag_B is M. e3 *M 31 Furthermore, the second abnormal path information related to the label of node B can be initialized as third path information, such as events 3 and 1 recorded in chronological order. This third path information records a list of events, and the events in the list can be sorted in chronological order to facilitate the reconstruction of the most abnormal path. The path corresponding to the most abnormal path information recorded here is: M->A->B, where information indicating the relationship between the nodes can exist between the two nodes in this path.

[0324] Based on the modules in Figure 1 above, the anomaly path mining module 202 can extract the current event using the reported log data and read the occurrence frequency of the event from the event frequency storage module 102. Then, based on the label of the source node of the event, it calculates the normal score of the third path and uses this normal score to obtain the anomaly score of the third path. Then, the node data caching module 203 can store the anomaly score of the third path in memory as the label of the target node of the event, and store the path information of the third path in memory as path information related to the label of the target node. In this way, only a portion of the paths in the streaming origin graph containing anomalies are constructed in memory, and these paths are the paths containing anomalies.

[0325] In the above embodiments, the abnormal path mining module 202 queries the normal behavior model to find the current event e. i Taking the frequency of occurrence of event e as an example, this can be explained by... i An event that falls within the normal behavior model is also called a known event.

[0326] However, in one possible implementation, the normal behavior model within the event frequency storage module 102 does not contain information about event e. i When the frequency of occurrence is high, it indicates that event e i Since it is an unknown event, this application can set the occurrence frequency of unknown events to a frequency threshold (UNSEEN_EVENT_SCORE), for example, 0.1. Of course, this application does not impose any restrictions on this; it can be configured as needed to ensure it can be compared with the M of known events. e The values ​​are distinguished.

[0327] Among them, M represents events that did not appear in the baseline database (which stores normal behavior models). e Value (UNSEEN_EVENT_SCORE): This value should not be too small to avoid insufficient information collected from the baseline database, and it also needs to have a certain degree of differentiation from the events (also referred to as known events) in the baseline database. In practice, the M value for unseen events can be set to... ei The value is set to 0.1, and then this M is used. ei Use formula 4 or formula 6 above to calculate the normal score RS for the event.

[0328] In one possible implementation, in order to distinguish the frequency of occurrence of events (also known as known events) in the baseline database from the frequency of occurrence of the unknown event, this application can perform analysis on each known event e in the baseline database. i The frequency of occurrence M ei Process according to Formula 7, and use the processed M. eiM' is used as the normal fraction RS for calculating the path in Formula 4 or Formula 6. ei .

[0329] M ei ′=M ei *0.7+0.3, formula 7;

[0330] In this way, the frequency of occurrence of the known event can be made equal to the frequency M after processing according to Formula 7. ei The value of ′ is in the range of (0.3-1] and is somewhat different from that of unknown events.

[0331] In one possible implementation, the normal behavior model within the event frequency storage module 102 contains information about event e. i When the frequency of occurrence is high, it indicates that event e i If it is a known event, then the event can be read from the normal behavior model (e.g., a baseline database). i The frequency of occurrence M ei Optionally, the frequency M can be... ei Calculate M according to Formula 7 above. ei ′ to be the event e i The frequency of occurrence M ei This is used to calculate event e according to formula 4 or formula 6. i The normal score RS for the path AB.

[0332] Example 5.2

[0333] Figure 4c illustrates an example of event e. i The diagram illustrates the process of determining the latest label and the latest most abnormal path of the target node dst, which currently has a label.

[0334] In the process shown in Figure 4c, the updated label for the target node is the second label in S203 of Figure 4a, and the updated second abnormal path information for the target node is the second abnormal path information related to the second label in S203.

[0335] Example 5.2 is largely the same in implementation or principle as Example 5.1 above. The main difference is: Difference 1, in Example 5.2, the target node of the detected event already has a label. Therefore, it is necessary to determine whether to use the calculated third anomaly score to update the target node's label instead of initializing it. Specifically: Since the detected event in Example 5.2 already has a label, after calculating the third anomaly score of the third path according to the label propagation method of the source node in this application, such as according to Formulas 4 and 5 above, or Formulas 6 and 5, if the third anomaly score is greater than the second anomaly score indicated by the current label of the target node, then the label of the target node needs to be updated, as well as the most anomaly path information related to the label needs to be updated, instead of directly initializing the calculated third path anomaly score and anomaly path information as the target node's label and anomaly path as in Example 5.1. Thus, by using a streaming origin graph, a local origin graph is gradually constructed based on the found most anomaly path, reducing memory usage and computational overhead of the origin graph.

[0336] As shown in Figure 4c, the process executed by the streaming platform 100 may include the following steps:

[0337] S2001, Determine event e i The target node has a second label.

[0338] For example, the abnormal path mining module 202 can query the tag storage table in memory. When the tag information of the target node is found, it indicates that the target node has a tag, which further indicates that this application has detected an event e that can reach the current node. i The abnormal path of the target node, and the event e i The label (second label) of the target node stores the normal score (or abnormal score) of the most abnormal path ending at the target node. In addition, the path information (second path information) of the most abnormal path is also stored in memory.

[0339] S2002, based on the first tag of the source node src, determine the third anomaly score of the third path.

[0340] In S2002, the third abnormal score can be determined by optionally incorporating a normal behavior model.

[0341] S2004, when the third anomaly score is greater than the second anomaly score indicated by the second label of the target node dst, the second label of the target node dst is updated based on the third anomaly score.

[0342] S2005, when the third abnormal score is greater than the second abnormal score, the second abnormal path information related to the second label of the target node dst before the update is updated to the third path information.

[0343] Optionally, S2003, in event e i If the source node does not have a first label, event e can be determined when the third anomaly score is greater than the first threshold. i S2004 to S2005 will only be executed if it is an abnormal event.

[0344] For example, the third path is for the single event e. i The first abnormal path associated with the first label of the source node, and the single event e i The path after the connection.

[0345] With event e i Taking event 1, which involves moving from node A to node B, as an example, process A opens file B. Here, process A is the source node, and file B is the target node.

[0346] Referring to Figure 1, the anomaly path mining module 202 can first query the event e from the event frequency storage module 102. i Frequency of occurrence M ei For example, event e i The frequency of occurrence is M e1 .

[0347] Based on event e i Whether the source node has a label makes the implementation of S2002 fall into two categories.

[0348] Please refer to Figure 3d(1), Case 1: Node A has no label, and Node B has a label.

[0349] As shown in Figure 3d(1), this application previously detected event 4 with path CB, causing node B to have label Tag_B as M. e4 .

[0350] Then, this application detects event 1 with path AB, and the occurrence frequency of event 1 can be obtained from the normal behavior model as M. e1 Since node A has no label, the normal score RS = M for path AB can be calculated using formula 4 above (where length = 1). e1 .

[0351] This application also presets a label threshold (an example of the first threshold mentioned above). In this example, the label threshold is a threshold for the normal score indicated by the label, such as 0.35, or other data.

[0352] The size of the label threshold affects the number of labels initialized during real-time anomaly detection in this application. When the label threshold is the threshold indicating the normal score, a larger label threshold results in more initialized labels, but also higher computational and memory overhead. However, a small label threshold may lead to missed detections of abnormal events. Therefore, this application reasonably sets it to a value around 0.35, for example, a value within the range of 0.2 to 0.45. The specific value is not limited to this range and can be flexibly set according to the required detection accuracy of abnormal behavior and the computational and memory overhead requirements of the system.

[0353] The normal score RS = M is obtained for path AB. e1 Afterwards, due to the event e i Since the source node lacks label information, its calculated RS needs to be compared with a label threshold (e.g., 0.35). If the RS is greater than or equal to 0.35, then the event e is considered valid. i If it's a normal event, the processing for that event ends, and you can continue monitoring for the next single event in the real-time log data. Conversely, if the RS is less than 0.35, it indicates that event e... i If it is an abnormal event, then the event e needs to be handled. i The target node in the table is, here, the updater label of node B and the related abnormal path.

[0354] Next, RS (where M is a constant) can be compared. e1 ), and Tag_B (for M) e4 The size relationship between them, for example, M e1 <M e4 This means that the normal score of path AB is less than the normal score of path CB, thus indicating that the abnormal score of path AB is (1-M). e1 The anomaly score (1-M) greater than path CB e4 Thus, path AB is more abnormal than the path indicated by the label of node B. Therefore, the abnormal path mining module 202, as shown in Figure 1, can detect RS=M e1 The data caching module 203 of the node notifies the path AB (e.g., event 1) of the node's data caching module 203 of the tag Tag_B=M cached in memory for node B. e4 Refresh to Tag_B=M e1 And update the cached information of the most unusual path associated with the tag_B in memory from event 4 to event 1, where event 1 can indicate path AB.

[0355] Conversely, if M e1 ≥M 34If the anomaly score of path AB is less than or equal to the anomaly score of path CB as shown in Figure 3d, then there is no need to update the label of node B and the anomaly path information associated with that label.

[0356] In this embodiment, the label of the node is the normal score of the corresponding path as an example. In other embodiments, the label may also be information indicating the normal score, information indicating the abnormal score of the path, or other historical state information. There are no restrictions here.

[0357] Please refer to Figure 3e(1), Case 2: Node A has a label, and node B also has a label.

[0358] As shown in Figure 3e(1), the method of this application has previously detected event 3 consisting of path MA, and has already initialized the label of node A of the event, for example, Tag_A is M. 33 For example, M e3 This represents the frequency of event 3 in the normal behavior model.

[0359] Therefore, the currently detected event e i The source node, i.e., node A, has a label.

[0360] Furthermore, as shown in Figure 3e(1), the method of this application has previously detected event 4 formed by path CB, and has already initialized and set the label of node B of event 4, for example, Tag_B is M. e4 For example, M e4 This represents the frequency of event 4 in the normal behavior model.

[0361] Therefore, the currently detected event e i The target node, node B, also has a label.

[0362] Then, the abnormal path mining module 202 can calculate the RS of the path MAB (an example of a third path) from path MA to path AB according to the above formula 4, which is the label of the source node (node ​​A) and the occurrence frequency M of event 1. e1 The product of these two factors yields the normal score RS = M for path MAB. e3 *M e1 .

[0363] Next, RS=M can be compared. e3 *M e1 , with Tag_B=M e4 The size relationship between them, for example, M e3 *M e1 <M e4This indicates that the normal score of path MAB is less than the normal score of path CB, thus indicating that the abnormal score of path MAB (1-M) is less than the abnormal score of path CB. e3 *M e1 The anomaly score (1-M) greater than path CB e4 Thus, path MAB is more abnormal than the path indicated by the label of node B. Therefore, as shown in Figure 3e(2) and Figure 1, the abnormal path mining module 202 can detect RS=M e3 *M e1 The notification is sent to the node's data caching module 203, which can then store the cached tag Tag_B=M for node B in memory. e4 Refresh to Tag_B=M e3 *M e1 .

[0364] Furthermore, as mentioned above, the memory also stores the most abnormal path information related to the tag_A of node A. This most abnormal path is path MA as shown in Figure 3e (the recording method can be recording information about event 3). The recorded information about event 3 may include, but is not limited to: the source node is node M, the target node is node A, and the relationship between node M and node A. Then, as shown in Figure 1, the abnormal path mining module 202 can also notify the node's data caching module 203 to connect event 1 and the tag-related most abnormal path information of node A to form the tag-related most abnormal path of node B. In this way, the tag-related most abnormal path information of node B can be updated from path CB to path MAB.

[0365] In short, the updated label-related path information at node B is: the label-related path information of the source node of the current event 1, and the result of the connection with the path of event 1. From the perspective of events, this path information consists of event 3 with path MA and event 1 with path AB, recorded in chronological order.

[0366] Returning to Figure 4a, optionally, after S201 and before S203, the process may also include S202 for reducing the label of the source node of the currently detected event.

[0367] S202, in event e i When the first tag of the source node src or the first abnormal path information related to the first tag meets the first preset condition, the first tag of the source node src and the first abnormal path information related to the first tag are removed.

[0368] Among them, event e i The length of the most unusual path of the source node src can indicate the number of tag propagation rounds;

[0369] If the number of propagation rounds of the tag exceeds a preset threshold, it indicates that the first abnormal path information meets the first preset condition. Therefore, the tag of the source node and the most abnormal path information indicated by the RS in the tag can be deleted. This reduces memory usage and prevents excessive false alarms caused by excessive tag propagation.

[0370] Furthermore, in event e i If the RS (normal score) stored in the first label of a source node is greater than a preset normal score threshold, it indicates that the path associated with that first label is not abnormal enough; or if the AS indicated by that first label is less than a preset abnormal score threshold, it indicates that the path associated with that first label is not abnormal enough. Both cases indicate that the first label of the source node meets the preset conditions. Therefore, the label of the source node and the information of the most abnormal path indicated by the RS in that label can be deleted. This reduces memory usage and prevents the propagation of labels with normal scores (RS) greater than the preset normal score threshold.

[0371] In some embodiments, this application may also determine the value of α and M. ei This allows us to infer the average distance the tag travels.

[0372] Additionally, if the initialization time (i.e., the time when the node's label was initialized) of a node's cached labels exceeds a preset time threshold from the current time, it indicates that the label for that node has been created for too long, thus meeting the third preset condition. In this case, the content of the cached label for that node meeting the third preset condition can be cleared, or the label can be deleted. Furthermore, any abnormal path information associated with that label can also be removed from memory. Optionally, the initialization time (or creation time) of each node's label in the cache can be periodically checked to see if the difference from the current time is too large, thereby cleaning up labels that have been created for too long and clearing any abnormal path information associated with those labels.

[0373] The preset time threshold is the alarm corruption time threshold (DECAY_TIME_THRESHOLD). To cache a large number of tags in the tag database and to avoid outdated tags interfering with detection, old tags can be removed. For example, the preset time threshold can be set to 30*60*1000000ms.

[0374] Generally, attack behavior does not mix with a lot of normal behavior. When the path is long and the alarm conditions have not been reached, the score of normal processes can be prevented from being polluted by clearing the label.

[0375] Returning to Figure 4a, optionally, after S203, the process may also include S204 for reducing the label of the target node of the currently detected event.

[0376] It should be understood that the steps S202 above, which reduces the label of the source node, and S204 here, which reduces the label of the target node of the event, can be executed selectively as needed, and do not both need to be executed.

[0377] S204, in event e i When the second label of the target node dst or the second abnormal path information related to the second label meets the second preset condition, the second label of the target node dst and the second abnormal path information related to the second label are removed.

[0378] The implementation principle of S204 is the same as that of S202 above. The only difference is that the second preset condition in S204 can be the same as or different from the first preset condition in S202. It can be flexibly configured according to needs and scenarios.

[0379] Returning to Figure 4a, after S203, the method may also include S205.

[0380] S205, when the second label of the target node dst meets the alarm conditions, output the second abnormal path in the streaming origin graph.

[0381] As described above, in the embodiments of Figures 4b and 4c, the second abnormal path indicated by the second label of the target node dst can be initialized or updated to a third path calculated for the current event. Therefore, the second abnormal path here can be the third path. The third path is the path formed by the first abnormal path information related to the first label of the source node of the event and the path of the event. The second abnormal score indicated by the second label of the target node dst is also updated to the third abnormal score of the third path.

[0382] For example, if the label of the target node dst stores the normal score RS of the corresponding third path, then when determining whether to alarm for the third path, it can be determined whether the normal score RS is less than the preset normal score alarm threshold. If the normal score RS stored in the label is less than the preset normal score alarm threshold, it means that the label of the target node meets the alarm conditions, and the second abnormal path in the streaming origin graph can be output.

[0383] For example, the preset alarm threshold can also be a preset abnormal score alarm threshold. When determining whether an alarm needs to be triggered for the third path, the normal score RS indicated by the second tag of the target node can be calculated according to the above formula 5. Then, it can be determined whether the AS is greater than the preset abnormal score alarm threshold. If the AS is greater than the preset abnormal score alarm threshold, it means that the tag of the target node meets the alarm conditions, and the second abnormal path in the streaming origin graph can be output. The larger the preset abnormal score alarm threshold is, the more difficult it is to trigger an alarm. The threshold in the alarm conditions can be flexibly set according to the need to trigger an alarm, such as a preset normal score alarm threshold or a preset abnormal score alarm threshold, etc., which indicate the alarm conditions.

[0384] As shown in Figure 1, the abnormal path mining module 202 can notify the abnormal path alarm module 204 of the most abnormal path (i.e., the second abnormal path information) related to the second label of the target node whose second label meets the alarm conditions.

[0385] The abnormal path alarm module 204 can obtain the abnormal paths with alarms from the node's data cache module 203.

[0386] This application caches the most abnormal path information related to tags (such as the second abnormal path related to the second tag) in memory, thereby caching the tag transmission history information in memory. When an alarm occurs, it indicates that abnormal behavior has been detected during real-time detection. The most abnormal path can be quickly restored based on the cached most abnormal path information related to the tag with the alarm, thus realizing the restoration of the attack path without having to re-query massive amounts of logs.

[0387] When outputting the second abnormal path, this application can output each node in the second abnormal path, as well as the relationships between each node.

[0388] The second abnormal path can be output in the form of text or a streaming origin graph; there are no restrictions on this.

[0389] For ease of explanation, this application may output the second abnormal path with alarms in the form of an alarm graph (e.g., a graph format of a streaming origin graph).

[0390] In one possible implementation, the abnormal path alarm module 204 may also notify the abnormal path with alarms to the abnormal path aggregation module 205. The abnormal path aggregation module 205 may aggregate the second abnormal path with alarms and the local streaming origin graphs of candidate nodes in the two abnormal paths to obtain an aggregated local streaming origin graph. The candidate nodes are nodes whose labels meet the alarm conditions. The aggregated local streaming origin graph is output as a local streaming origin graph related to the endpoint of the target event.

[0391] For example, upon receiving an alarm, the most abnormal path related to the second tag of the alarm (i.e., the second abnormal path mentioned above) can be obtained. This second abnormal path is the path with the highest abnormal score AS that is currently detected and terminates at the target node. This second abnormal path is represented by singleAlertPath.

[0392] Then, the abnormal path aggregation module 205 can traverse the nodes in the traversal path singleAlertPath. For example, when traversing to node 1, the path singleAlertPath is added to the alarm graph 1 (an example of a local streaming origin graph of a candidate node) where node 1 is located, and the aggregated alarm graph 1' is obtained.

[0393] Then, the abnormal path aggregation module 205 traverses to node 2 in the path singleAlertPath, and can aggregate the alarm graph 2 (an example of a local streaming origin graph of a candidate node) where node 2 is located with the above alarm graph 1' to obtain alarm graph 1".

[0394] Next, the abnormal path aggregation module 205 traverses to node 3 in the path singleAlertPath, and then aggregates the alarm graph 3 (an example of a local streaming origin graph of a candidate node) containing node 3 with the above-mentioned Figure 1” to obtain Figure 1”':

[0395] This process continues until all nodes in the path singleAlertPath have been traversed and the corresponding aggregation operations have been completed, resulting in the final target aggregation graph M (an example of an aggregated local streaming origin graph).

[0396] Optionally, each node in the target aggregation graph M is traversed, and the alarm graph to which each node belongs is updated to the target aggregation graph M.

[0397] Finally, the target aggregate graph M can be output.

[0398] In the above process, the alarm graph of each node represents a local streaming origin graph consisting of the abnormal path related to the label of a target node with an alarm, and the path formed by aggregating with other abnormal paths with alarms.

[0399] The tag propagation path detection method of this application may include a storage structure and four logic components. The storage structure is a tag cache data structure, and the four logic components are: tag initialization logic, tag propagation logic, tag reduction logic, and alarm triggering logic. This process utilizes the association analysis characteristics of a graph, but does not require the complete construction of the origin graph in memory, thus saving memory overhead. Simultaneously, the processing results of each event are cached, avoiding redundant calculations and thus saving computational overhead. This type of algorithm can be modified to adapt to different computational scenarios. For the specific problem of abnormal path detection, the information stored within the tag can be the abnormal score or the normal score of the path, and the most abnormal path with that abnormal score or normal score is cached for the target node.

[0400] By employing a streaming tag propagation method, local anomaly scores are propagated and aggregated along the path, enabling fast, lightweight, and real-time anomaly detection. By caching tags to the path with the highest anomaly severity, rapid attack path reconstruction can be achieved based on the cache when anomalies are detected in real time, making it both lightweight and efficient.

[0401] Figure 1 is a schematic diagram illustrating the structure of an exemplary streaming processing platform 100. It should be understood that the streaming processing platform 100 shown in Figure 1 is merely an example; the system of this application may have more or fewer modules than shown in the figure, may combine two or more modules, or may have different module configurations. The various modules shown in Figure 1 can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0402] Example 6

[0403] The implementation process of the method in this application will be explained by combining specific events detected in real-time from real-time log data. Taking a normalization coefficient α of 2 as an example, other embodiments can be flexibly set according to the scenario.

[0404] a) Analyzing the event `redis->FILE_WRITE-> / var / spool / cron / root` [the process wrote a file], neither the source nor the target node in this event has a label. Since the event's frequency was not found in the offline normal behavior model, it is an unknown event. Therefore, the normal score `RS` for this unknown event is assigned as `UNSEEN_EVENT_SCORE = 0.1`. The normal score `RS` for this event is 0.1 < 0.35, where 0.35 is the label threshold for a single event. Therefore, this event is an abnormal event, and the label and path need to be written to the target node ` / var / spool / cron / root`.

[0405] The source node of the event is the Redis process, which has no label, so its anomaly score is 0. The label in the / var / spool / cron / root file stores the most abnormal path redis->FILE_WRITE-> / var / spool / cron / root. The normal score RS of the anomalyPath is 0.1, and the abnormal score AS = 1 - 0.1 = 0.9. Since it does not exceed the anomaly score alarm threshold ALERT_THRESHOLD (which is 0.95 here), no alarm is triggered.

[0406] b) Analyze the event / var / spool / cron / root->FILE_READ->cron. In the offline normal behavior model, the occurrence frequency of this event is found to be M( / var / spool / cron / root,FILE_READ,cron) = 0.6. The normal score of this event is RS = 0.6*0.7+0.3 = 0.72 (calculated according to formula 7 above). The source node's label / var / spool / cron / root (a above) with RS = 0.1 is propagated to the target node's cron process. The normal score RS for the anomaly path is calculated as 0.1 * 0.72 * α = 0.144, and the anomaly score AS = 1 - 0.144 = 0.856, which does not exceed ALERT_THRESHOLD (e.g., 0.95), so no alarm is triggered. The anomaly path is cached in the target node's cron label, here it is 0.144, and the most abnormal path is cached as follows: redis->FILE_WRITE-> / var / spool / cron / root->FILE_READ->cron.

[0407] c) Analyze the event cron->PROCESS_FORK->bash. In the offline normal behavior model, query M(cron,PROCESS_FORK,bash) = 0.2. The normal score for this event is RS = 0.2 * 0.7 + 0.3 = 0.44. Propagate the cron tag to the bash process. Calculate the normal score RS for anomalyPath as 0.144 * 0.44 * α = 0.127. The abnormal score AS = 1 - 0.127 = 0.873, which does not exceed ALERT_THRESHOLD (0.95). No alarm is triggered, and the anomalyPath is cached in the bash tag: redis->FILE_WRITE-> / var / spool / cron / root->FILE_READ->cron->PROCESS_FORK->bash.

[0408] d) Analyzing the event bash->PROCESS_EXEC->curl, the normal score RS = UNSEEN_EVENT_SCORE = 0.1 for the event M(bash,PROCESS_EXEC,curl) was not found in the offline normal behavior model. Propagating the bash tag to the curl process, the normal score RS for anomalyPath is calculated to be 0.127 * 0.1 * α (2) = 0.0254, the anomaly score AS = 1 - 0.0254 = 0.9746, which exceeds ALERT_THRESHOLD (0.95), so an alarm is triggered and the anomalyPath is cached in the curl process label: redis->FILE_WRITE-> / var / spool / cron / root->FILE_READ->cron->PROCESS_FORK->bash->PROCESS_EXEC->curl.

[0409] This process is repeated for each event, and real-time alerts are issued for processes whose anomalyPath score in the tag exceeds ALERT_THRESHOLD.

[0410] Each event's anomalyPath is analyzed for label propagation and caching. Processes with anomaly scores exceeding ALERT_THRESHOLD are detected in real time and alerts are issued.

[0411] When an abnormal score of the curl process is detected to exceed the alarm threshold, the attack path is reconstructed based on the anomalyPath cached in the curl label: redis->FILE_WRITE-> / var / spool / cron / root->FILE_READ->cron->PROCESS_FORK->bash->PROCESS_EXEC->curl, and a visualized alarm graph (a local bleed origin graph) as shown in Figure 5 is generated.

[0412] In the above embodiments, the list of events to be detected in real time is configurable; for example, it can include at least one of processes, networks, and files. As an anomaly detection scheme, this scheme can selectively analyze anomalies in parts of the event chain. For example, if only process-related events are considered, then the method and system of this application are an anomaly detection system targeting the process tree. Generally speaking, the more event types considered, the stronger the detection capability, but the greater the overhead. In practice, it is recommended to parse events such as process establishment, file creation and reading / writing, and network connection and data sending / receiving.

[0413] This application designs a real-time anomaly path discovery system based on origin graph and label propagation technology. It utilizes the correlation analysis capabilities of the origin graph to associate scattered anomaly events and construct paths for anomalous behaviors. The value of this path construction for detection lies primarily in two aspects: 1) Real-time attack detection: Correlation analysis can aggregate smaller, localized anomalies into larger ones, discovering hidden suspicious behaviors while avoiding numerous false alarms; 2) Real-time attack path reconstruction: The anomaly path itself reflects the attack path, helping analysts understand the attack behavior and take appropriate action.

[0414] Furthermore, this application configures a label propagation framework based on the origin graph. Based on security experience, it designs a specific implementation algorithm for anomaly label propagation, using local anomaly scores as node labels to propagate and aggregate along the path of the origin graph. Anomaly detection is performed in real time based on the anomaly scores, and an alarm is generated in real time when the anomaly score reaches a set threshold. During anomaly label propagation, the cached anomaly path with the highest anomaly score to each node is updated in real time. Each node has multiple paths to it; one branch leads to a node, and another branch leads to that node. Each node has cached information storing the path corresponding to the anomaly score in the label. If another path to that node causes its anomaly score to exceed the current score, the node's cached information is updated to the path with the highest anomaly score. When an anomaly is detected, the attack path (the path with the highest anomaly score) can be reconstructed in real time based on the node's cached data, and multiple anomaly paths can be merged to generate a complete attack graph.

[0415] This application provides a data processing device 800. Optionally, the device 800 can be deployed on a cloud management platform. The cloud management platform is used to manage the infrastructure that provides cloud services. The infrastructure includes multiple cloud data centers located in different regions, with at least one cloud data center in each region.

[0416] Figure 6 is a schematic diagram of the structure of an exemplary data processing device 800. Referring to Figure 6, the data processing device 800 includes:

[0417] In one possible implementation, this application provides a data processing apparatus. The apparatus may include: a first determining module 801, configured to determine a target event to be detected based on real-time collected log data, the target event including a start point, an end point, and a relationship between the start point and the end point; a second determining module 802, configured to determine a second label for the end point based on a first label of the start point; wherein, the label of each node in the target event is used to indicate historical state information of a target path ending at that node; wherein, the target path is one of at least one paths ending at that node, wherein the at least one path is a path composed of at least one event to be detected determined based on the log data; wherein, the nodes in the target event include the start point and the end point of the target event; a third determining module 803, configured to determine second path information related to the second label based on first path information related to the first label; and an alarm module 804, configured to output a local streaming origin graph related to the end point of the target event based on the second path information when the second label meets an alarm condition.

[0418] In one possible implementation, the device further includes: a fourth determining module, configured to determine the normality level of the target event; and a second determining module 802, specifically configured to: determine historical state information of a third path based on a first label of the starting point and the normality level of the target event; wherein the third path is a path composed of the first path information and the target event; and determine a second label of the endpoint based on the historical state information of the third path.

[0419] In one possible implementation, the fourth determining module is specifically used to determine the normality level of the target event based on a pre-built information table; wherein the information table includes first information indicating the normality level of the preset event.

[0420] In one possible implementation, the fourth determining module is specifically used to determine the normality of the target event based on first information in the information table indicating the normality of the target event when the target event is the preset event in the information table.

[0421] In one possible implementation, the fourth determining module is specifically configured to set the preset normality threshold as the normality of the target event when the target event does not belong to the preset event in the information table.

[0422] In one possible implementation, the second determining module 802 is specifically used to determine the historical state information of the third path based on the normality of the target event when the first label of the starting point is empty data and the normality of the target event meets the first preset condition.

[0423] In one possible implementation, the historical state information includes anomaly degree information, and the second determining module 802 is specifically used to: determine the normality of the target path indicated by the first tag based on the first tag of the starting point; and determine the anomaly degree information of the third path based on the normality of the target path and the normality of the target event.

[0424] In one possible implementation, the second determining module 802 is specifically configured to initialize the second label of the endpoint to indicate the historical status information of the third path when the second label of the endpoint is empty data.

[0425] In one possible implementation, the third determining module 803 is specifically used to initialize the second path information related to the second tag of the endpoint as the information of the third path when it is determined that the first path information is empty data.

[0426] In one possible implementation, the second determining module 802 is specifically used to update the second label of the endpoint to information indicating the degree of abnormality of the third path when the second label of the endpoint is not empty data and the degree of abnormality of the third path is greater than the degree of abnormality of the target path indicated by the second label.

[0427] In one possible implementation, the third determining module 803 is specifically used to determine, based on the historical status information of the third path and the historical status information of the target path indicated by the second tag, that when it is necessary to refresh the second tag, update the second tag of the destination to the historical status information indicating the third path.

[0428] In one possible implementation, the first tag and the first path information, the second tag and the second path information are all cached in memory.

[0429] In one possible implementation, the device further includes a deletion module, configured to delete the information of the first tag and the first path information cached in memory when at least one of the first tag at the starting point and the first path information associated with the first tag satisfies a second preset condition.

[0430] In one possible implementation, the deletion module is specifically used to delete the information of the first tag and the first path information cached in memory when the historical state information of the target path indicated by the first tag at the starting point meets the tag reduction condition.

[0431] In one possible implementation, the deletion module is specifically used to delete the information of the first tag and the first path information cached in memory when the path length of the first path information associated with the first tag is greater than a preset length threshold.

[0432] In one possible implementation, the historical state information includes anomaly degree information; the alarm module 804 is specifically used to output a local streaming origin graph related to the endpoint of the target event based on the second path information when the anomaly degree information of the target path indicated by the second tag is greater than a third preset anomaly degree threshold.

[0433] In one possible implementation, the apparatus further includes: an aggregation module, configured to aggregate the second path information and the target node in the second path information with a local streaming origin graph to obtain an aggregated local streaming origin graph; wherein the target node is a node whose label satisfies the alarm condition; and an output module, configured to output the aggregated local streaming origin graph as a local streaming origin graph related to the endpoint of the target event.

[0434] In one possible implementation, the apparatus further includes an update module, configured to update the local streaming origin graph of each node in the aggregated local streaming origin graph to the aggregated local streaming origin graph.

[0435] The effects of the data processing apparatus 800 in the above embodiments are similar to those of the data processing methods in the above embodiments, and will not be repeated here.

[0436] All of the above modules can be implemented in software or hardware. As an example of a software functional unit, a module may include code running on a computing instance. A computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, there may be one or more computing instances. For example, the first determining module 801 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0437] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0438] As an example of a hardware functional unit, the module mentioned above may include at least one computing device, such as a server. Alternatively, the module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0439] The aforementioned data processing apparatus includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the multiple computing devices can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0440] It should be noted that, in other embodiments, the above-described modules can be used to execute corresponding steps in the data processing method to realize all the functions of the data processing device.

[0441] This application also provides a computing device 900. As shown in FIG7, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 909. The processor 904, the memory 906, and the communication interface 909 communicate with each other via the bus 902. The computing device 900 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0442] Bus 902 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 7, but this does not imply that there is only one bus or one type of bus. Bus 902 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, communication interface 909).

[0443] Processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0444] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0445] The memory 906 stores executable program code, and the processor 904 executes this executable program code to implement the functions of the aforementioned first determining module, second determining module, third determining module, and alarm module, thereby realizing the data processing method. That is, the memory 906 stores instructions for executing the data processing method.

[0446] The communication interface 909 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0447] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0448] As shown in Figure 8, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for executing data processing methods.

[0449] The computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002.

[0450] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing data processing methods.

[0451] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the data processing device. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules among the first determining module, the second determining module, the third determining module, and the alarm module.

[0452] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 illustrates one possible implementation. As shown in Figure 9, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through communication interfaces 1108 in each computing device.

[0453] The computing device 1100A includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102.

[0454] The computing device 1100B includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102.

[0455] In this type of possible implementation, the memory 1106 in computing device 1100A stores instructions for executing the functions of the first determination module, the second determination module, the third determination module, and the alarm module. Meanwhile, the memory 1106 in computing device 1100B stores instructions for executing the functions of the deletion module, the update module, the aggregation module, the fourth determination module, and the output module.

[0456] It should be understood that the functions of computing device 1100A shown in Figure 9 can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0457] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 7 and 9. The difference is that the memory 1106 of one or more computing devices 1100 in this computing device cluster can store the same instructions for executing data processing methods.

[0458] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for executing data processing methods.

[0459] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions for executing some functions of the data processing device. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more devices in the data processing device.

[0460] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the data processing method described in the above embodiments.

[0461] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the data processing method described in the above embodiments.

[0462] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

A data processing method, characterized in that, The method includes: Based on real-time collected log data, the target event to be detected is determined. The target event includes a start point, an end point, and the relationship between the start point and the end point. Based on the first label of the starting point, determine the second label of the ending point; The label of each node in the target event is used to indicate the historical state information of the target path ending at that node; The target path is one of at least one paths ending at the node, wherein the at least one path is a path formed by at least one event to be detected determined based on the log data; wherein the nodes in the target event include the start point and the end point of the target event. Based on the first path information associated with the first tag, determine the second path information associated with the second tag; When the second tag meets the alarm conditions, a local streaming origin graph related to the endpoint of the target event is output based on the second path information. The method according to claim 1, characterized in that, The method further includes: Determine the normality level of the target event; Based on the first label of the starting point and the normality of the target event, determine the historical state information of the third path; The third path is the path formed by the first path information and the target event; Based on the historical state information of the third path, the second label of the destination is determined. The method according to claim 2, characterized in that, The method further includes: Based on a pre-built information table, the normality level of the target event is determined; The information table includes first information indicating the normality of a preset event. The method according to claim 3, characterized in that, The method further includes: When the target event is the preset event in the information table, the normality of the target event is determined based on the first information in the information table indicating the normality of the target event. The method according to claim 3 or 4, characterized in that, The method further includes: When the target event does not belong to the preset event in the information table, the preset normality threshold is configured as the normality of the target event. The method according to any one of claims 2 to 5, characterized in that, The determination of the historical state information of the third path based on the first label of the starting point and the normality of the target event includes: When the first label at the starting point is empty data, if the normality of the target event meets the first preset condition, then the historical state information of the third path is determined based on the normality of the target event. The method according to any one of claims 2 to 5, characterized in that, The historical state information includes anomaly level information. The determination of the historical state information for the third path based on the first label of the starting point and the normality level of the target event includes: Based on the first label of the starting point, determine the normality of the target path indicated by the first label; Based on the normality of the target path and the normality of the target event, the abnormality information of the third path is determined. The method according to any one of claims 2 to 7, characterized in that, The determination of the second label of the destination based on the historical state information of the third path includes: If the second label of the endpoint is empty, then the second label of the endpoint is initialized to indicate the historical status information of the third path. The method according to claim 8, characterized in that, The step of determining the second path information related to the second tag based on the first path information related to the first tag includes: When it is determined that the second path information is empty data, the second path information related to the second tag of the destination is initialized as the information of the third path. The method according to any one of claims 2 to 7, characterized in that, The determination of the second label of the destination based on the historical state information of the third path includes: Based on the historical status information of the third path and the historical status information of the target path indicated by the second label, when it is determined that the second label needs to be refreshed, the second label of the destination is updated to indicate the historical status information of the third path. The method according to claim 10, characterized in that, The step of determining the second path information related to the second tag based on the first path information related to the first tag includes: The second path information associated with the second tag of the destination is refreshed to the information of the third path. The method according to any one of claims 1 to 11, characterized in that, The first tag and the first path information, the second tag and the second path information are all cached in memory. The method according to claim 12, characterized in that, The method further includes: When at least one of the first tag at the starting point and the first path information associated with the first tag meets the second preset condition, the information of the first tag and the first path information cached in memory are deleted. The method according to claim 13, characterized in that, The method further includes: When the historical state information of the target path indicated by the first tag at the starting point meets the tag reduction condition, the information of the first tag and the first path information cached in memory are deleted. The method according to claim 13 or 14 is characterized in that, The method further includes: When the path length of the first path information associated with the first tag is greater than a preset length threshold, the cached information of the first tag and the first path information in memory are deleted. The method according to any one of claims 1 to 15, characterized in that, The historical status information includes information on the degree of abnormality; When the second tag meets the alarm condition, the step of outputting the local streaming origin graph of the endpoint of the target event based on the second path information includes: When the anomaly level information of the target path indicated by the second label is greater than a preset anomaly level threshold, a local streaming origin graph related to the endpoint of the target event is output based on the second path information. The method according to any one of claims 1 to 16, characterized in that, The step of outputting the local streaming origin graph of the endpoint of the target event based on the second path information includes: The second path information, and are aggregated with the local streaming origin graph of the target node in the second path information to obtain the aggregated local streaming origin graph. The target node is a node whose label satisfies the alarm conditions; The aggregated local streaming origin graph is output as a local streaming origin graph associated with the endpoint of the target event. The method according to claim 17, characterized in that, The method further includes: The local cytokine origin graph of each node in the aggregated local cytokine origin graph is updated to the aggregated local cytokine origin graph. The method according to any one of claims 1 to 18, characterized in that, The determination of the second label of the endpoint based on the first label of the starting point includes: Based on the first label of the starting point and the preset normalization coefficient, the second label of the ending point is determined. A data processing device, characterized in that, The device includes: The first determining module is used to determine the target event to be detected based on real-time collected log data. The target event includes a starting point, an ending point, and the relationship between the starting point and the ending point. The second determining module is used to determine the second label of the endpoint based on the first label of the starting point; The label of each node in the target event is used to indicate the historical state information of the target path ending at that node; the target path is the path with the highest degree of abnormality among at least one path ending at that node, and the at least one path is a path composed of at least one event to be detected determined based on the log data; the nodes in the target event include the start point and the end point of the target event. The third determining module is used to determine the second path information related to the second tag based on the first path information related to the first tag; The alarm module is used to output a local streaming origin graph related to the endpoint of the target event based on the second path information when the second tag meets the alarm conditions. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 19. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 19. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 19.

Images