Abnormal aggregation method and system based on attribute weight and rule-driven

Through an abnormal aggregation method based on attribute weights and rules, the problems of large amount of abnormal data and poor aggregation effect in the prior art are solved, efficient abnormal data analysis and reduced manual intervention, and network security analysis efficiency is improved.

CN113849497BActive Publication Date: 2024-11-08CROSS BORDER CLOUD (HENGQIN) TECH INNOVATION RES CENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110881855.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-02
Publication Date
2024-11-08
Estimated Expiration
2041-08-02

AI Technical Summary

Technical Problem

The existing network exception detection and aggregation methods generate too much abnormal data when detecting abnormalities, which is difficult to analyze. The aggregation method based on attribute similarity ignores the timing relationship. The aggregation method based on timing and network features requires artificial definition of a large number of rules. , the polymerization effect is limited.

Method used

Using an exception aggregation method driven by attribute weights and rules, the attribute weights of each condition attribute are calculated by obtaining the original exception data of different devices, unified format, and constructing an exception information decision table, calculating the knowledge granularity and rough condition entropy of the condition attribute set, and calculating the attribute weights of each condition attribute. , perform exception aggregation according to the aggregation conditions and attribute weights, including exception type identical and time thresholds, and use predefined aggregation rules and attribute weight similarity for aggregation.

Benefits of technology

Effectively reduce the number of abnormalities, improve analysis efficiency, take into account the time intervals and self-scaling capabilities of abnormalities, and reduce the need for manual intervention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113849497B_ABST
    Figure CN113849497B_ABST
Patent Text Reader

Abstract

The present invention provides an abnormal aggregation method and system based on attribute weight and rule-driven. The method unifies the format of multiple original abnormal data to construct an abnormal information decision table. According to the same value relationship between the condition attribute and the decision attribute of the abnormality, the knowledge granularity and rough condition entropy of the condition attribute set are calculated for the abnormalities of multiple abnormal types; for each abnormal type, the attribute weight of each condition attribute for each abnormal type is calculated; when the current abnormality satisfies the preset aggregation condition, the current abnormality is aggregated into the existing aggregated abnormality that matches it; if the current abnormality does not meet the preset aggregation condition, the current abnormality is added as a new aggregated abnormality to the existing aggregated abnormality. The present invention obtains attribute similarity by calculating attribute weights and performs abnormal aggregation according to preset aggregation rules, which can effectively reduce the number of abnormalities and solve the problem of low analysis efficiency caused by a large number of abnormalities.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data identification technology, and in particular to an anomaly aggregation method and system based on attribute weight and rule-driven. Background Art

[0002] With the development of network technology, when a network attack occurs, the system will generate a large number of anomalies. In terms of system traffic, there will be anomalies related to traffic analysis, in terms of system applications, there will be application log anomalies, and in terms of the operation of the system itself, there will be system audit log anomalies. Therefore, the identification of abnormal data is a necessary means to maintain network security.

[0003] The existing network anomaly detection and aggregation methods generate a large amount of abnormal data in the process of detecting anomalies, which is difficult to analyze one by one. When the network is attacked, a large number of anomalies and alarms are generated. It is difficult to explore the laws of a large amount of data by manual means. In order to reduce the amount of analysis data of anomalies, it is necessary to aggregate the anomalies. The existing anomaly aggregation methods are mainly divided into aggregation methods based on attribute similarity and aggregation methods based on temporal relationships. However, the aggregation method based on attribute similarity focuses more on the similarity of anomalies and ignores the temporal relationship between anomalies. The aggregation method based on temporal and network characteristics requires manual definition of a large number of aggregation rules, and the aggregation effect on the anomaly itself is also very limited. Summary of the invention

[0004] The present invention provides an attribute weight-based and rule-driven anomaly aggregation method and system to overcome at least one technical problem existing in the prior art.

[0005] In a first aspect, the present invention provides an anomaly aggregation method based on attribute weight and rule-driven, comprising:

[0006] Get multiple original exceptions and current exceptions generated by different devices from different manufacturers;

[0007] Unifying the formats of the multiple original exception data so that the formats of the multiple original exceptions are unified into a data model definition format that describes multiple conditional attributes of the original exceptions;

[0008] Based on the multiple original exceptions in the format defined by the data model, construct an exception information decision table;

[0009] The abnormal information decision table includes the original abnormal set, a finite non-empty attribute set containing all attributes of the original abnormality, a value range of the finite non-empty attribute set, an information function representing the attribute value of the original abnormal data, a conditional attribute set, and a decision attribute set; the conditional attribute set includes other conditional attributes in the data model definition format except the abnormality type;

[0010] According to the same value relationship between the condition attribute and the decision attribute, the knowledge granularity and rough condition entropy of the condition attribute set of the anomalies of multiple anomaly types are calculated;

[0011] For each anomaly type, the attribute weight of each conditional attribute for each anomaly type is calculated based on the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set;

[0012] When the current exception meets the preset aggregation condition, the current exception is aggregated into the existing aggregated exception that matches it; if the current exception does not meet the preset aggregation condition, the current exception is added as a new aggregated exception to the existing aggregated exception;

[0013] Among them, the aggregation conditions include: the same anomaly type, the anomaly occurrence time is less than the time threshold, and there is at least one aggregation method, the aggregation method includes a first aggregation method according to a predefined aggregation rule and / or a second aggregation method according to attribute weight similarity.

[0014] Among them, the data model definition format includes exception type, exception occurrence time, exception source address, exception source port, exception destination address, exception destination port attack direction, exception level, attack stage and exception description; the existing aggregated exception includes at least one original exception under multiple exception types, and the original exceptions of the same exception type are aggregated together to form an aggregated exception.

[0015] Optionally, the calculating the knowledge granularity of the condition attribute set and the rough condition entropy of the decision attribute set for anomalies of multiple anomaly types according to the same value relationship of the condition attribute and the decision attribute includes:

[0016] Based on the knowledge granularity of the conditional attribute set, calculating the discernibility of the conditional attribute set;

[0017] For multiple original anomalies, the rough conditional entropy of the decision attribute set is calculated based on the conditional information entropy principle.

[0018] Optionally, for each abnormality type, calculating the attribute weight of each conditional attribute for each abnormality type according to the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set includes:

[0019] For each anomaly type, the importance of each conditional attribute in the conditional attribute set for the anomaly type is calculated based on the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set;

[0020] According to the importance of the condition attribute set for each abnormality type, the attribute weight of each condition attribute for each abnormality type is calculated.

[0021] Among them, the first aggregation method is an aggregation method for aggregating according to whether the values ​​of the original anomaly and the current anomaly on the conditional attribute set defined by the aggregation rule are the same, and the second aggregation method is an aggregation method for aggregating according to whether the attribute weight similarity exceeds the similarity threshold.

[0022] Optionally, when the current exception satisfies a preset aggregation condition, aggregating the current exception into an existing aggregated exception that matches the current exception includes:

[0023] Determine whether there is an aggregated exception of the same exception type as the current exception among the existing aggregated exceptions; if so, continue to determine whether the exception occurrence time of the current exception is less than the time threshold;

[0024] If the occurrence time of the current exception is less than the time threshold, determine whether there is an aggregated exception in the existing aggregated exceptions that has the same value as the current exception in the condition attribute set defined by the aggregation rule;

[0025] If there is an aggregated exception in the existing aggregated exceptions that has the same value as the current exception in the condition attribute set defined by the aggregation rule, it is determined that the existing aggregated exception matches the current exception;

[0026] If there is no aggregated anomaly with the same value as the current anomaly in the condition attribute set defined by the aggregation rule among the existing aggregated anomalies, then the similarity between the current anomaly and the existing aggregated anomaly in the same condition attribute is calculated, and the existing aggregated anomaly with a similarity greater than a threshold is determined to match the current anomaly;

[0027] Aggregate the current exception into the set of existing exceptions that match it.

[0028] Optionally, the time threshold is determined by the following steps:

[0029] When the current anomaly does not meet the preset aggregation condition, the mean and variance of the occurrence time interval of each original anomaly under each aggregated anomaly are calculated;

[0030] For each aggregate anomaly, the mean and variance of the aggregate anomaly are used as the first attribute and the second attribute respectively;

[0031] A time threshold is determined according to the first attribute and the second attribute.

[0032] Optionally, after determining the time threshold according to the first attribute and the second attribute, the anomaly aggregation method further includes:

[0033] When the current exception occurs, updating the first attribute and the second attribute;

[0034] The time threshold is re-determined using the updated first attribute and the second attribute.

[0035] Optionally, if the current exception does not meet a preset aggregation condition, adding the current exception as a new aggregate exception to the existing aggregate exceptions includes:

[0036] If the exception occurrence time of the current exception is not less than the re-determined time threshold, the current exception is connected to the existing aggregated exceptions as an aggregated exception of a new exception type.

[0037] In a second aspect, the present invention provides an anomaly aggregation system based on attribute weight and rule-driven, comprising:

[0038] The acquisition module is used to obtain multiple original exceptions and current exceptions generated by different devices from different manufacturers;

[0039] A unification module, used for unifying the formats of the multiple original exception data, so that the formats of the multiple original exceptions are unified into a data model definition format for describing multiple conditional attributes of the original exceptions;

[0040] A construction module, used to construct an exception information decision table based on a plurality of original exceptions in a format defined by the data model;

[0041] The abnormal information decision table includes the original abnormal set, a finite non-empty attribute set containing all attributes of the original abnormality, a value range of the finite non-empty attribute set, an information function representing the attribute value of the original abnormal data, a conditional attribute set, and a decision attribute set; the conditional attribute set includes other conditional attributes in the data model definition format except the abnormality type;

[0042] The first calculation module is used to calculate the knowledge granularity and rough conditional entropy of the conditional attribute set of anomalies of multiple anomaly types according to the same value relationship between the conditional attribute and the decision attribute;

[0043] The second calculation module is used to calculate the attribute weight of each condition attribute for each abnormal type according to the rough condition entropy of the decision attribute set and the discernibility of the condition attribute set for each abnormal type;

[0044] The aggregation module is used to aggregate the current exception into the existing aggregated exception that matches it when the current exception meets the preset aggregation condition; if the current exception does not meet the preset aggregation condition, the current exception is added as a new aggregated exception into the existing aggregated exception.

[0045] Among them, the aggregation conditions include: the same anomaly type, the anomaly occurrence time is less than the time threshold, and there is at least one aggregation method, the aggregation method includes a first aggregation method according to a predefined aggregation rule and / or a second aggregation method according to attribute weight similarity.

[0046] The present invention provides an abnormal aggregation method and system based on attribute weight and rule-driven. The method unifies the format of multiple original abnormal data to construct an abnormal information decision table. According to the same value relationship between the condition attribute and the decision attribute of the abnormality, the knowledge granularity and rough condition entropy of the condition attribute set are calculated for the abnormalities of multiple abnormal types; for each abnormal type, the attribute weight of each condition attribute for each abnormal type is calculated; when the current abnormality satisfies the preset aggregation condition, the current abnormality is aggregated into the existing aggregated abnormality that matches it; if the current abnormality does not meet the preset aggregation condition, the current abnormality is added as a new aggregated abnormality to the existing aggregated abnormality. The present invention obtains attribute similarity by calculating attribute weights and performs abnormal aggregation according to preset aggregation rules, which can effectively reduce the number of abnormalities and solve the problem of low analysis efficiency caused by a large number of abnormalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a processing flow diagram of the anomaly aggregation method based on attribute weight and rule-driven of the present invention;

[0049] Figure 2 The framework diagram of the abnormal aggregation method of the present invention is mainly divided into two stages: offline analysis and online aggregation;

[0050] Figure 3 A schematic diagram of polymerization in the online polymerization stage of the present invention;

[0051] Figure 4 The polymerization flow chart of the online polymerization stage of the present invention;

[0052] Figure 5 A schematic diagram of the structure of the anomaly aggregation system based on attribute weights and rule-driven according to the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0055] The embodiment of the present invention discloses an anomaly aggregation method and system based on attribute weight and rule-driven. Before introducing the present invention, the symbols and abbreviations involved in the present invention are first introduced.

[0056] DDoS distributed denial of service attack; IDMEF intrusion detection message exchange format; XML extensible markup language; H(X) entropy of discrete random variable X; IPS intrusion prevention system; Json JavaScript object notation; T d Abnormal information decision table; U original abnormal data set; At original abnormal attribute set; V a The value range of the abnormal attribute a; I a Information function, which represents the value of attribute a of anomaly x; R equivalence relation; E r (A) Rough entropy of anomaly A; J j (C) The importance of the abnormal attribute set C; P(A|B) The conditional probability of abnormal A occurring after abnormal B occurs; The following is a detailed description of an abnormal aggregation method and system based on attribute weight and rule-driven.

[0057] This paper proposes an anomaly aggregation method based on attribute weight and rule-driven. Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the processing flow of the anomaly aggregation method based on attribute weight and rule-driven of the present invention. Figure 1 As shown in FIG. 1 , an anomaly aggregation method based on attribute weight and rule-driven includes the following steps:

[0058] S1, obtain multiple original exceptions and current exceptions generated by different devices from different manufacturers;

[0059] The present invention provides an anomaly aggregation method based on attribute weights and rule-driven, which supports manual definition of aggregation rules for various types of anomalies, and also supports the anomaly aggregation method of the present invention to automatically aggregate according to the attribute weights of different types of anomalies, while taking into account the time interval and interval self-expansion capability of the anomalies during aggregation. The anomaly aggregation method of the present invention is mainly divided into two stages: offline analysis and online aggregation. Figure 2 As shown. When a network attack occurs, the system will generate a large number of exceptions. In terms of system traffic, there will be exceptions related to traffic analysis. In terms of system applications, there will be application log exceptions. In terms of the operation of the system itself, there will be system audit log exceptions. The present invention does not make a specific distinction between the levels of exceptions, but focuses on the anomalies and alarms identified by enterprise security equipment when the enterprise application system is attacked. Since the exception information results sent by different devices from different manufacturers are not completely consistent, it is necessary to define the basic data structure and attribute meaning of the exception, and pre-process and format the exceptions of the entire system. The formatted exception information will be persisted to the exception information database for attribute weight calculation in the offline analysis phase.

[0060] S2, unifying the formats of the multiple original exception data so that the formats of the multiple original exceptions are unified into a data model definition format that describes multiple conditional attributes of the original exceptions;

[0061] The data model definition format includes exception type, exception occurrence time, exception source address, exception source port, exception destination address, exception destination port attack direction, exception level, attack stage and exception description.

[0062] In the field of computer security, IDMEF (Intrusion Detection Information Exchange Format) is used to describe the anomalies generated by the intrusion detection system, and the XML data model is used to express the format of the anomalies. The anomaly format defined by IDMEF contains information such as the source, destination, and category required to describe the anomaly. This model is defined as the basic attribute definition of the anomaly model of the present invention as shown in Table 1, and the different anomaly information of different devices will be unified into the format shown in Table 1. Among them, the type of anomaly occurrence is the key indicator of aggregation. In the present invention, the anomaly aggregation method of the present invention sets that only anomalies of the same type can be aggregated, and anomalies of different types can only be associated. In addition, the time when the anomaly occurs is also an important factor in determining whether it can be aggregated. Only anomalies that occur within a certain time interval will be aggregated.

[0063] Table 1 Data model definition of original anomaly

[0064]

[0065] The aggregate anomaly (super anomaly) model of the present invention is shown in Table 2. In addition to the original anomaly model, the aggregate anomaly model adds the number of aggregations and modifies the occurrence time to the start time and the end time. In addition, it also describes information such as the aggregation level and the aggregation reason. However, these model definitions only describe the specific anomaly attributes of the research object of the present invention, and the research method of the present invention is not limited to specific attributes, but can be adapted to any attribute through simple adjustments. Exceptions from different manufacturers and security devices need to be formatted into a unified form to facilitate subsequent exception aggregation and exception analysis. The system's exception model and super anomaly model design provide a reference. Since the XML format is not convenient for Web exchange, the system defines two models using the Json format.

[0066] Table 2 Data model definition of aggregation anomaly

[0067]

[0068] S3, builds an exception information decision table based on multiple original exceptions in the data model definition format;

[0069] The abnormal information decision table includes the original abnormal set, the finite non-empty attribute set containing all the attributes of the original abnormality, the value range of the finite non-empty attribute set, the information function representing the attribute value of the original abnormal data, the conditional attribute set and the decision attribute set; the conditional attribute set includes other conditional attributes except the abnormality type in the data model definition format;

[0070] The present invention is divided into an offline training phase and an online aggregation phase. In the offline training phase, the system calculates the attribute weights of different types of anomalies based on conditional rough entropy and knowledge granularity, and persists the attribute weights to form an anomaly attribute weight table. The subsequent online aggregation phase requires the use of different types of anomaly attribute weights for aggregation. The anomaly data space studied in the present invention can be regarded as a decision table, expressed as:

[0071]

[0072] Where T d represents the abnormal information decision table, U is the set of original abnormal data in the decision table, also known as the domain, At is a finite non-empty attribute set, including all the attributes of the original abnormality in Table 1, V a is the value range of attribute a∈At, I a is an information function, Ia(x) represents the value of attribute a of object x, that is, for any a ∈At, x∈U, there is Ia(x)∈Va. C represents the conditional attribute set, and D represents the decision attribute set. The conditional attribute is needed to determine the value of the decision attribute. Specifically, the decision attribute is D={type}, and the conditional attributes include:

[0073]

[0074] The purpose of offline analysis is to calculate the aggregation weight of conditional attributes. In the anomaly aggregation stage, the anomaly aggregation method of the present invention is designed to aggregate only anomalies of the same anomaly type. For the same type, if all the conditional attributes of two anomalies are the same, then there is a high probability that the two anomalies can be aggregated together, but it is often rare that they are completely identical. In most cases, the attributes of the anomalies are partially similar. How to determine the aggregation attribute weight of a specific type of anomaly is a problem that needs to be solved in offline analysis.

[0075] S4, according to the same value relationship between the conditional attributes and decision attributes of the anomalies, for anomalies of multiple anomaly types, the knowledge granularity and rough conditional entropy of the conditional attribute set are calculated;

[0076] As an optional implementation of the present invention, according to the same value relationship between the condition attribute and the decision attribute of the anomaly, for anomalies of multiple anomaly types, calculating the knowledge granularity of the condition attribute set and the rough conditional entropy of the decision attribute set includes:

[0077] Step a: Based on the knowledge granularity of the conditional attribute set, calculate the discernibility of the conditional attribute set;

[0078] Step b: For multiple original anomalies, the rough conditional entropy of the decision attribute set is calculated based on the conditional information entropy principle.

[0079] Based on the decision table, the present invention can define two equivalence relations R C and R D , which means that the values ​​of the anomalies on the condition attributes or decision attributes are equal. The formal expression of the equivalence relation is as follows:

[0080]

[0081]

[0082] Equivalence relation R C and R D The equivalent partitioning on the anomaly set U is:

[0083] U / R C = {X1, X2, ..., X n}

[0084] U / R D ={Y1, Y2, ..., Y m}.

[0085] U / R CAn equivalence class of represents that these anomalies are equal in attribute set C, and the number of equivalence classes represents how many classes the attribute set can divide the entire set of anomalies into. From the perspective of rough set theory, the attributes of each anomaly need to reflect the characteristics of this type of anomaly, and the conditional attributes of the anomaly can largely determine the value of the conditional attributes. This requires that the conditional attributes of the anomaly need to have high discernibility, that is, a smaller knowledge granularity. The knowledge granularity of a single attribute set can be expressed as:

[0086]

[0087] However, the knowledge granularity expressed by formula (1) is limited to a single attribute set, which cannot reflect the effect of conditional attributes on the division of decision attributes. It reflects the knowledge granularity of a single attribute set when all anomalies are equivalently divided, which is not required by the anomaly aggregation method of the present invention. Since the weights of different types of anomaly attributes should be inconsistent, for anomalies of type j, the knowledge granularity of the conditional attribute set C is:

[0088]

[0089] Where Z = {X i ∣X i ∩Y j ≠Φ}, representing the equivalence class X divided by the conditional attribute set C i contains an exception of type j. Therefore, for an exception of type j, the discernibility of the conditional attribute set C is:

[0090]

[0091] On the other hand, in the rough set theory, rough entropy describes the roughness of information. The larger the rough entropy, the less obvious the effect of the attribute set on the division of the whole set. Then this attribute subset cannot represent a class of anomalies. The same attributes of this attribute subset cannot represent that two original anomalies belong to the same class of anomalies that can be aggregated. The concept of rough entropy is as follows:

[0092]

[0093] In the anomaly aggregation process, the anomaly aggregation method of the present invention focuses more on the rough entropy of the decision attribute set (anomaly type) under the condition of known condition attribute set. j After that, the anomaly type j can be finely divided, which means that the attribute set C can represent the anomaly type j and can have a larger weight in the anomaly aggregation process. According to the theory of conditional information entropy, for anomalies of type j, when the conditional attribute set C is known, the decision attribute set D j The rough conditional entropy of is:

[0094]

[0095] S5, for each anomaly type, according to the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set, calculate the attribute weight of each conditional attribute for each anomaly type;

[0096] As an optional implementation of the present invention, for each abnormality type, according to the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set, calculating the attribute weight of each conditional attribute for each abnormality type includes:

[0097] Step a: For each anomaly type, the importance of each conditional attribute in the conditional attribute set for the anomaly type is calculated based on the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set;

[0098] Step b: Calculate the attribute weight of each condition attribute for each anomaly type based on the importance of the condition attribute set for each anomaly type.

[0099] In summary, when judging the attribute aggregation weight, the anomaly aggregation method of the present invention requires that the knowledge granularity of the equivalent partition of the candidate attribute set is as small as possible, the discernibility is as high as possible, and when the candidate condition attribute set is known, the rough entropy of the decision attribute set is as small as possible, and the partition is more refined. Considering the influence of division by zero and negative numbers, the importance of the condition attribute set C for a specific anomaly type can be expressed as:

[0100]

[0101] Through formula (6), we can calculate the importance of attribute a for a specific anomaly type j, which can be expressed as:

[0102] J j (a) = J j (C)-J j (C-{a}) (7)

[0103] After calculating the attribute set C = {a1, a2, ..., a k After calculating the importance of all attributes in} for a specific type of anomaly j, the attribute aggregation weight of attribute a for this type can be calculated by proportion, as shown in formula (8).

[0104]

[0105] By traversing all exception types and all conditional attributes, the exception aggregation attribute weight table can be obtained for automatic weight aggregation in the subsequent online association stage.

[0106] S6, if the current exception meets the preset aggregation condition, aggregate the current exception into the existing aggregated exception that matches it; if the current exception does not meet the preset aggregation condition, add the current exception as a new aggregated exception into the existing aggregated exception;

[0107] The aggregation conditions include: the same anomaly type, the anomaly occurrence time is less than the time threshold, and there is at least one aggregation method, the aggregation method includes a first aggregation method of aggregation according to a predefined aggregation rule and / or a second aggregation method of aggregation according to attribute weight similarity. The existing aggregated anomaly includes at least one original anomaly under multiple anomaly types, and the original anomalies of the same anomaly type are aggregated together to form an aggregated anomaly.

[0108] Among them, the first aggregation method is an aggregation method based on whether the values ​​of the original anomaly and the current anomaly on the conditional attribute set defined by the aggregation rule are the same, and the second aggregation method is an aggregation method based on whether the attribute weight similarity exceeds the similarity threshold.

[0109] As an optional implementation of the present invention, when the current exception meets the preset aggregation condition, aggregating the current exception into the existing aggregated exception matching it includes:

[0110] Step a: Determine whether there is an aggregated anomaly of the same anomaly type as the current anomaly among the existing aggregated anomalies; if so, continue to determine whether the anomaly occurrence time of the current anomaly is less than the time threshold;

[0111] Step b: If the abnormal occurrence time of the current abnormality is less than the time threshold, determine whether there is an aggregated abnormality in the existing aggregated abnormalities that has the same value as the current abnormality in the condition attribute set defined by the aggregation rule;

[0112] Step c: If there is an aggregated anomaly in the existing aggregated anomaly that has the same value as the current anomaly in the condition attribute set defined by the aggregation rule, it is determined that the existing aggregated anomaly matches the current anomaly;

[0113] Step d: If there is no aggregated anomaly with the same value as the current anomaly on the condition attribute set defined by the aggregation rule in the existing aggregated anomalies, then calculate the similarity between the current anomaly and the existing aggregated anomaly on the same condition attribute, and determine that the existing anomaly aggregation with a similarity greater than a threshold matches the current anomaly;

[0114] Step e: Aggregate the current exception into the set of existing exceptions that match it.

[0115] The following is the online aggregation stage of the present invention. The online aggregation stage needs to jointly decide the abnormal aggregation strategy based on the aggregation rules, attribute weights and abnormal time. The process is as follows: Figure 4As shown in the figure, when the current exception arrives, it will be matched with a certain number of aggregated exceptions according to the occurrence time. Once an exception that can be aggregated is matched, the current exception will be merged into the existing aggregated exception. Otherwise, a new aggregated exception will be created and placed at the head of the queue. The current exception must meet the following conditions to be merged into the aggregated exception:

[0116] The following three conditions:

[0117] (1) The exception type is consistent

[0118] The type of the current exception must be consistent with the type of the aggregated exception (type attribute). This does not mean that only exceptions of the same type belong to a security event, but under the evaluation framework of the present invention, the exception aggregation stage only aggregates exceptions of the same type, and exceptions of inconsistent types will be associated and analyzed in subsequent stages. Therefore, exceptions of inconsistent types cannot be aggregated into the same aggregated exception at this stage.

[0119] (2) The abnormality occurred less than the time threshold

[0120] Under the aggregation framework of the present invention, only exceptions whose occurrence time is less than the time threshold will be aggregated into the same aggregated exception. In actual industrial application scenarios, most exceptions come from existing attack scripts and tools. There are certain rules in the attack time initiated by these scripts, but such rules cannot be specified manually or calculated based on historical data. They can only be adjusted automatically based on the personalized existing aggregated exceptions. The present invention uses a self-expanding time window to adjust the time threshold of the aggregation.

[0121] (3) There is at least one polymerization mode

[0122] After meeting the type and time requirements, the anomaly aggregation method of the present invention needs to find a way to aggregate. The aggregation method comes from the manually defined aggregation rules and the attribute weight aggregation method automatically calculated by the anomaly aggregation method of the present invention. The anomaly aggregation method of the present invention will give priority to finding the aggregation rules of the specified type and the global manually defined aggregation rules. In the absence of rules, the anomaly aggregation method of the present invention will calculate the attribute similarity. If the similarity is still less than the threshold, then aggregation cannot be performed and a new aggregation anomaly needs to be created.

[0123] In the process of exception aggregation, two aspects need to be manually defined:

[0124] (1) Abnormal aggregation rules: Security personnel can customize aggregation rules for specific types or global rules. However, the abnormal aggregation method of the present invention can also work normally without rules.

[0125] (2) Similarity threshold: The anomaly aggregation method of the present invention needs to set a similarity threshold. When there is no matching aggregation rule, the anomaly aggregation method of the present invention will aggregate according to the weighted similarity of the anomaly attributes. If the similarity exceeds the threshold, it can be aggregated into an aggregated anomaly.

[0126] It is understandable that if a current anomaly cannot find an anomaly that can be aggregated, then a new aggregate anomaly will be created and only contain the information of this current anomaly, waiting for subsequent anomalies to be merged into the aggregate anomaly. Each aggregate anomaly has a specific time threshold, indicating the time range within which the original anomaly can be merged into the aggregate anomaly. How to determine this time threshold is an issue that needs to be discussed. Manually setting the time threshold is not a good choice. An aggregate anomaly represents the same type of security incidents that occur frequently in a short period of time. A fixed time threshold can neither represent the frequency of all security incidents nor the frequency of the same type of security incidents. Therefore, each aggregate anomaly should have its own specific time threshold. The anomaly aggregation method of the present invention uses a self-expanding time window based on standard deviation to solve this problem.

[0127] Therefore, the time threshold is determined by the following steps:

[0128] Step a: When the current anomaly does not meet the preset aggregation condition, calculate the mean and variance of the occurrence time interval of each original anomaly under each aggregated anomaly;

[0129] Step b: for each aggregate anomaly, the mean and variance of the aggregate anomaly are used as the first attribute and the second attribute respectively;

[0130] Step c: Determine a time threshold according to the first attribute and the second attribute.

[0131] The time threshold can be adjusted to a certain extent by calculating the mean and standard deviation of the occurrence intervals of all original anomalies in the aggregated anomaly. Assume that the occurrence time of the original anomalies contained in an aggregated anomaly is {t1, t2, …, t n}, then at t n+1 time, when a new current exception occurs, can be calculated by i -t i -1, to determine the mean and standard deviation of t n+1 -t n Whether it is within a reasonable fluctuation range. Of course, every time a new anomaly arrives, it is time-consuming and unreasonable to recalculate the time intervals of all original anomalies. Therefore, the anomaly aggregation method of the present invention needs to add and maintain two attributes for each aggregated anomaly, which are the first attribute:

[0132]

[0133] Second attribute:

[0134]

[0135] If a new current exception occurs at time tn+1, the exception aggregation method of the present invention only needs to update these two attributes:

[0136] bias n+1 =bias n +(t n+1 -t n )

[0137] var n+1 =var n +(t n+1 -t n ) 2

[0138] Then, when the abnormal time tn+1 satisfies the requirements shown in formula (9), it means that the time interval meets the abnormal aggregation conditions.

[0139]

[0140] As an optional implementation of the present invention, after determining the time threshold according to the first attribute and the second attribute, the anomaly aggregation method further includes:

[0141] Step a: When the current exception occurs, update the first attribute and the second attribute;

[0142] Step b: re-determine the time threshold using the updated first attribute and the second attribute.

[0143] It can be understood that by judging whether the time interval meets the requirements, if an aggregated exception has only a single original exception, then when a new current exception of the same type arrives, aggregation is allowed by default, and the mean and variance of the time interval are updated for subsequent time judgment.

[0144] After confirming the type and time, the anomaly aggregation method of the present invention needs to determine whether the attribute can be aggregated. Attribute aggregation needs to first determine whether the rule is applicable based on the priority. If there is no applicable rule, it is necessary to calculate the weighted attribute similarity and determine whether to aggregate based on the threshold. The data model of the rule is shown in Table 3:

[0145] Table 3 Data model definition of aggregation rules

[0146]

[0147] The rules attribute is a collection of attribute names, and the attributes are separated by ",". To determine whether the current exception and the original exception satisfy the aggregation rule, in essence, is to determine whether the values ​​of the current exception and the original exception in the attribute set defined by rules are the same. If they are the same, it means that the rules match. If there is no matching rule, the exception aggregation method of the present invention will try to use weighted attribute similarity to determine whether aggregation can be performed. In the offline analysis stage, the exception aggregation method of the present invention has calculated the attribute weight Wt for each type of exception t, so the attribute similarity between the current exception and the original exception can be calculated:

[0148]

[0149] where s i Indicates whether the i-th attribute of the current exception is the same as the i-th attribute of the original exception. If they are the same, it is 1, otherwise it is 0. If the similarity is greater than the preset threshold, it can be aggregated as a rule with the lowest priority, and the rules attribute of the aggregated rule can be all the same attributes.

[0150] As an optional implementation of the present invention, if the current exception does not meet the preset aggregation condition, adding the current exception as a new aggregation exception to the existing aggregation exceptions includes:

[0151] If the exception occurrence time of the current exception is not less than the re-determined time threshold, the current exception is connected to the existing aggregated exception as an aggregated exception of a new exception type.

[0152] Online aggregation process Figure 5 As shown. First, the existing aggregated anomalies are traversed in chronological order. For each aggregated anomaly, the anomaly aggregation method of the present invention will determine in turn whether the type of anomaly is the same and whether the time interval of the anomaly meets the aggregation requirements. Then the anomaly aggregation method of the present invention will determine the aggregation strategy based on the aggregation rules or attribute similarity defined by the security personnel, and merge it into the existing aggregated anomaly. If any point does not meet the requirements or a matching aggregation rule cannot be found, a new aggregated anomaly will be recreated and added to the head of the queue.

[0153] In the offline calculation stage, the anomaly aggregation method of the present invention calculates the attribute weights of different types of anomalies based on historical anomaly data. For example, in a certain type of scanning event, the source address may be more able to mark a typical scanning behavior, indicating that the attacker initiated a vulnerability scan for multiple devices from a specific source address, while in a specific type of password brute force cracking behavior, the same destination address may be more able to mark a security event, indicating that the attacker changed multiple attack source addresses to carry out password brute force cracking actions against a specific target. Therefore, in the offline analysis stage, the anomaly aggregation method of the present invention needs to automatically calculate the attribute weights of specific types of anomalies based on historical anomaly data, construct an anomaly attribute weight table, and use it in the subsequent online aggregation stage. The online aggregation stage aggregates anomalies according to the anomaly attribute weight table and the anomaly aggregation rule table. When anomalies arrive, the anomaly aggregation method of the present invention will first adapt the existing anomaly aggregation rules and trace back to the super anomalies that have been aggregated for a period of time before. If the current anomaly and the super anomaly meet the aggregation rules, the current anomaly will be aggregated into the super anomaly, and the attributes of the super anomaly will also be updated accordingly. If the aggregation rules cannot be matched, the anomaly aggregation method of the present invention will determine whether to aggregate based on the attribute similarity and attribute weight. In the process of anomaly aggregation, the anomaly aggregation method of the present invention will consider the time attribute of the anomaly separately. Since most anomalies are generated by attack scripts, the time frequency of anomalies follows a certain pattern, but the time intervals between different types of anomalies in different attack scripts are often not exactly the same, and it is impossible to manually define specific thresholds. The anomaly aggregation method of the present invention uses a threshold judgment method of a self-expanding time window according to the time frequency of occurrence of different types of anomalies, automatically calculates the frequency of occurrence of specific anomalies and adjusts possible time windows, thereby effectively utilizing the occurrence time for anomaly aggregation. Aggregated super anomalies will be retained in both the memory and the database. The super anomalies in the memory are used to match the aggregation strategy with the current anomaly. If the current anomaly is aggregated into the super anomaly, the attributes of the super anomaly will be updated, and the update results will also be synchronized to the database for subsequent anomaly association and analysis.

[0154] The present invention provides an abnormal aggregation method based on attribute weight and rule-driven, which unifies the format of multiple original abnormal data, builds an abnormal information decision table based on multiple original abnormalities in the format defined by the data model, and calculates the knowledge granularity and rough conditional entropy of the conditional attribute set for abnormalities of multiple abnormal types according to the same value relationship of the conditional attribute and the decision attribute; calculates the attribute weight of each conditional attribute for each abnormal type for each abnormal type; aggregates the current abnormality into the existing aggregated abnormality that matches it when judging that the current abnormality meets the preset aggregation condition; and adds the current abnormality as a new aggregated abnormality to the existing aggregated abnormality when the current abnormality does not meet the preset aggregation condition. The present invention obtains attribute similarity by attribute weight and performs abnormal aggregation by preset aggregation rules, which can effectively reduce the number of abnormalities and solve the problem of low analysis efficiency caused by a large number of abnormalities.

[0155] It supports manually defining aggregation rules for various types of anomalies, and also supports the anomaly aggregation method of the present invention to automatically aggregate according to the attribute weights of different types of anomalies, while taking into account the time interval and interval self-expansion capability of the anomalies during aggregation.

[0156] The present invention provides an abnormal aggregation system based on attribute weight and rule-driven, including:

[0157] An acquisition module 61 is used to acquire multiple original exceptions and current exceptions generated by different devices from different manufacturers;

[0158] A unification module 62, configured to unify the formats of the plurality of original exception data so that the formats of the plurality of original exceptions are unified into a data model definition format for describing a plurality of conditional attributes of the original exceptions;

[0159] A construction module 63 is used to construct an exception information decision table based on multiple original exceptions in a data model definition format;

[0160] The abnormal information decision table includes the original abnormal set, the finite non-empty attribute set containing all the attributes of the original abnormality, the value range of the finite non-empty attribute set, the information function representing the attribute value of the original abnormal data, the conditional attribute set and the decision attribute set; the conditional attribute set includes other conditional attributes except the abnormality type in the data model definition format;

[0161] A first calculation module 64 is used to calculate the knowledge granularity and rough conditional entropy of the conditional attribute set of anomalies of multiple anomaly types according to the same value relationship between the conditional attribute and the decision attribute of the anomaly;

[0162] A second calculation module 65 is used to calculate the attribute weight of each condition attribute for each abnormal type according to the rough condition entropy of the decision attribute set and the discernibility of the condition attribute set for each abnormal type;

[0163] Aggregation module 66 is used to aggregate the current exception into the existing aggregated exception that matches it when the current exception meets the preset aggregation condition; if the current exception does not meet the preset aggregation condition, add the current exception as a new aggregated exception into the existing aggregated exception.

[0164] Among them, the aggregation conditions include: the same anomaly type, the anomaly occurrence time is less than the time threshold, and there is at least one aggregation method, the aggregation method includes a first aggregation method of aggregating according to a predefined aggregation rule and / or a second aggregation method of aggregating according to attribute weight similarity.

[0165] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0166] Those skilled in the art can understand that the modules in the system in the embodiment can be distributed in the system of the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more systems different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for anomaly aggregation based on attribute weight and rule-driven, characterized in that: include: Get multiple original exceptions and current exceptions generated by different devices from different manufacturers; Unifying the formats of the multiple original exception data so that the formats of the multiple original exceptions are unified into a data model definition format that describes multiple conditional attributes of the original exceptions; Based on the multiple original exceptions in the format defined by the data model, construct an exception information decision table; The abnormal information decision table includes the original abnormal set, a finite non-empty attribute set containing all attributes of the original abnormality, a value range of the finite non-empty attribute set, an information function representing the attribute value of the original abnormal data, a conditional attribute set, and a decision attribute set; the conditional attribute set includes other conditional attributes in the data model definition format except the abnormality type; According to the same value relationship between the condition attribute and the decision attribute, the knowledge granularity and rough condition entropy of the condition attribute set of the anomalies of multiple anomaly types are calculated; For each anomaly type, the attribute weight of each conditional attribute for each anomaly type is calculated based on the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set; When the current exception meets the preset aggregation condition, the current exception is aggregated into the existing aggregated exception that matches it; if the current exception does not meet the preset aggregation condition, the current exception is added as a new aggregated exception to the existing aggregated exception; Among them, the aggregation conditions include: the same anomaly type, the anomaly occurrence time is less than the time threshold, and there is at least one aggregation method, the aggregation method includes a first aggregation method according to a predefined aggregation rule and / or a second aggregation method according to attribute weight similarity.

2. The abnormal polymerization method according to claim 1, characterized in that: The data model definition format includes exception type, exception occurrence time, exception source address, exception source port, exception destination address, exception destination port attack direction, exception level, attack stage and exception description; the existing aggregated exception includes at least one original exception under multiple exception types, and the original exceptions of the same exception type are aggregated together to form an aggregated exception.

3. The abnormal polymerization method according to claim 2, characterized in that: The calculating of the knowledge granularity of the condition attribute set and the rough condition entropy of the decision attribute set for the anomalies of multiple anomaly types according to the same value relationship between the condition attribute and the decision attribute includes: Based on the knowledge granularity of the conditional attribute set, calculating the discernibility of the conditional attribute set; For multiple original anomalies, the rough conditional entropy of the decision attribute set is calculated based on the conditional information entropy principle.

4. The abnormal polymerization method according to claim 2, characterized in that: For each abnormality type, according to the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set, calculating the attribute weight of each conditional attribute for each abnormality type includes: For each anomaly type, the importance of each conditional attribute in the conditional attribute set for the anomaly type is calculated based on the rough conditional entropy of the decision attribute set and the discernibility of the conditional attribute set; According to the importance of the condition attribute set for each abnormality type, the attribute weight of each condition attribute for each abnormality type is calculated.

5. The abnormal polymerization method according to claim 1, characterized in that: The first aggregation method is an aggregation method that performs aggregation based on whether the values ​​of the original anomaly and the current anomaly on the conditional attribute set defined by the aggregation rule are the same, and the second aggregation method is an aggregation method that performs aggregation based on whether the attribute weight similarity exceeds a similarity threshold.

6. The abnormal polymerization method according to claim 5, characterized in that: When the current exception meets the preset aggregation condition, aggregating the current exception into the existing aggregated exception that matches it includes: Determine whether there is an aggregated exception of the same exception type as the current exception among the existing aggregated exceptions; if so, continue to determine whether the exception occurrence time of the current exception is less than the time threshold; If the occurrence time of the current exception is less than the time threshold, determine whether there is an aggregated exception in the existing aggregated exceptions that has the same value as the current exception in the condition attribute set defined by the aggregation rule; If there is an aggregated exception in the existing aggregated exceptions that has the same value as the current exception in the condition attribute set defined by the aggregation rule, it is determined that the existing aggregated exception matches the current exception; If there is no aggregated anomaly with the same value as the current anomaly in the condition attribute set defined by the aggregation rule among the existing aggregated anomalies, then the similarity between the current anomaly and the existing aggregated anomaly in the same condition attribute is calculated, and the existing aggregated anomaly with a similarity greater than a threshold is determined to match the current anomaly; Aggregate the current exception into the set of existing exceptions that match it.

7. The abnormal polymerization method according to claim 6, characterized in that: The time threshold is determined by the following steps: When the current anomaly does not meet the preset aggregation condition, the mean and variance of the occurrence time interval of each original anomaly under each aggregated anomaly are calculated; For each aggregate anomaly, the mean and variance of the aggregate anomaly are used as the first attribute and the second attribute respectively; A time threshold is determined according to the first attribute and the second attribute.

8. The abnormal polymerization method according to claim 7, characterized in that: After determining the time threshold according to the first attribute and the second attribute, the anomaly aggregation method further includes: When the current exception occurs, updating the first attribute and the second attribute; The time threshold is re-determined using the updated first attribute and the second attribute.

9. The abnormal polymerization method according to claim 8, characterized in that: If the current exception does not meet the preset aggregation condition, adding the current exception as a new aggregation exception to the existing aggregation exceptions includes: If the exception occurrence time of the current exception is not less than the re-determined time threshold, the current exception is connected to the existing aggregated exceptions as an aggregated exception of a new exception type.

10. An anomaly aggregation system based on attribute weight and rule-driven, characterized in that: include: The acquisition module is used to obtain multiple original exceptions and current exceptions generated by different devices from different manufacturers; A unification module, used for unifying the formats of the multiple original exception data, so that the formats of the multiple original exceptions are unified into a data model definition format for describing multiple conditional attributes of the original exceptions; A construction module, used to construct an exception information decision table based on a plurality of original exceptions in a format defined by the data model; The abnormal information decision table includes the original abnormal set, a finite non-empty attribute set containing all attributes of the original abnormality, a value range of the finite non-empty attribute set, an information function representing the attribute value of the original abnormal data, a conditional attribute set, and a decision attribute set; the conditional attribute set includes other conditional attributes in the data model definition format except the abnormality type; The first calculation module is used to calculate the knowledge granularity and rough conditional entropy of the conditional attribute set of anomalies of multiple anomaly types according to the same value relationship between the conditional attribute and the decision attribute; The second calculation module is used to calculate the attribute weight of each condition attribute for each abnormal type according to the rough condition entropy of the decision attribute set and the discernibility of the condition attribute set for each abnormal type; An aggregation module, configured to aggregate the current exception into an existing aggregated exception that matches the current exception when the current exception meets a preset aggregation condition, and to add the current exception as a new aggregated exception into the existing aggregated exception if the current exception does not meet the preset aggregation condition; Among them, the aggregation conditions include: the same anomaly type, the anomaly occurrence time is less than the time threshold, and there is at least one aggregation method, the aggregation method includes a first aggregation method according to a predefined aggregation rule and / or a second aggregation method according to attribute weight similarity.