Security data acquisition optimization method based on characteristic value feedback

Abstract

The invention provides a safety data acquisition optimization method based on characteristic value feedback. The method comprises an iteration step based on alarm feedback and an acquisition optimization step based on safety characteristic value feedback. In the iteration step based on alarm feedback, multiple iterations are carried out in an existing alarm model according to an intermediate alarm result, and finally a more accurate alarm knowledge model is generated. And the collection optimization step based on the safety characteristic value feedback adopts the safety characteristic value to filter the real-time data collection data. Through the method, the security feature value is extracted, and the network flow data is filtered according to the security feature value, so that the purposes of reducing data, reducing redundant data and improving real-time analysis efficiency are achieved, and security data acquisition optimization in a big data environment is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of security data collection methods, and in particular to a security data collection optimization method based on characteristic value feedback. Background Art

[0002] With the increasing number of network security threats, security data collection has become an important part of network security protection. Effective data collection can not only help identify and respond to potential security incidents, but also provide a basis for the formulation and optimization of security strategies. However, traditional security data collection methods often have the following shortcomings: Data redundancy: In actual applications, security devices and systems generate a large amount of logs and data, which contain a lot of redundant information. These redundant data not only take up storage space, but also increase the complexity of analysis and processing.

[0003] Lack of real-time performance: Security data collection usually requires a quick response in order to detect and handle security threats in a timely manner. However, traditional methods often fail to provide real-time feedback and optimization, resulting in the inability to identify potential threats in a timely manner.

[0004] Difficulty in feature selection: In a complex security environment, how to accurately select and extract effective features to improve the efficiency and accuracy of data collection is a challenging task. Incorrect feature selection may lead to false positives or false negatives.

[0005] System resource consumption: Large amounts of data collection will put pressure on system resources and affect the overall performance of the system. How to optimize resource consumption while ensuring data integrity is a problem that needs to be solved. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a security data collection optimization method based on eigenvalue feedback. Through this method, security eigenvalues ​​are extracted, and network traffic data is filtered according to the security eigenvalues, so as to achieve data reduction, reduce redundant data, and improve real-time analysis efficiency, thereby realizing security data collection optimization in a big data environment.

[0007] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a security data acquisition optimization method based on characteristic value feedback, comprising an iterative step based on alarm feedback and an acquisition optimization step based on security characteristic value feedback; the iterative step based on alarm feedback performs multiple iterations in the existing alarm model with intermediate alarm results, and finally generates a more accurate alarm knowledge model; the acquisition optimization step based on security characteristic value feedback uses security characteristic values ​​to filter real-time data acquisition data.

[0008] In a preferred embodiment, the alarm knowledge model is established by adopting a hybrid method in ontology theory; in ontology theory, the ontology includes five types of modeling primitives: alarm class, alarm mapping relationship, alarm function, alarm axiom and alarm instance.

[0009] In a preferred embodiment, the network security situation awareness platform relies on real-time correlation monitoring of data obtained through pre-collection, and collects and monitors security devices, network devices, security subsystems, and business systems. It uses agents, syslog, webservice, jdbc, web crawlers, and network capture methods to uniformly collect and preprocess large-scale, multi-dimensional, and high-noise massive information security basic data based on a security alarm model.

[0010] In a preferred embodiment, the real-time data collection data is filtered using security feature values, thereby achieving the purpose of screening and dimensionality reduction of security data.

[0011] In a preferred embodiment, the collected network traffic data is mirrored into two parts, one part is sent to offline storage, and the other part is used as input data for real-time analysis.

[0012] In a preferred embodiment, security feature analysis is performed on offline stored network traffic data, and the obtained security feature values ​​are stored in a security feature library; security feature analysis is performed periodically, and security feature analysis is performed on network traffic data stored offline in the most recent period to obtain new security feature values ​​and update the security feature library; when performing security feature analysis, the offline stored network traffic data is reorganized and restored into session flows, and vector segmentation is performed on the session flows as input for security feature analysis, an appropriate machine learning algorithm is selected, and vector parameters are adjusted to obtain security feature values.

[0013] In a preferred embodiment, during real-time analysis, the network traffic data is filtered according to the security feature value in the security feature library. If it matches the security feature value, the network traffic data is security data, and the security data is analyzed in real time. During the reorganization and restoration process, data with the same five-tuple is marked as a session flow. During filtering, if it does not match the security feature value, the network traffic data is discarded. The network traffic data is transmitted in the form of a message, and the security feature value includes a regular expression and a fingerprint feature word. During filtering, the message header content is parsed to analyze whether the message header content matches the regular expression. The specific content of the message payload is not parsed, and the degree of conformity with the fingerprint feature word is judged through its binary stream. When the message header content matches the regular expression and the binary stream of the message payload part matches the fingerprint feature word, it means that the message matches the security feature value.

[0014] Compared with the prior art, the present invention has the following beneficial effects: through this method, security feature values ​​are extracted, and network traffic data is filtered according to the security feature values, so as to achieve the purpose of data reduction, reduce redundant data, and improve real-time analysis efficiency, thereby realizing the optimization of security data collection in a big data environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a model diagram of the alarm knowledge base of a preferred embodiment of the present invention; Figure 2 This is a flow chart of a method for optimizing security data collection based on eigenvalue feedback according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0017] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0018] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0019] The present invention provides a safety data collection optimization method based on characteristic value feedback, referring to Figure 1-2 , improving the effectiveness and flexibility of network security protection.

[0020] Security event correlation analysis can filter and aggregate these relatively isolated data through effective correlation between various security events and between security events and the context of the environment, eliminate the false and retain the true, and discover the hidden According to the real connection behind it, it provides administrators with more complete, more reliable, more readable and more valuable information. The security event correlation model is an analysis and description of security events and related information, and is the basis for effective correlation of security events.

[0021] Currently, there are not many models for security event correlation. The M2D2 model proposed by the industry is a formalized alarm correlation information model. It uses information system characteristics, vulnerabilities, security tools and event information to correlate alarms generated by different IDSs to reduce the number of alarms and obtain more useful information than a single alarm. The M2D2 model can reduce the false alarm rate to a certain extent, but it is not perfect. The four types of information it uses are not enough to effectively reduce the false alarm rate, and it is powerless against missed alarms.

[0022] If we only perform correlation analysis on the data without providing feedback on the generated results to form a closed processing loop, the results of the correlation analysis will not be accurate enough.

[0023] The present invention proposes an iterative method based on alarm feedback and a collection optimization method based on safety feature value feedback, and introduces a feedback mechanism into a real-time monitoring framework to make the correlation analysis result more accurate.

[0024] 1) Iterative analysis method based on alarm feedback The monitoring of complex attacks with multiple steps is considered as a continuous process. It is not enough to detect only single-point alarms. It is necessary to refine the existing correlation analysis results, so as to conduct a deeper analysis. For the alarms generated by the correlation analysis, the analysis results are extracted again, and the correlation analysis is iterated to generate new alarms. The present invention refers to the research methods of automated domain concept extraction and concept relationship mining in domain modeling, and iterates the intermediate alarm results in the existing alarm model for multiple times to finally generate more accurate alarms. For this reason, the demand of "forming a complete set of models or libraries, and the results of the models or libraries can be fed back to form a closed loop of disposal" arises at the historic moment.

[0025] This project forms a complete and highly scalable alarm knowledge base model. The alarm knowledge model is established using a hybrid approach in ontology theory. In ontology theory, ontology contains five types of modeling primitives: class (concept), relationship, function, axiom, and instance. The alarm knowledge model also includes these five elements.

[0026] ①Alarm Using ontology-based knowledge modeling tools, a full set of alarm ontology is established for all data sources. The alarm ontology defines a rough range by the data source that generates the alarm, and roughly depicts a standard logical model knowledge base containing about 110 alarms. Because the alarm ontologies from different sources are basically different, the same alarms can be merged because the concept is consistent. Taking "phishing email" as an example, "phishing email" may come from two different classes of security equipment and scene association, but the actual meaning of the alarm class is the same, so it is classified as one class in the alarm knowledge base model.

[0027] ②Alarm mapping relationship Then establish a local ontology for each type of alarm. Based on the existing general alarm ontology, define the mapping between the alarm ontology and the alarm source ontology, the alarm ontology and the data source, and the alarm ontology and the four quadrants of security situation - internal threats, external attacks, operating status, and behavioral anomalies. Since the attributes (that is, the attributes of the alarm ontology) are extensible, the alarm mapping relationship reflects the relationship between the alarm class and the attributes, so the alarm mapping relationship can also be expanded.

[0028] ③Alarm function The alarm function is also a manifestation of the alarm mapping relationship, which reflects the specific rules for converting data into alarms. However, in the alarm model, unlike the mathematical model, there is no fixed function expression to determine how the alarm and data are generated, but a rule with a configurable threshold is used to convert data into alarms. Taking the "phishing email" alarm class as an example, the relevant alarm function is: 1) From the log of the "phishing email" alarm class scanned by the security device, the "high-risk" and "medium-risk" logs scanned, one log generates one alarm; at the same time, the logs are merged based on the three attributes of "attack source IP", "attacked IP", and "alarm event type" according to the predetermined threshold (tentative value: 100 times in 5 minutes), and the logs that meet the three attributes and the predetermined threshold are merged into one alarm, and the highest level of the alarm logs that meet the conditions determine the level of the alarm. 2) The generation rule of the "phishing email" alarm class derived from the scene association analysis is: the email body contains a URL and the URL is in the malicious library.

[0029] ④ Alarm Axiom Ontology is a group of concepts connected by inclusion relationships. Adding axioms to the ontology means that the relationship between richer concepts can be expressed, so that all the facts in the field can be described as true. Alarm axioms are the specific descriptions and behaviors of different alarms, which are well-known and certain facts in the security field. Take "phishing emails" as an example. In the information security industry, "phishing emails" are a type of disguised emails that deceive recipients into replying information such as account numbers and passwords to designated recipients, or guide recipients to connect to specially designed web pages to enter key information, in order to steal users' private information such as bank cards. This is the alarm axiom about "phishing emails."

[0030] ⑤Alarm Example In the alarm model, it is the specific name of the alarm, rather than a vague and general name such as "virus". The instance is the smallest object in the ontology. Because it is atomic, all the alarms represented are more readable and the handling of these alarm instances is more convenient. Take "virus", "worm", "Trojan horse" and "botnet" from security devices as examples.

[0031] The ontology-based information security alarm model limits the scope of the alarm knowledge base, so that the original massive alarm quantity and types can be compressed. By applying this alarm model in S6000, the effects of dimensionality reduction, orderly organization of alarm information, and quantity compression of massive alarms can be achieved. Based on the reasoning results of the alarm ontology, global query statements are analyzed to generate query statements for different alarms to achieve unified access.

[0032] In the alarm knowledge base, alarm knowledge is mainly displayed based on alarm (name), and alarms are used to uniquely identify different types of alarms, supporting the expansion of various dimensional attributes of alarms. The inherent attributes are as follows.

[0033] a) "Alarm (name)" is summarized and summarized into about 100 common alarms according to various log sources and the principle of "understanding the meaning by name". "Alarm" is used as the center point to expand its different attributes, and the attributes themselves can also be expanded; b) The "rule" attribute describes the rules for merging and deduplicating the collected normalized logs for different alarms (i.e., alarm names); c) The "Source" attribute is the system from which different alarms (i.e., alarm names) / logs come from, and is also a basis for sorting alarm categories (alarm names).

[0034] d) The "security situation classification" attribute is an important dimension basis for situation analysis and is divided into four categories: abnormal behavior, internal network warning, external attack, and operating status.

[0035] e) The "national standard first-level classification" attribute is used to classify alarms (i.e., alarm names) according to the national standard shown in the attachment. The classification in the national standard is divided into the following six categories: sabotage events, inherent defects, abnormal behavior, information content security events, network attack events, and harmful program events. It is the most coarse-grained search and statistical dimension basis for alarms; f) The "National Standard Secondary Classification" attribute divides the alarms according to the secondary classification in some national standards. There are about twenty secondary classifications. When users want to see a rough alarm, they can view this attribute. For example, for different types of Trojans that are of concern, the alarm name needs a slightly more detailed representation feature. But if the user does not need such a detailed alarm name, they can view this attribute. This attribute is a coarse-grained representation. In this example, the national standard secondary classification is "Trojan event". It is a fine-grained search and statistical dimension basis for alarms, but it is still not "clearly pointed in the alarm name"; g) Security event categories The security event categories defined by the Joint Research Institute at the beginning of system design are as follows: advanced analysis events, correlation analysis events, email security scenario events, security violation events, website security scenario events, specific business security scenario events, vulnerability events, configuration non-compliance events, untimely policy update events, and boundary security scenario events.

[0036] h) "Level" attribute analyzes and defines different alarm levels according to actual business needs. Currently, there are the following levels: high, medium, low, based on rules, and based on logs. Because some levels are based on logs, i) “Handling Suggestions” attribute: Suggestions on actions that users can take when seeing a corresponding alarm in the system.

[0037] j) The "alarm description" attribute represents the description of different alarms, describing what kind of behavior the alarm is. The alarm knowledge base model diagram is shown in Figure 2.

[0038] 2) Acquisition optimization method with safety feature value feedback The real-time correlation monitoring of the network security situation awareness platform relies on the data obtained by the pre-collection. It is aimed at the collection and monitoring objects such as security equipment, network equipment, security subsystems, business systems, etc., and adopts agents, syslog, webservice, jdbc, network crawlers, network capture, etc., based on the security alarm model to uniformly collect and preprocess the large-scale, multi-dimensional, high-noise massive information security basic data. In order to improve the efficiency of monitoring and analysis processing and improve the hit rate, the present invention proposes a security data collection optimization method based on feature value feedback, which uses security feature values ​​to filter real-time data collection data, thereby achieving the purpose of screening and reducing the dimension of security data, realizing the iterative update of the security feature rule base, and improving the real-time performance of the system in the big data environment.

[0039] Feedback method is as follows Figure 2 shown.

[0040] Method steps: a) The collected network traffic data is mirrored into two parts, one part is sent to offline storage, and the other part is used as input data for real-time analysis; i. Perform security feature analysis on the offline stored network traffic data and store the obtained security feature values ​​into the security feature library; Periodically perform security feature analysis on the network traffic data stored offline in the most recent period, obtain new security feature values, and update the security feature library.

[0041] When performing security feature analysis, the offline stored network traffic data is reorganized and restored into session flows, and the session flows are vector-segmented as input for security feature analysis. An appropriate machine learning algorithm is selected, and the vector parameters are adjusted to obtain security feature values.

[0042] ii During real-time analysis, the network traffic data is filtered according to the security feature value in the security feature library. If it matches the security feature value, the network traffic data is security data and real-time analysis is performed on the security data. During the reorganization and restoration process, data with the same five-tuple is marked as a session flow. During filtering, if it does not match the security feature value, the network traffic data is discarded.

[0043] Network traffic data is transmitted in the form of messages, and security feature values ​​include regular expressions and fingerprint feature words. During filtering, the message header content is parsed to analyze whether the message header content matches the regular expression. The specific content of the message payload is not parsed, and its binary stream is used to determine the degree of compliance with the fingerprint feature word. When the message header content matches the regular expression and the binary stream of the message payload part matches the fingerprint feature word, it means that the message matches the security feature value.

[0044] Through this method, security feature values ​​are extracted and network traffic data is filtered according to the security feature values, so as to achieve the purpose of data reduction, reduce redundant data, improve real-time analysis efficiency, and realize the optimization of security data collection in a big data environment.

[0045] The beneficial effects of the present invention include Optimize data collection strategy: This solution aims to dynamically adjust the data collection strategy through the characteristic value feedback mechanism to achieve priority collection of important data and reduce the collection of redundant information.

[0046] Improve real-time performance: Through real-time feature value analysis and feedback, security incidents can be quickly identified and responded to, ensuring that immediate measures can be taken when security incidents occur, thereby improving overall security protection capabilities.

[0047] Effective feature selection: Utilize the feature value feedback mechanism to continuously monitor and evaluate the importance of features, thereby dynamically adjusting the feature set to improve the accuracy and efficiency of data collection.

[0048] Reduce system burden: By optimizing the data collection process, reducing unnecessary data storage and processing, and reducing the consumption of system resources, ensure that the system can still run stably under high load.

[0049] Strong adaptability: As the network environment and attack methods change, the importance of features will also change. This solution will achieve adaptive adjustment to environmental changes and ensure that the data collection strategy is always effective.

Claims

1. A safety data acquisition optimization method based on eigenvalue feedback, characterized in that: It includes an iterative step based on alarm feedback and an acquisition optimization step based on safety feature value feedback; the iterative step based on alarm feedback performs multiple iterations in the existing alarm model with the intermediate alarm result, and finally generates a more accurate alarm knowledge model; The collection optimization step based on safety feature value feedback uses safety feature values ​​to filter the real-time data collection data.

2. The method for optimizing security data collection based on eigenvalue feedback according to claim 1, characterized in that: The alarm knowledge model is established by using a hybrid method in ontology theory; in ontology theory, the ontology includes five types of modeling primitives: alarm class, alarm mapping relationship, alarm function, alarm axiom and alarm instance.

3. The method for optimizing security data collection based on eigenvalue feedback according to claim 1, characterized in that: The real-time correlation monitoring of the network security situation awareness platform relies on the data obtained by the pre-collection. It collects and monitors security devices, network devices, security subsystems, and business systems. It uses agents, syslog, webservice, jdbc, web crawlers, and network capture methods to uniformly collect and preprocess large-scale, multi-dimensional, and high-noise massive information security basic data based on the security alarm model.

4. The method for optimizing security data collection based on characteristic value feedback according to claim 3 is characterized in that: The security feature value is used to filter the real-time data collection data, so as to achieve the purpose of screening and reducing the dimension of security data.

5. The method for optimizing safety data collection based on characteristic value feedback according to claim 4 is characterized in that: The collected network traffic data is mirrored into two parts, one part is sent to offline storage, and the other part is used as input data for real-time analysis.

6. The method for optimizing safety data collection based on characteristic value feedback according to claim 5, characterized in that: Perform security feature analysis on the offline stored network traffic data, and store the obtained security feature values ​​in the security feature library; periodically perform security feature analysis on the network traffic data stored offline in the most recent period, obtain new security feature values, and update the security feature library; When performing security feature analysis, the offline stored network traffic data is reorganized and restored into a session flow, and the session flow is vector-segmented as the input of the security feature analysis. An appropriate machine learning algorithm is selected, and the vector parameters are adjusted to obtain the security feature value.

7. The method for optimizing safety data collection based on characteristic value feedback according to claim 5, characterized in that: During real-time analysis, the network traffic data is filtered according to the security feature value in the security feature library. If it matches the security feature value, the network traffic data is security data and the security data is analyzed in real time. During the reorganization and restoration process, data with the same five-tuple is marked as a session flow. During filtering, if it does not match the security feature value, the network traffic data is discarded. The network traffic data is transmitted in the form of messages, and the security feature value includes a regular expression and a fingerprint feature word. During filtering, the message header content is parsed to analyze whether the message header content matches the regular expression. The specific content of the message payload is not parsed, and the degree of conformity with the fingerprint feature word is judged through its binary stream. When the message header content matches the regular expression and the binary stream of the message payload part matches the fingerprint feature word, it means that the message matches the security feature value.

Images