Network data collection method, device, computer-readable storage medium and electronic device

Abstract

The present disclosure provides a network data collection method, device, computer-readable storage medium and electronic device, which relate to the field of network security analysis technology, wherein the method includes: determining task configuration information; determining the association relationship between the task and the data source, the association relationship between the task and the detection engine, the association relationship between the task, the detection rule and the detection engine according to the task configuration information; in response to the operation of task initiation, receiving real-time traffic data carrying the data source identifier, and for each task in the basic information of the task, performing corresponding operations based on the task configuration information. The present disclosure achieves the technical effect of fully expanding the analyzable data dimensions and enhancing the analysis capability under the premise of efficient analysis, thereby effectively and accurately screening out pre-selected data for threat analysis.

Description

Technical Field

[0001] The present disclosure relates to the technical field of network security analysis, and in particular to a network data collection method, device, computer-readable storage medium, and electronic device. Background Art

[0002] Network security usually refers to the security of computer networks, but in fact it can also refer to the security of computer communication networks. It requires that the hardware, software and data of the network system be fully protected. Therefore, in order to ensure network security, real-time network traffic attack (or threat) analysis is particularly important.

[0003] Currently, the market mainly relies on traffic detection equipment to detect and analyze traffic data, directly obtain detection results, and discover potential threats.

[0004] However, the relevant technologies lack the ability to make comprehensive use of data, and are therefore unable to effectively and accurately screen data for threat analysis. Summary of the invention

[0005] The main purpose of the present disclosure is to provide a network data collection method, device, computer-readable storage medium and electronic device to solve the problem in the related art that data cannot be effectively and accurately screened for threat analysis.

[0006] In order to achieve the above object, the first aspect of the present disclosure provides a network data collection method, comprising:

[0007] Determine task configuration information, wherein the task configuration information includes configuration information of at least one task, and the configuration information for each task includes basic task information, data source configuration information, detection engine configuration information, and detection rule configuration information;

[0008] Determine, according to the task configuration information, the association relationship between the task and the data source, the association relationship between the task and the detection engine, and the association relationship between the task, the detection rule, and the detection engine;

[0009] In response to the task start operation, real-time traffic data carrying a data source identifier is received, and for each task in the task basic information, the following operations are performed based on the task configuration information:

[0010] According to the data source identifier of the real-time traffic data, target real-time traffic data identical to the data source identifier in the data source configuration information is obtained from the real-time traffic data, and according to the association relationship between the task and the data source, the task identifier of the task is marked on the target real-time traffic data;

[0011] Distribute the detection rules in the detection rule configuration information to corresponding detection engines according to the association relationship among the tasks, the detection rules and the detection engines;

[0012] According to the association between the task and the detection engine, the target real-time traffic data carrying the task identifier is sent to the corresponding detection engine, and the detection engine performs real-time detection on the target real-time traffic data based on the distributed detection rules to obtain alarm data; wherein the alarm data is used to determine pre-selected data for threat analysis.

[0013] Optionally, further, sending the target real-time traffic data carrying the task identifier to the corresponding detection engine according to the association relationship between the task and the detection engine includes:

[0014] Determining a target detection engine for real-time detection in the task according to an association relationship between the task and the detection engine;

[0015] If there are multiple target detection engines corresponding to the task, the target real-time traffic data is copied;

[0016] The copied target real-time traffic data is sent to the multiple target detection engines respectively; wherein one piece of target real-time traffic data corresponds to one target detection engine.

[0017] Optionally, further, if there are multiple target detection engines corresponding to the task, copying the target real-time traffic data includes:

[0018] If there are multiple target detection engines corresponding to the task, determine the number of the target detection engines; wherein the detection engines include at least two of the following: an intrusion detection system IDS attack traffic detection engine, an attack injection behavior detection engine, a sample analysis detection engine, and an encryption consistency detection engine, and the detection rules are used to represent a detection feature description language that conforms to grammatical specifications;

[0019] If the target real-time traffic data is real-time traffic data carrying a data source identifier, the target real-time traffic data is copied so that the number of the copied target real-time traffic data is the same as the number of the target detection engines;

[0020] If the target real-time traffic data is real-time traffic data carrying multiple data source identifiers, then for each data source identifier, the real-time traffic data of each data source identifier is copied respectively, so that the number of real-time traffic data of each data source identifier after copying is the same as the number of the target detection engines.

[0021] Optionally, further, the detecting engine performs real-time detection on the target real-time traffic data based on the distributed detection rules to obtain alarm data, including:

[0022] If the detection rule includes an extended rule type, the detection rule is uniformly converted into an XML standard syntax rule;

[0023] Among them, the rules corresponding to the extended rule type are determined based on threat intelligence information and attack indicators IOC.

[0024] Optionally, further, the method further includes:

[0025] Send the alarm data output by each detection engine to the Kafka message processor to aggregate and cache the alarm data;

[0026] According to each of the task identifiers, obtain the alarm data corresponding to each of the tasks from the kafka message processor;

[0027] For each of the tasks, the warning data corresponding to the task is processed to determine pre-selected data for threat analysis.

[0028] Optionally, further, for each of the tasks, processing the warning data corresponding to the task to determine pre-selected data for threat analysis includes:

[0029] For each of the tasks, deduplication and clean the alarm data corresponding to the task;

[0030] Standardize the field values ​​of heterogeneous data in the deduplicated and cleaned alarm data;

[0031] The standardized alarm data is enriched to generate the pre-selected data for threat analysis.

[0032] Optionally, further, the method further includes:

[0033] Perform threat analysis based on the pre-selected data to obtain intelligence information, wherein the intelligence information includes threat intelligence information and attack indicators;

[0034] The detection rules are updated or expanded according to the threat intelligence information and the compromise indicators.

[0035] A second aspect of the present disclosure provides a network data collection device, comprising:

[0036] A determination unit, configured to determine task configuration information, wherein the task configuration information includes configuration information of at least one task, and the configuration information for each task includes basic task information, data source configuration information, detection engine configuration information, and detection rule configuration information;

[0037] The determining unit is further used to determine, according to the task configuration information, an association relationship between a task and a data source, an association relationship between a task and a detection engine, and an association relationship between a task, a detection rule, and a detection engine;

[0038] The processing unit is used to receive the real-time traffic data carrying the data source identifier in response to the task start operation, and perform the following operations for each task in the task basic information based on the task configuration information:

[0039] According to the data source identifier of the real-time traffic data, target real-time traffic data identical to the data source identifier in the data source configuration information is obtained from the real-time traffic data, and according to the association relationship between the task and the data source, the task identifier of the task is marked on the target real-time traffic data;

[0040] Distribute the detection rules in the detection rule configuration information to corresponding detection engines according to the association relationship among the tasks, the detection rules and the detection engines;

[0041] According to the association between the task and the detection engine, the target real-time traffic data carrying the task identifier is sent to the corresponding detection engine, and the detection engine performs real-time detection on the target real-time traffic data based on the distributed detection rules to obtain alarm data; wherein the alarm data is used to determine pre-selected data for threat analysis.

[0042] A third aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the network data collection method provided by any one of the first aspects.

[0043] The fourth aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the network data collection method provided in any one of the first aspects.

[0044] A fifth aspect of the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the network data collection method provided in any one of the first aspects is implemented.

[0045] In the network data collection method provided in the embodiment of the present disclosure, after the task is started, based on the task configuration information and the association relationship between the task and the data source, the detection engine, and the detection rule, data detection is performed synchronously for each task, and the target real-time traffic data matching each task is preliminarily screened out from the received real-time traffic data carrying the data source identifier, thereby achieving preliminary screening of the received real-time traffic data and providing relatively effective traffic data for subsequent detection to obtain pre-selected data; by using multi-dimensional concurrent detection of multiple detection engines with configurable detection rules, the purpose of obtaining suspected attack pre-selected data from multiple detection perspectives is achieved, thereby achieving the technical effect of fully expanding the analyzable data dimensions and enhancing the analysis capability under the premise of efficient analysis, thereby solving the technical problem that the pre-selected data cannot be effectively and accurately screened out for threat analysis due to insufficient comprehensive data application capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of a scenario of a network data collection method provided by an embodiment of the present disclosure;

[0048] Figure 2A A schematic diagram of a scenario for data source identification provided in an embodiment of the present disclosure;

[0049] Figure 2B A schematic diagram of a scenario of a data source identifier and a task identifier provided in an embodiment of the present disclosure;

[0050] Figure 3 A flowchart of a network data collection method provided by an embodiment of the present disclosure;

[0051] Figure 4 A schematic diagram of a task-driven process provided by an embodiment of the present disclosure;

[0052] Figure 5 A schematic diagram of data identification and length definition provided in an embodiment of the present disclosure;

[0053] Figure 6 A schematic diagram of the flow and structural changes of pre-selected data provided in an embodiment of the present disclosure;

[0054] Figure 7 A block diagram of a network data collection device provided in an embodiment of the present disclosure;

[0055] Figure 8 A block diagram of an electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the scheme of the present disclosure, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present disclosure.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0058] In the present disclosure, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present disclosure and its embodiments, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0059] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this disclosure can be understood according to specific circumstances.

[0060] In addition, the terms "installed", "set", "provided with", "connected", "connected", and "socketed" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection, or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be an internal connection between two devices, elements, or components. For those of ordinary skill in the art, the specific meanings of the above terms in this disclosure can be understood according to specific circumstances.

[0061] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] At present, in the relevant technologies, since various detection products are targeted at specific detection targets, the target alarm types usually generated are single, and the comprehensive application capabilities of real-time traffic data are insufficient; at the same time, due to various detection equipment and single indicator detection, the alarm false alarm rate is high, and effective information is usually submerged in the massive alarm information, requiring a lot of manual identification to remove duplicate false alarm information. Therefore, the relevant technology cannot effectively and accurately filter out data for threat analysis, and thus cannot improve network security.

[0063] In order to solve the above problems, the technical concept of the present invention is to configure tasks before in-depth network attack threat analysis. Based on task configuration information, for any task, relatively comprehensive and accurate threat alarm pre-selected information can be extracted from massive real-time network traffic from multiple dimensions. Before analyzing the pre-selected data, the association between the task and the data source identifier in the task configuration information can be used to remove massive invalid information from the real-time traffic data carrying the data source identifier. In order to improve the comprehensive application ability of traffic data, multiple detection engines configured with task rules are used to synchronously perform real-time detection on the target real-time traffic data obtained by preliminary screening, thereby improving the effectiveness and completeness of the threat analysis of the pre-selected data, and improving the efficiency and accuracy of real-time analysis.

[0064] In practical applications, the execution subject of the present disclosure may be a network data acquisition device, which may be deployed in an electronic device, such as a terminal device, a server, etc. A user may configure a task through an electronic device deployed with a network data acquisition device, such as configuring: basic task information, data source configuration information, detection engine configuration information, and detection rule configuration information. Through the above configuration, the task configuration information also stores the association between the task and the data source, the association between the task and the detection engine, and the association between the task, the detection rule, and the detection engine. When a task is started, the electronic device deployed with the network data acquisition device may perform detection operations synchronously through multiple detection engines based on one or more tasks to obtain alarm data, and obtain pre-selected data for threat analysis based on the alarm data processing of each task.

[0065] For example, see Figure 1 As shown, Figure 1 A schematic diagram of a scenario of a network data collection method provided in an embodiment of the present disclosure. The network data collection method here can be used for pre-collection of pre-selected data based on real-time traffic threat analysis, and can also be used for collection of other pre-selected data, which is not specifically limited here. The scenario includes a data collection system (or a pre-data collection system, such as Figure 1 The data collection system includes a task driver, an extended rule manager, a rule definition diverter (or a rule definition diverter device, which will not be described in detail below), a multi-dimensional concurrent detection engine, a Kafka message processor, and a data preprocessing engine. The threat analysis system is used to generate threat analysis pre-selected data based on real-time standard alarm data from the data collection system to perform threat analysis, obtain intelligence data, and send the intelligence data (such as IOC intelligence) to the extended rule manager so that the extended rule manager can perform rule expansion.

[0066] The task driver supports multi-task concurrency, and is used to send task configuration information to the rule definition diversion device, and send rule configuration information to the extended rule manager based on the configuration information.

[0067] The rule defines the diversion device, which is used to receive real-time traffic data, and filter out the data that is not included in the data source according to the task configuration information distributed by the task driver, and mark the real-time traffic data of the included data source with the task identifier at the end, and finally copy multiple copies of the traffic according to the detection engine used for the task traffic, and distribute them to the designated port connected to the engine. Among them, the rule defines the diversion device data source tag format description, see Figure 2AAs shown. For task labeling, specifically, by copying and distributing traffic data, multiple copies of real-time traffic data with task identification (ID) (here refers to the target real-time traffic data carrying the task identification) are sent to the corresponding detection engine for detection (for example: Intrusion Detection Systems (IDS) attack traffic detection, attack injection behavior detection, sample analysis detection, encryption consistency, etc.). One way to implement labeling is to add a task identification at the end of the real-time traffic data of the included data source based on the association between the task and the data source identification; another way to implement labeling is to uniformly convert both the carried data source identification and the task identification based on the association between the task and the data source identification, see Figure 2B As shown: the solid line frame (i.e. thin solid line) is the data source ID (identification) part: the data source ID identification is converted to "9876" and "0012"; the dotted line frame is the task ID part: the task ID identification is converted to "9877" and "0112".

[0068] The extended rule manager is used for rule type management, extended rule base management, rule conversion, etc. Specifically, after the task is started, if the extended rule is configured in the task, the task driver completes the unified rule conversion based on the configuration information and the extended rule base. Among them, the rule base management and the conversion method of issuing rules are: unified management of detection rules, support for converting threat intelligence and indicators of compromise (IOC) (for example: signatures, malicious sample hash values, virus signatures, behavior models) into detection rules, as well as batch import and editing of sericata, snort, yara, and sigma syntax rules, and converting them into statements of unified syntax rules according to the task to achieve multi-engine adaptation of detection rules.

[0069] Multi-dimensional concurrent detection engines (including: intrusion detection system IDS attack traffic detection engine, attack injection behavior detection engine, sample analysis detection engine, encryption consistency detection engine) are used to synchronize concurrent detection according to the detection rules in the task configuration information after the task is started. Specifically, if no extended rules are configured in the task, each detection engine uses the engine's built-in rules for concurrent threat detection; if extended rules are configured in the task, each engine obtains new rules and updates them according to the task-driven configuration, and synchronizes concurrent detection according to the extended rules. Each engine detection generally generates independent alarm information results (for example, alarm data) based on different detection contents.

[0070] The Kafka message processor is used for the aggregation and cache management of alarm data. Specifically, the Kafka message processor receives the multi-channel detection alarm information sent by the multi-dimensional concurrent detection engine, and aggregates and caches the multi-channel detection alarm information according to the task ID.

[0071] The data preprocessing engine is used for preprocessing of aggregated alarm data. The preprocessing of aggregated alarm data mainly includes distributed high-speed data collection, data deduplication and cleaning, data standardization, and primary automatic enrichment of data.

[0072] Through the above-mentioned data collection system, the present invention can extract relatively comprehensive and accurate threat alarm pre-selected information from massive real-time network traffic from multiple dimensions before in-depth network attack threat analysis, and remove massive invalid information before analysis, thereby improving the effectiveness and completeness of the pre-selected data threat analysis and improving the efficiency of real-time analysis.

[0073] It should be noted that in the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of information such as traffic data or user data involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0074] The technical solution of the present disclosure is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0075] The present disclosure provides a network data collection method, such as Figure 3 As shown, the method includes the following steps S301 to S303:

[0076] Step S301: determine task configuration information, the task configuration information includes configuration information of at least one task, and the configuration information for each task includes basic task information, data source configuration information, detection engine configuration information, and detection rule configuration information.

[0077] Step S302: According to the task configuration information, determine the association relationship between the task and the data source, the association relationship between the task and the detection engine, and the association relationship between the task, the detection rule, and the detection engine.

[0078] Step S303: In response to the task start operation, real-time traffic data carrying a data source identifier is received, and corresponding operations (eg, the first operation) are performed for each task in the task basic information based on the task configuration information.

[0079] The first operation includes:

[0080] Step a1: according to the data source identifier of the real-time traffic data, obtaining target real-time traffic data having the same data source identifier as that in the data source configuration information from the real-time traffic data, and marking the task identifier of the task on the target real-time traffic data according to the association relationship between the task and the data source;

[0081] Step a2: Distributing the detection rules in the detection rule configuration information to corresponding detection engines according to the association relationship among the tasks, detection rules and detection engines;

[0082] Step a3: According to the association between the task and the detection engine, the target real-time traffic data carrying the task identifier is sent to the corresponding detection engine, and the detection engine performs real-time detection on the target real-time traffic data based on the distributed detection rules to obtain alarm data; wherein the alarm data is used to determine the pre-selected data for threat analysis.

[0083] In the disclosed embodiment, before in-depth network attack threat analysis is conducted, by configuring tasks, based on task configuration information, relatively comprehensive and accurate threat alarm pre-selected information can be extracted from massive real-time network traffic for any task from multiple dimensions. Before the pre-selected data is analyzed, a large amount of invalid information can be removed from the real-time traffic data carrying the data source identifier through the association relationship between the task and the data source identifier in the task configuration information. In order to improve the comprehensive application capability of traffic data, multiple detection engines configured with task rules are used to synchronously perform real-time detection on the target real-time traffic data obtained through preliminary screening, thereby improving the effectiveness and completeness of the threat analysis of the pre-selected data, and improving the efficiency and accuracy of real-time analysis.

[0084] The network data here includes but is not limited to traffic data. The data source identifier may be one or more. The detection engine configuration information includes information of multiple engines. The detection rule configuration information includes a detection rule set and may also include extended rules.

[0085] Real-time traffic data: refers to real-time traffic, which is data transmitted in real time in the network. After being aggregated by the front-end equipment, the data is marked with the data source.

[0086] Task: It is defined as a pre-selected data detection task constructed to achieve threat analysis. The basic elements of the task include: data source set, detection rule set, and detection engine set. The system drives the circuit rule diversion device to perform raw data screening, label conversion, replication, and diversion through these task elements.

[0087] Data source: refers to various types of real-time network traffic data. After passing through the network switching device, a group of data source IDs are marked at the end of the traffic packet. Each data source ID serves as a unique mark of the source of the traffic data.

[0088] Specifically, by providing a multi-task (drive) management strategy, the circuit rule diversion device is driven by task startup to complete the replication and distribution of real-time traffic to each pre-selected engine to achieve parallel detection requirements. The rule diversion device is driven by tasks to determine which engine to use to detect data with which task source.

[0089] Among them, the circuit rule diversion equipment: a network traffic switching device that receives real-time traffic data, provides policy management services, completes flag conversion for hit-tagged data, performs data mirroring replication, load balancing within the group, and distributes the data to designated ports.

[0090] Exemplary, combined Figure 4 As shown, the present disclosure designs a task-driven process to realize data collection: the steps of realizing data collection through the task-driven process include: S1: task creation; S2: rule circuit diversion; S3: rule management; S4: concurrent real-time detection; S5: alarm data aggregation; S6: alarm data preprocessing.

[0091] For multi-task (driver) management: In the multi-task management scenario, for the single-task process: S1: Task creation. Specifically, taking the single-task creation process as an example, the task ID is used as the unique identifier of the task. First, the basic information of the task is created, such as: task creator, owner, task target, planned start and completion time, etc. Then the task configuration also includes data source configuration, detection engine configuration, and detection rule configuration.

[0092] S11: Data source configuration. The data source is a set of data source IDs marked at the end of the traffic packet after passing through the network switching device. Each data source ID is used as a unique mark of the traffic data source. The main data identifier and length are defined as follows: Figure 5 shown.

[0093] According to the task objective requirements, each task can select one or more data sources as input data. The MM association table between the task and the data source is saved in the task configuration (the association table can determine the association between the task and the data source). The structure of the association table is shown in Table 1.

[0094] Table 1 Example of the relationship between tasks and data sources

[0095] Data Source ID Task ID 206357017282088960 274875418799116291

[0096] S12: Detection rule configuration.

[0097] Detection rules refer to the detection feature description language that conforms to the grammatical specifications and can act on a specific detection engine to obtain initial alarm information that conforms to the feature description. Select one or more detection rules in a task, combine them into a rule set, and issue them uniformly after the task is started.

[0098] The corresponding syntax specifications supported by each pre-selected detection engine are as follows:

[0099] IDS attack traffic detection engine: sericata

[0100] Attack injection behavior detection: snort, sericata

[0101] Sample analysis and testing: yara

[0102] Encryption consistency detection: sericata

[0103] By configuring the detection rules for the task, the capabilities of the detection engine preset rules can be expanded, the timeliness of the conversion and utilization of real-time threat intelligence can be improved, and effective pre-selected data can be provided for the discovery of known unknown threats and the synthetic analysis of subsequent events.

[0104] S13: Detection engine configuration. According to the task objectives, select one or more detection engines to be used to achieve parallel detection of traffic data, rational use of detection resources, avoid traffic data blocking, reduce detection backlog and data packet loss. The key data saved in the task configuration is the MM association table between the task and the detection engine (the association table can determine the association between the task and the detection engine).

[0105] S14: Task configuration management. Provides new save (such as task configuration save), edit, modify, view and delete management functions for task configuration information.

[0106] S15: Mission started.

[0107] After the task is started, traffic detection and data collection are performed. The input data detected by the collection system device includes two types: one is the real-time traffic of the data source (that is, the real-time traffic data with the data source identifier), and the other is the task configuration information. Figure 1 The flow detection and data collection principles shown are not repeated here.

[0108] Specifically, after the task is started, the rule-defined diversion device receives real-time data source traffic, and the task driver can support multi-task concurrency and send task configuration information (1 or N task configuration information) to the rule-defined diversion device. Among them, S2: rule circuit diversion. The traffic replication and distribution process depends on the rule-defined diversion device. The main function of the device is to receive real-time traffic data, pre-screen and distribute the traffic data through the data source identification, the relationship between the task and the data source (for example, the identification distribution mapping table), and partially reduce the pressure on the detection engine. The focus is to consider the segmentation and load balancing management of data packets.

[0109] After the task is started, the rule-defined diversion device receives real-time data source traffic. The task driver can support multiple tasks concurrently and send the task configuration information to the rule-defined diversion device. The diversion device first filters out data that is not included in the data source based on the task driver, and then marks the included data source traffic with the task identifier at the end.

[0110] Specifically, S2: rule circuit diversion includes: S21: task matching and task labeling through task and data source mapping; S22: traffic replication and distribution through task and engine (here refers to the detection engine) mapping, and sending the task pre-selected real-time traffic data (here refers to the target real-time traffic data carrying the task identifier) ​​to the corresponding engine for detection. That is, according to the task pre-selected real-time traffic data, S4 is executed.

[0111] S3: Rule management. For a single rule process: S31: Rule creation. During the execution of S12: Detection rule configuration, the created rule set is configured in the detection rule configuration. Unified management of detection rules supports the conversion of threat intelligence and IOC attack indicators (signature words, malicious sample hash values, virus signatures, behavior models) into detection rules, as well as batch import and editing of Sericata, Snort, Yara, and Sigma syntax rules, and conversion into statements of unified syntax rules according to tasks to achieve multi-engine adaptation of detection rules. Specifically, S31 includes: S311: Setting the rule syntax type; S312: Selecting the rule applicable engine; S313: Rule generation and verification.

[0112] After the task is started, if the extended rules are configured in the task, the task driver completes the unified conversion of the rules based on the configuration information and the extended rule base. For the conversion method of the rule base management and the distribution of the rules, please refer to the content described in S12. Each engine consumes and updates the rules through the relevant identifiers in the xml unified rules. Specifically, after the task is started, if the extended rules are configured in the task, execute S32 and S33. Among them, S32: normalized description of detection rules based on xml; S33: distribution of unified rules based on tasks (distribute the unified description detection rule set to the detection engine to execute S4).

[0113] S4: Concurrent real-time detection includes: S41: IDS rule application and real-time detection; S42: sample detection rule application and real-time detection; S43: attack injection rule application and real-time detection; S44: encryption threat rule application and real-time detection; S45: alarm data aggregation / kafka. Specifically, each engine detection generally generates independent alarm information results according to different detection contents. Figure 1 As shown in the collection principle, each engine concurrently generates alarm results and sends them to the Kafka message processor to achieve alarm data aggregation and cache management. The data preprocessing engine performs Kafka data consumption and preliminary clustering based on the task ID.

[0114] S5: Alarm data preprocessing. Including: S51: Data deduplication and cleaning; S52: Data standardization; S53: Data enrichment with intelligence to generate pre-selected data for threat analysis.

[0115] Therefore, the disclosed embodiment can quickly obtain real-time detection results in real time and detect threats at an early stage. That is: based on the replication and distribution of traffic data, through multiple engine detection, it solves the problem that various detection products usually generate a single type of target alarm for specific detection targets, and the comprehensive use of real-time traffic data is insufficient; at the same time, due to various detection devices and multiple indicator detection, the false alarm rate of alarms is reduced, and effective information is usually avoided from being drowned in massive alarm information, and there is no need for a large amount of manual identification to remove false alarm information; thirdly, in addition to relying on internal models, various detection engines also support extensible rules and have the ability to expand the detection of the latest threat intelligence, which can detect unknown threats in a timely manner. Then, pre-selected data is effectively and preparedly collected for threat or attack analysis.

[0116] Optionally, sending the target real-time traffic data carrying the task identifier to a corresponding detection engine according to the association relationship between the task and the detection engine includes:

[0117] Determining a target detection engine for real-time detection in the task according to an association relationship between the task and the detection engine;

[0118] If there are multiple target detection engines corresponding to the task, the target real-time traffic data is copied;

[0119] The copied target real-time traffic data is sent to the multiple target detection engines respectively; wherein one piece of target real-time traffic data corresponds to one target detection engine.

[0120] In the embodiment of the present disclosure, when one task corresponds to multiple detection engines, in order to perform multi-dimensional detection and thereby ensure the accuracy of detection, it is necessary to copy and distribute the screened target real-time traffic data to multiple detection engines for detection in different dimensions.

[0121] Specifically, based on the relationship between tasks and detection engines, it is possible to determine which detection engines are required for each task, and use the detection engine corresponding to each task as the target detection engine to perform real-time detection in the corresponding task. In order to improve the comprehensive use of real-time traffic data, when configuring tasks, multiple detection engines can be configured for the task, and then the detection results in multiple dimensions can be obtained: the target real-time traffic data is copied and then distributed to multiple target detection engines for detection.

[0122] Optionally, if there are multiple target detection engines corresponding to the task, copying the target real-time traffic data includes:

[0123] If there are multiple target detection engines corresponding to the task, determine the number of the target detection engines; wherein the detection engines include at least two of the following: an intrusion detection system IDS attack traffic detection engine, an attack injection behavior detection engine, a sample analysis detection engine, and an encryption consistency detection engine, and the detection rules are used to represent a detection feature description language that conforms to grammatical specifications;

[0124] If the target real-time traffic data is real-time traffic data carrying a data source identifier, the target real-time traffic data is copied so that the number of the copied target real-time traffic data is the same as the number of the target detection engines;

[0125] If the target real-time traffic data is real-time traffic data carrying multiple data source identifiers, then for each data source identifier, the real-time traffic data of each data source identifier is copied respectively, so that the number of real-time traffic data of each data source identifier after copying is the same as the number of the target detection engines.

[0126] In the embodiment of the present disclosure, the replication method of the data source identification dimension under any task can be determined according to the number of data source identifications of the target real-time traffic data. There are at least two situations:

[0127] Case 1: If the data source identifier of the filtered target real-time traffic data under this task is one, it is considered that the number of this target real-time traffic data is one. If there are multiple detection engines corresponding to this task, the target real-time traffic data is copied so that the number of target real-time traffic data is the same as the number of detection engines, and then the copied multiple target real-time traffic data are distributed to multiple detection engines respectively;

[0128] Case 2: If the data source identifier of the filtered target real-time traffic data under this task is one, then the number of this target real-time traffic data is considered to be one. If there are multiple detection engines corresponding to this task, then this target real-time traffic data is copied so that the number of target real-time traffic data is the same as the number of detection engines, and then the copied multiple target real-time traffic data are distributed to multiple detection engines respectively.

[0129] Optionally, the method further comprises:

[0130] For any first real-time traffic data in the target real-time traffic data after marking the task identifier, if the first real-time traffic data corresponds to multiple task identifiers, the first real-time traffic data is copied; wherein the number of the copied first real-time traffic is the sum of the number of engines corresponding to each task identifier;

[0131] The copied first real-time traffic is sent to the detection engine corresponding to each task in the multiple task identifiers, so that each detection engine performs detection.

[0132] In the embodiment of the present disclosure, based on the replication method of the task dimension, if a target real-time traffic data corresponds to multiple tasks, it is necessary to replicate the target real-time traffic so that the number of replicated target real-time traffic is the same as the number of all engines corresponding to the multiple tasks. If there are identical engines between multiple tasks, they are also counted as different numbers (because even if the engines are the same in two tasks, the detection rules configured in the tasks may be different, and even if the same data is detected by the detection engine to which the detection rules are distributed, the detection results may be different. Therefore, replication is also required in this case) to ensure that the detection rules distributed to each engine under each task can have input data for detection.

[0133] Optionally, the performing real-time detection on the target real-time traffic data based on the distributed detection rules by the detection engine to obtain alarm data includes:

[0134] If the detection rule includes an extended rule type, the detection rule is uniformly converted into an XML standard syntax rule;

[0135] Among them, the rules corresponding to the extended rule type are determined based on threat intelligence information and attack indicators IOC.

[0136] In the disclosed embodiment, a multi-dimensional detection strategy design is provided, which selects multiple detection engines to obtain pre-selected alarm data from different analysis perspectives. That is, multiple detection engines are used to perform multi-dimensional concurrent detection on the same traffic, obtain pre-selected data of suspected attacks from multiple detection perspectives, and fully expand the analyzable data dimensions under the premise of achieving efficient analysis, thereby enhancing the analysis capability.

[0137] Among them, this strategy uses a variety of combined detection engines including: IDS attack traffic detection engine, attack injection behavior detection engine, sample analysis detection engine and encrypted data consistency detection engine.

[0138] IDS attack traffic detection includes signature detection, traffic data feature statistics, protocol analysis and behavior analysis. Signature detection mainly solves the signature comparison of typical threats, and is used to detect direct signature features (such as IP, domain name, organization, malicious sample md5, etc.) and variant features (inverse code, complement code, etc. of direct features) carried in the detected traffic to discover potential direct threat attacks. Traffic data feature statistical analysis discovers suspected reconnaissance behaviors such as host and port scanning by counting the size and feature frequency of traffic data packets. Behavioral analysis of protocol analysis identifies suspected host or network device behaviors, DDos attack behaviors, and brute force cracking by analyzing traffic protocols, payload information, a large number of attempted connections, frequent logins, etc.

[0139] The attack injection behavior detection engine regularly updates the consistent vulnerability library to identify special detections of vulnerability exploits such as application layer and data injection, such as Tomcat server penetration attacks, Redis database injections, Oracle database injections, MySQL database injections, Microsoft SQL Server database injections, Apache server penetration attacks, IIS server penetration attacks, etc.

[0140] Sample analysis detection extracts samples and observes the static features and dynamic behaviors of sample files. Static detection discovers malicious samples and marks them with malicious attributes, such as virus signature scanning, vulnerability code identification, vulnerability exploits, Trojan downloaders, worm viruses, etc. Dynamic behavior detection creates an operating system and major software running simulation environment, observes sample running behavior, detects environmental parameters, and discovers malicious sample characteristics and escape behavior characteristics. Detection sample types include executable files, office documents (mainly macro utilization), email types, script files, media files, compressed files, etc. Detection of environmental parameters (operating system, browser, WEB server, DNS server, file server, mail server and related network environment) Escape adversarial behavior detection (hardware environment detection, virtual machine feature detection, CPU instruction detection, system time detection and user behavior detection)

[0141] Encryption consistency detection detects whether data can be encrypted by checking the cryptographic protocol format, cryptographic protocol specified algorithm, including cryptographic parameters, certificate format compliance and rationality, and cryptographic entropy of the protocol header agreement and the encryption features actually used in the transmitted data.

[0142] After the task is started, if no extended rules are configured in the task, each detection engine uses the engine's built-in rules for concurrent threat detection; if extended rules are configured in the task, each engine obtains new rules and updates them according to the task driver configuration, and performs concurrent detection according to the extended rules. Specifically, after the task is started, if extended rules are configured in the task, the task driver completes the unified conversion of rules based on the configuration information and the extended rule base. For the conversion method of rule base management and issued rules, please refer to the content described in S12. Each engine consumes and updates rules through the relevant identifiers in the xml unified rules.

[0143] For unified management of detection rules, it supports the conversion of threat intelligence and IOC attack indicators (signature words, malicious sample hash values, virus signature codes, behavior models) into detection rules, as well as batch import and editing of Sericata, Snort, Yara, and Sigma syntax rules, and converts them into statements of unified syntax rules according to tasks to achieve multi-engine adaptation of detection rules.

[0144] The XML standard syntax rules are used to make the detection syntax rules compatible with various detection engines, and the pre-detection features can be quickly distributed to various engines to work in parallel. The compatible syntax rules include: Sericata, Snort, Yara, Sigma and regular syntax rules.

[0145] This policy defines the following main label formats:

[0146] (xml syntax rules)

[0147] <type> Rule Description Type< / type>

[0148] <engineer>Applicable detection engine< / engineer>

[0149] <rule>Rule Body< / rule>

[0150] Optionally, the method further comprises:

[0151] Send the alarm data output by each detection engine to the Kafka message processor to aggregate and cache the alarm data;

[0152] According to each of the task identifiers, obtain the alarm data corresponding to each of the tasks from the kafka message processor;

[0153] For each of the tasks, the warning data corresponding to the task is processed to determine pre-selected data for threat analysis.

[0154] In the disclosed embodiment, each detection engine generally generates independent alarm information results according to different detection contents. Figure 1 As shown, each engine concurrently generates alarm results and sends them to the Kafka message processor to achieve alarm data aggregation and cache management. The data preprocessing engine performs Kafka data consumption and preliminary clustering based on the task ID.

[0155] Optionally, for each of the tasks, processing the warning data corresponding to the task to determine pre-selected data for threat analysis includes:

[0156] For each of the tasks, deduplication and clean the alarm data corresponding to the task;

[0157] Standardize the field values ​​of heterogeneous data in the deduplicated and cleaned alarm data;

[0158] The standardized alarm data is enriched to generate the pre-selected data for threat analysis.

[0159] In the disclosed embodiment, the warning data is deduplicated, standardized, and enriched in order to generate pre-selected data based on the task identification dimension. Through warning pre-processing, relatively rich, diverse, task-specific, and standardized pre-selected threat analysis data can be formed.

[0160] Specifically, step b1: distributed high-speed acquisition.

[0161] Distributed Kafka message channels are used to cache and receive high-concurrency detection alarm data. Data is collected (consumed) through topic information defined in the producer. Kafka maintains multiple Partition partitions for each topic. Each partition is an ordered, immutable sequence of structured commit log records. In each partition, each log record is assigned a unique serial number. To ensure the security and reliability of data, the Kafka cluster uses distributed multi-copy form to back up information and improve the data robustness of high-speed information transmission.

[0162] Step b2: Data deduplication and cleaning.

[0163] Due to the diversity of engines or the data source itself, the same anomaly detection result may generate multiple related alarm records, so the data needs to be deduplicated and cleaned. This system uses filtering and merging configuration management strategies to complete the first round of automatic deduplication of data. Including vacancy value processing strategy, similarity calculation strategy, duplicate record deletion strategy, data dictionary mapping conversion, attribute value validity verification strategy, etc.

[0164] Step b3: Data standardization processing.

[0165] The main purpose of standardization is to improve the compatibility of unified processing of alarm engine data. Since the alarm data of each detection engine is heterogeneous in alarm structure, data type and operation method, it is necessary to standardize the field values ​​of heterogeneous data to achieve the internal unification of storage structure and field representation meaning. Typical information includes unified conversion of time and date format, conversion of data dictionary mapping value, standardization of alarm level and alarm type, splitting and merging of representation field values, and data conflict value processing.

[0166] Step b4: preliminary automatic enrichment of data.

[0167] The primary automatic enrichment of data referred to in the embodiment of the present disclosure is to first cluster the data sources according to the task ID under the condition of concurrent detection of multiple tasks and multiple data sources, extract the main key IOC indicators and data association relationships, including source IP, port, destination IP, port, transmission protocol, alarm type, threat characteristics, threat description, detection rules, data source ID, etc., and define them as feature labels. Then, the data is complemented with each other according to the data similarity relationship to achieve primary enrichment of data and improve the quality of alarm data.

[0168] Step b5: Standardize data storage.

[0169] The basic characteristics of the data after deduplication, cleaning, standardization, and enrichment are standardized storage of key data, unified representation of data meaning, unified time series standards, and key label information that can be correlated for analysis, forming multi-dimensional detection and real-time analysis pre-selected data that can be analyzed with unified standards. ElasticSearch is used to store key metadata information of traffic, relational databases are used to store standardized alarm pre-selected data, and graph databases are used to store important IDs, IOC labels, and task clustering relationship information, thereby forming multi-dimensional, high-density value threat analysis pre-selected data (or threat pre-selected data) that can be used dynamically, in real time, and with high quality. Among them, the flow and structural changes of pre-selected data are as follows: Figure 6 shown.

[0170] Specifically, real-time traffic data with a data source ID is used as the detection data source. Task configuration information is used as a driver, and the rule-based diversion device converts, copies, and distributes the data to the multi-dimensional detection engine to form heterogeneous data of the initial detection results. After data aggregation, the data undergoes a normalization, completion, and enrichment process to form basic data of network sessions, multi-dimensional threat detection feature data such as attack data features, traffic features, behavior features, vulnerability features, and sample features. The preliminary enrichment of information such as IP-domain names-regions-organizations is completed by complementing each other's data. Finally, a clustered data set with tasks as the core is formed by associating tasks.

[0171] The detection pre-selection process can support the conversion of the latest feature intelligence obtained by users in real time into multi-dimensionally adaptable detection rules, and add them to the detection process in real time based on task-driven, completing the real-time expansion of detection capabilities with fresh information.

[0172] Optionally, the network data collection method may further include:

[0173] Perform threat analysis based on the pre-selected data to obtain intelligence information, wherein the intelligence information includes threat intelligence information and attack indicators;

[0174] The detection rules are updated or expanded according to the threat intelligence information and the compromise indicators.

[0175] In the disclosed embodiment, the intelligence information obtained through subsequent analysis can be fed back to the detection rules to update or expand the detection rules, so that the detection capabilities of the entire system are continuously improved, and the pre-selected data for threat analysis can be screened out more accurately.

[0176] From the above description, it can be seen that the present disclosure achieves the following technical effects: before in-depth network attack threat analysis, relatively comprehensive and accurate threat alarm pre-selected information is extracted from massive real-time network traffic from multiple dimensions, and massive invalid information is removed before analysis, thereby improving the effectiveness and completeness of the threat analysis of the pre-selected data and improving the efficiency of real-time analysis.

[0177] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0178] The present disclosure also provides a network data collection device for implementing the above method embodiment. Figure 7 As shown, the network data collection device 70 includes:

[0179] The determining unit 701 is used to determine task configuration information, wherein the task configuration information includes configuration information of at least one task, and the configuration information for each task includes basic task information, data source configuration information, detection engine configuration information, and detection rule configuration information;

[0180] The determining unit 701 is further used to determine the association relationship between the task and the data source, the association relationship between the task and the detection engine, and the association relationship between the task, the detection rule, and the detection engine according to the task configuration information;

[0181] The processing unit 702 is used to receive the real-time traffic data carrying the data source identifier in response to the task start operation, and perform the following operations for each task in the task basic information based on the task configuration information:

[0182] According to the data source identifier of the real-time traffic data, target real-time traffic data identical to the data source identifier in the data source configuration information is obtained from the real-time traffic data, and according to the association relationship between the task and the data source, the task identifier of the task is marked on the target real-time traffic data;

[0183] Distribute the detection rules in the detection rule configuration information to corresponding detection engines according to the association relationship among the tasks, the detection rules and the detection engines;

[0184] According to the association between the task and the detection engine, the target real-time traffic data carrying the task identifier is sent to the corresponding detection engine, and the detection engine performs real-time detection on the target real-time traffic data based on the distributed detection rules to obtain alarm data; wherein the alarm data is used to determine pre-selected data for threat analysis.

[0185] Optionally, when the processing unit 702 sends the target real-time traffic data carrying the task identifier to the corresponding detection engine according to the association relationship between the task and the detection engine, the processing unit 702 specifically includes:

[0186] Determining a target detection engine for real-time detection in the task according to an association relationship between the task and the detection engine;

[0187] If there are multiple target detection engines corresponding to the task, the target real-time traffic data is copied;

[0188] The copied target real-time traffic data is sent to the multiple target detection engines respectively; wherein one piece of target real-time traffic data corresponds to one target detection engine.

[0189] Optionally, the processing unit 702, when executing if there are multiple target detection engines corresponding to the task, copying the target real-time traffic data, specifically includes:

[0190] If there are multiple target detection engines corresponding to the task, determine the number of the target detection engines; wherein the detection engines include at least two of the following: an intrusion detection system IDS attack traffic detection engine, an attack injection behavior detection engine, a sample analysis detection engine, and an encryption consistency detection engine, and the detection rules are used to represent a detection feature description language that conforms to grammatical specifications;

[0191] If the target real-time traffic data is real-time traffic data carrying a data source identifier, the target real-time traffic data is copied so that the number of the copied target real-time traffic data is the same as the number of the target detection engines;

[0192] If the target real-time traffic data is real-time traffic data carrying multiple data source identifiers, then for each data source identifier, the real-time traffic data of each data source identifier is copied respectively, so that the number of real-time traffic data of each data source identifier after copying is the same as the number of the target detection engines.

[0193] Optionally, when the processing unit 702 performs real-time detection on the target real-time traffic data based on the distributed detection rules by the detection engine to obtain alarm data, the processing unit 702 specifically includes:

[0194] If the detection rule includes an extended rule type, the detection rule is uniformly converted into an XML standard syntax rule;

[0195] Among them, the rules corresponding to the extended rule type are determined based on threat intelligence information and attack indicators IOC.

[0196] Optionally, the network data collection device is further used to perform the following operations:

[0197] Send the alarm data output by each detection engine to the Kafka message processor to aggregate and cache the alarm data;

[0198] According to each of the task identifiers, obtain the alarm data corresponding to each of the tasks from the kafka message processor;

[0199] For each of the tasks, the warning data corresponding to the task is processed to determine pre-selected data for threat analysis.

[0200] Optionally, the network data collection device, when executing each of the tasks and processing the alarm data corresponding to the task to determine the pre-selected data for threat analysis, specifically includes:

[0201] For each of the tasks, deduplication and clean the alarm data corresponding to the task;

[0202] Standardize the field values ​​of heterogeneous data in the deduplicated and cleaned alarm data;

[0203] The standardized alarm data is enriched to generate the pre-selected data for threat analysis.

[0204] Optionally, the network data collection device is further used to perform the following operations:

[0205] Perform threat analysis based on the pre-selected data to obtain intelligence information, wherein the intelligence information includes threat intelligence information and attack indicators;

[0206] The detection rules are updated or expanded according to the threat intelligence information and the compromise indicators.

[0207] The specific manner in which each unit in the above device embodiment performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0208] The present disclosure also provides an electronic device, such as Figure 8 As shown, the electronic device includes one or more processors 81 and a memory 82. Figure 8 A processor 81 is taken as an example.

[0209] The controller may further include: an input device 83 and an output device 84 .

[0210] The processor 81, the memory 82, the input device 83 and the output device 84 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0211] The processor 81 may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips. The general-purpose processor may be a microprocessor or any conventional processor.

[0212] The memory 82 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the control method in the embodiment of the present disclosure. The processor 81 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 82, that is, the network data collection method of the above method embodiment is implemented.

[0213] The memory 82 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of a processing device operated by the server, etc. In addition, the memory 82 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 82 may optionally include a memory remotely arranged relative to the processor 81, and these remote memories may be connected to a network connection device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0214] The input device 83 can receive input digital or character information, and generate key signal input related to user settings and function control of the processing device of the server. The output device 84 can include display devices such as a display screen.

[0215] One or more modules are stored in the memory 82, and when executed by one or more processors 81, the execution is as follows: Figure 3 The method shown.

[0216] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute and implement the network data collection method as described above.

[0217] The embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the network data collection method as described above when executed by a processor.

[0218] Those skilled in the art can understand that the implementation of all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FM), a hard disk (HDD), or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0219] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.< / rule> < / engineer>

Claims

1. A network data collection method, characterized in that: include: Determine task configuration information, wherein the task configuration information includes configuration information of at least one task, and the configuration information for each task includes basic task information, data source configuration information, detection engine configuration information, and detection rule configuration information; Determine, according to the task configuration information, the association relationship between the task and the data source, the association relationship between the task and the detection engine, and the association relationship between the task, the detection rule, and the detection engine; In response to the task start operation, real-time traffic data carrying a data source identifier is received, and for each task in the task basic information, the following operations are performed based on the task configuration information: According to the data source identifier of the real-time traffic data, target real-time traffic data identical to the data source identifier in the data source configuration information is obtained from the real-time traffic data, and according to the association relationship between the task and the data source, the task identifier of the task is marked on the target real-time traffic data; Distribute the detection rules in the detection rule configuration information to corresponding detection engines according to the association relationship among the tasks, the detection rules and the detection engines; According to the association between the task and the detection engine, the target real-time traffic data carrying the task identifier is sent to the corresponding detection engine, and the detection engine performs real-time detection on the target real-time traffic data based on the distributed detection rules to obtain alarm data; wherein the alarm data is used to determine pre-selected data for threat analysis.

2. The method according to claim 1, characterized in that The step of sending the target real-time traffic data carrying the task identifier to the corresponding detection engine according to the association relationship between the task and the detection engine includes: Determining a target detection engine for real-time detection in the task according to an association relationship between the task and the detection engine; If there are multiple target detection engines corresponding to the task, the target real-time traffic data is copied; The copied target real-time traffic data is sent to the multiple target detection engines respectively; wherein one piece of target real-time traffic data corresponds to one target detection engine.

3. The method according to claim 2, characterized in that If there are multiple target detection engines corresponding to the task, the target real-time traffic data is copied, including: If there are multiple target detection engines corresponding to the task, determine the number of the target detection engines; wherein the detection engines include at least two of the following: an intrusion detection system IDS attack traffic detection engine, an attack injection behavior detection engine, a sample analysis detection engine, and an encryption consistency detection engine, and the detection rules are used to represent a detection feature description language that conforms to grammatical specifications; If the target real-time traffic data is real-time traffic data carrying a data source identifier, the target real-time traffic data is copied so that the number of the copied target real-time traffic data is the same as the number of the target detection engines; If the target real-time traffic data is real-time traffic data carrying multiple data source identifiers, then for each data source identifier, the real-time traffic data of each data source identifier is copied respectively, so that the number of real-time traffic data of each data source identifier after copying is the same as the number of the target detection engines.

4. The method according to claim 1, characterized in that: The detecting engine detects the target real-time traffic data in real time based on the distributed detection rules to obtain alarm data, including: If the detection rule includes an extended rule type, the detection rule is uniformly converted into an XML standard syntax rule; Among them, the rules corresponding to the extended rule type are determined based on threat intelligence information and attack indicators IOC.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Send the alarm data output by each detection engine to the Kafka message processor to aggregate and cache the alarm data; According to each of the task identifiers, obtain the alarm data corresponding to each of the tasks from the kafka message processor; For each of the tasks, the warning data corresponding to the task is processed to determine pre-selected data for threat analysis.

6. The method according to claim 5, characterized in that For each of the tasks, processing the warning data corresponding to the task to determine pre-selected data for threat analysis includes: For each of the tasks, deduplication and clean the alarm data corresponding to the task; Standardize the field values ​​of heterogeneous data in the deduplicated and cleaned alarm data; The standardized alarm data is enriched to generate the pre-selected data for threat analysis.

7. The method according to claim 6, characterized in that The method further comprises: Perform threat analysis based on the pre-selected data to obtain intelligence information, wherein the intelligence information includes threat intelligence information and attack indicators; The detection rules are updated or expanded according to the threat intelligence information and the compromise indicators.

8. A network data collection device, characterized in that: include: A determination unit, configured to determine task configuration information, wherein the task configuration information includes configuration information of at least one task, and the configuration information for each task includes basic task information, data source configuration information, detection engine configuration information, and detection rule configuration information; The determining unit is further used to determine, according to the task configuration information, an association relationship between a task and a data source, an association relationship between a task and a detection engine, and an association relationship between a task, a detection rule, and a detection engine; The processing unit is used to receive the real-time traffic data carrying the data source identifier in response to the task start operation, and perform the following operations for each task in the task basic information based on the task configuration information: According to the data source identifier of the real-time traffic data, target real-time traffic data identical to the data source identifier in the data source configuration information is obtained from the real-time traffic data, and according to the association relationship between the task and the data source, the task identifier of the task is marked on the target real-time traffic data; Distribute the detection rules in the detection rule configuration information to corresponding detection engines according to the association relationship among the tasks, the detection rules and the detection engines; According to the association between the task and the detection engine, the target real-time traffic data carrying the task identifier is sent to the corresponding detection engine, and the detection engine performs real-time detection on the target real-time traffic data based on the distributed detection rules to obtain alarm data; wherein the alarm data is used to determine pre-selected data for threat analysis.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the network data collection method described in any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor executes the network data collection method described in any one of claims 1 to 7.

Images