Task-driven massive weblog efficient screening and distributing method and device

Abstract

The invention relates to a task-driven massive weblog efficient screening and distributing method and device, and provides a multi-task efficient parallel screening and distributing innovative scheme aiming at the performance bottleneck problem existing in super-large-scale weblog data screening and distributing. The scheme focuses on intelligent screening and flexible distribution of weblog data, and aims at integrating generality calculation requirements, eliminating repeated operation, optimizing resource configuration and improving task concurrent processing capacity. In specific implementation, by decomposing and combining cross-task log screening logic and combining permanent real-time data screening and periodical offline screening tasks, refined on-demand screening, field enrichment, data classification and historical data pre-storage of data are systematically completed, and therefore the data can be classified and classified according to the actual requirements of upper-layer services. Appropriate data are dynamically and accurately distributed for different tasks, and data support is provided for applications such as network threat monitoring and flow behavior analysis.

Description

Technical Field

[0001] The present invention relates to a method and device for accurately identifying network logs in the field of network security, and in particular to a method and device for efficiently screening and distributing massive network logs driven by a task. Background Art

[0002] In order to accurately identify log information that meets the preset feature rules from network logs, the core logic of the current general solution strategy can be summarized as follows:

[0003] Use real-time computing frameworks such as Flink and Spark Streaming to filter out records that meet feature conditions from traffic logs in real time;

[0004] Based on offline computing frameworks such as Spark and MapReduce, records that meet the conditions are extracted from historical log records in batches;

[0005] Use log management systems such as Elasticsearch, Logstash, Kibana (ELK Stack for short) or Splunk to efficiently build indexes, retrieve and deeply analyze logs based on feature rules.

[0006] However, when processing huge volumes of log data, log screening and distribution tasks face the challenges of long execution cycles and high consumption of computing resources, and are difficult to adapt to the requirements of simultaneous processing of multiple tasks.

[0007] In view of the above-mentioned defects in the existing technology, the inventor has finally created the present invention which is of practical value after continuous research and design, repeated trial production and improvement. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the existing technology and provide a new type of task-driven method and device for efficient screening and distribution of massive network logs. The technical problem to be solved is to adapt it to a large-scale data environment and a technical solution for efficient screening and intelligent distribution of network logs with multi-task parallel execution, aiming to integrate shared computing needs, reduce redundant operations, optimize resource allocation, and significantly enhance the concurrent capability of task processing, so as to be more suitable for practical use.

[0009] The purpose of the present invention and the technical problem to be solved are achieved by adopting the following technical solutions. According to the present invention, 1. a task-driven method for efficiently screening and distributing massive network logs comprises the following steps:

[0010] Step 1: Configure data filtering rules

[0011] Users configure data filtering rules through the API interface or visual interface; filtering rules include filtering data types, matching rules and time range conditions;

[0012] Step 2: Time-based rule decomposition

[0013] According to the time range conditions in the filtering rules, determine whether the data filtering conditions include historical data and new data; based on time, classify into: real-time filtering rules, offline filtering rules, real-time and offline filtering rules; time-based rule decomposition includes: decomposing, deduplicating, and issuing real-time filtering rules, filtering new data from the message queue, and decomposing, deduplicating, and issuing offline filtering rules, filtering historical data from the distributed data warehouse;

[0014] Step 3: Data Distribution

[0015] Based on the data screening rules before decomposition, secondary screening is performed from the real-time screening results and offline screening results, and the result data is distributed on demand.

[0016] Furthermore, the specific steps of decomposing, deduplicating, and issuing the real-time screening rules in step 2 are:

[0017] a) Based on the matching rules, the rules are decomposed into atomic rules: equal value matching, string matching, regular matching and combination rules;

[0018] b) For each atomic rule, determine whether it is repeated based on the maintained and effective atomic rules. If it is repeated, it will not be issued repeatedly;

[0019] c) For atomic rules that are not in effect, send the rule to the common computing engine; at the same time, add a new atomic rule to the effective atomic rules.

[0020] d) The real-time screening task resident in the common computing engine dynamically queries the atomic rule list and automatically takes effect.

[0021] Furthermore, the offline screening rules described in step 2 are decomposed, deduplicated, and distributed to screen out historical data from the distributed data warehouse; the specific steps are as follows:

[0022] a) Based on the matching rules, the rules are decomposed into atomic rules: equal value matching, string matching, regular matching and combination rules;

[0023] b) For each atomic rule, determine whether it is repeated based on the maintained and effective atomic rules. If it is repeated, it will not be issued repeatedly;

[0024] c) For atomic rules that are not in effect, send the rule to the common computing engine; at the same time, add a new atomic rule to the effective atomic rules.

[0025] d) The periodic offline screening and distribution tasks in the common computing engine query the atomic rule list before each startup and automatically take effect.

[0026] According to the present invention, a device for efficiently screening and distributing a large amount of network logs using a task-driven method includes: a rule management module, a task management module, a rule storage module, a data retention module, and a commonality calculation module, wherein:

[0027] The rule management module provides full life cycle management of adding, deleting, modifying, checking, approving and status tracking of data screening rules based on both interface and visual interface;

[0028] The rule storage module provides a retention function for rule data based on a relational database; it includes task rules defined by users through interfaces and interfaces, real-time screening rules and offline screening rules formed based on time decomposition, and principle rules formed based on matching method decomposition.

[0029] The task management module provides two modes: API interface and visual interface, supports configuration of data screening tasks, and decomposes data screening tasks according to task association pairs. Different screening subtasks after decomposition are scheduled using different scheduling strategies and computing resource queues; in addition, it also provides a status monitoring function for data screening tasks, which can periodically monitor the execution status of tasks and return;

[0030] The common computing module combines the common computing logic units of different data screening tasks through real-time screening tasks, offline screening tasks, data enrichment tasks, data classification tasks, real-time distribution tasks and offline file sharing tasks in the form of resident services, so as to realize that the same data is read only once, reducing disk IO; the same screening logic is executed only once, avoiding repeated calculations, saving computing resources and improving task execution efficiency;

[0031] The data stream storage module is based on Kafka data bus and distributed data warehouse Hi ve, and caches the real-time screening, offline screening, data enrichment, and data classification result data of the common computing module in the form of real-time messages and database tables. In addition, when the bus or offline storage resources are sufficient, it also supports short-term caching of the final result data of the user's data screening task;

[0032] The rule data generated by the rule management module’s distribution, review, decomposition, and deduplication operations are actually stored and managed through the rule storage module;

[0033] The task management module configures tasks based on the rules stored in the rule storage module. The configured tasks are decomposed and scheduled and transformed into subtasks of the common computing module.

[0034] The pre-screened data formed by the common calculation module through real-time and offline data screening, enrichment and other operations are retained by the data retention module;

[0035] The common computing module will also use the data reserved by the data retention module for on-demand distribution;

[0036] The task monitoring module monitors various tasks of the common computing module.

[0037] Further, the rule management module includes a rule issuing module, which supports the user to configure the data screening rules into the device through an API interface or a visual interface;

[0038] The rule management module also includes a rule review module, which supports the administrator to review the rules issued by the user and eliminate illegal data screening rules that exceed the processing capacity;

[0039] The rule management module also includes a rule decomposition module, which supports decomposing the data screening rules issued by the user into real-time screening rules and offline screening rules according to the time range; supports decomposing the screening rules into equal value matching rules, string matching rules, regular matching rules and combination rules according to the data field matching method;

[0040] The rule management module also includes a rule deduplication module, which supports deduplication of the decomposed data screening rules to avoid sending duplicate data screening rules to the common calculation module;

[0041] When managing rules, first issue the rules, review the issued rules, and then decompose and deduplicate the rules that pass the review.

[0042] Furthermore, the task management module includes a task configuration module, which supports the user to configure the matching rules, time range, data type and other information associated with the screening task through an interface or screen;

[0043] The task management module also includes a task decomposition module, which supports decomposing the screening task into real-time screening tasks and offline screening tasks based on the decomposition of task rules; decomposing the task into new tasks or regular running tasks according to whether the atomic screening rule has been issued;

[0044] The task management module also includes an execution scheduling module, which supports filtering the computing resource queue, scheduling strategy, dependency order and other information of the task;

[0045] The task management module also includes a status monitoring module, which supports monitoring the scheduling status, execution status, and return result data size statistics of the task;

[0046] When managing tasks, first configure the task, then decompose the task into multiple subtasks, schedule the execution of each subtask, and finally monitor the status of each task.

[0047] Furthermore, the rule storage module includes an atomic rule storage module, which supports storing rule information such as equal value matching, string matching, regular matching, and combination matching in a relational database.

[0048] The rule storage module also includes a task rule storage module, which supports storing user-configured task rules and real-time screening rules and offline screening rule information formed by time decomposition in a relational database.

[0049] Furthermore, the data retention module includes an atomic rule screening result data retention module, which supports caching real-time screening result data in the form of Kafka messages; and retaining offline screening data in the form of HI VE library tables;

[0050] The data retention module also includes a task rule screening result data retention module, which supports caching real-time distribution data in the form of Kafka messages; and retaining offline distribution data in the form of HIVE database tables;

[0051] Furthermore, the common computing module includes a real-time screening module, which provides a resident Flink real-time screening task, supports dynamic reception of data screening atomic rules and takes effect immediately, and writes the screening result data back to the Kafka data bus;

[0052] The common computing module also includes an offline screening module, which provides periodic startup of batch Spark offline screening tasks, obtains the latest full data screening rules before each startup, and writes the screening result data back to the Kafka data bus;

[0053] The common computing module also includes a data enrichment module, which provides a resident memory Flink real-time enrichment task, supports enriching the result data of real-time screening and offline screening according to the agreed data enrichment rules, and writes the enriched result data back to the Kafka data bus;

[0054] The common computing module also includes a data classification module, which provides a real-time classification task of Flink in memory, supports the classification of enriched result data according to the agreed data classification rules, and writes the classification result data back to the Kafka data bus. Different levels of data have different retention strategies corresponding to the data retention module, and high-level data has a longer retention time;

[0055] When performing common calculations, first perform real-time and offline screening as needed, enrich the screening result data, and classify the data. Finally, distribute the enriched result data in real time and offline as needed.

[0056] Furthermore, the common computing module includes a real-time distribution module, which provides a resident memory Flink real-time distribution task, supports filtering the final result data again from the cross-task data filtering result data according to the data filtering rules set by the user, and writes it back to the specified Kafka data bus topic ic;

[0057] The common computing module also includes an offline distribution module, which provides offline distribution tasks resident in memory, supports re-screening the final result data from the cross-task data screening result data according to the data screening rules set by the user, and distributes it to the specified data directory in the form of files.

[0058] Compared with the prior art, the present invention has obvious advantages and beneficial effects. It has at least the following advantages:

[0059] 1. Through the rule review module, invalid data screening rules that exceed the cluster processing capacity are prevented from entering the processing flow, occupying a large amount of computing resources and congesting other data screening tasks;

[0060] 2. Based on the multi-level data screening rule mechanism, it can not only meet the user's needs for simplified rule configuration, but also achieve the maximum degree of rule merging and reduce repeated data screening;

[0061] 3. The data screening rules can be flexibly configured, supporting multiple screening rules such as equal value, character, regular expression, combination, etc.;

[0062] 4. Supports filtering out data that meets feature rules based on IP, domain name, MD5, email, etc., to meet business needs such as unknown threat discovery and key target security monitoring;

[0063] 5. Support multiple tasks and multiple screening rules to take effect at the same time, avoiding repeated calculations and saving computing resources;

[0064] 6. Supports both offline and real-time data screening methods, which can track new data in real time and trace back historical data.

[0065] 7. Through the pre-screening and retention mechanism, data distribution is converted from being based on large-scale original data to being based on pre-reserved data, which reduces the scale of data reading and improves the efficiency of data screening and distribution;

[0066] 8. By decomposing rules and removing duplicate atomic rules, repeated screening can be avoided and resource utilization can be improved.

[0067] The specific implementation of the present invention is given in detail by the following examples and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a device framework diagram of the present invention.

[0069] Figure 2 It is a flow chart of the method of the present invention DETAILED DESCRIPTION

[0070] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a task-driven massive network based on the present invention in combination with the accompanying drawings and preferred embodiments.

[0071] The specific implementation, structure, features and efficacy of the network log efficient screening and distribution method and device are described in detail below. The examples are only used to explain the present invention and are not used to limit the scope of the present invention.

[0072] Figure 1 The figure shows a task-driven method and device framework for efficiently screening and distributing massive network logs according to the present invention, which mainly includes a rule management module, a task management module, a rule storage module, and a data retention module. The functional design of each module is as follows:

[0073] 1. The rule management module provides full life cycle management of adding, deleting, modifying, checking, approving and status tracking of data screening rules based on interfaces and visual interfaces;

[0074] 2. The rule storage module provides the function of retaining rule data based on the relational database; including task rules defined by users through interfaces and interfaces, real-time screening rules and offline screening rules formed based on time decomposition, and principle rules formed based on matching method decomposition.

[0075] 3. The task management module provides two methods: API interface and visual interface. It supports the configuration of data screening tasks and decomposes the data screening tasks according to the task association. Different screening subtasks after decomposition are scheduled using different scheduling strategies and computing resource queues. In addition, it also provides the status monitoring function of data screening tasks, which can periodically monitor the execution status of tasks and return;

[0076] 4. The common computing module combines the common computing logic units of different data screening tasks through real-time screening tasks, offline screening tasks, data enrichment tasks, data classification tasks, real-time distribution tasks and offline file sharing tasks in the form of resident services, so as to realize that the same data is read only once, reducing disk IO; the same screening logic is executed only once, avoiding repeated calculations, saving computing resources and improving task execution efficiency;

[0077] 5. The data stream storage module is based on the Kafka data bus and the distributed data warehouse Hive, and caches the real-time screening, offline screening, data enrichment, and data classification result data of the common computing module in the form of real-time messages and database tables. In addition, when the bus or offline storage resources are sufficient, it also supports short-term caching of the final result data of the user's data screening task;

[0078] The rule data generated by the rule management module’s distribution, review, decomposition, and deduplication operations are actually stored and managed through the rule storage module;

[0079] The task management module configures tasks based on the rules stored in the rule storage module. The configured tasks are decomposed and scheduled and transformed into subtasks of the common computing module.

[0080] The pre-screened data formed by the common calculation module through real-time and offline data screening, enrichment and other operations are retained by the data retention module;

[0081] The common computing module will also use the data reserved by the data retention module for on-demand distribution;

[0082] The task monitoring module monitors various tasks of the common computing module.

[0083] Specifically, the rule management module includes a rule issuing module, which supports users to configure data screening rules into the device through an API interface or a visual interface;

[0084] The rule management module also includes a rule review module, which supports administrators to review the rules issued by users and eliminate illegal data screening rules that exceed processing capabilities;

[0085] The rule management module also includes a rule decomposition module, which supports decomposing the data screening rules issued by users into real-time screening rules and offline screening rules according to the time range; supports decomposing the screening rules into equal value matching rules, string matching rules, regular matching rules and combination rules according to the data field matching method;

[0086] The rule management module also includes a rule deduplication module, which supports deduplication of decomposed data screening rules to avoid sending duplicate data screening rules to the common calculation module;

[0087] When managing rules, first issue the rules, review the issued rules, and then decompose and deduplicate the rules that pass the review.

[0088] Specifically, the task management module includes a task configuration module, which supports users to configure the matching rules, time range, data type and other information associated with the screening task through an interface or screen;

[0089] The task management module also includes a task decomposition module, which supports decomposing the task rules into real-time screening rules and offline screening rules, and decomposing the screening task into real-time screening tasks and offline screening tasks; according to whether the atomic screening rules have been issued, the task is decomposed into new tasks or regular running tasks;

[0090] The task management module also includes an execution scheduling module, which supports filtering the computing resource queue, scheduling strategy, dependency order and other information of the task;

[0091] The task management module also includes a status monitoring module, which supports monitoring the scheduling status, execution status, and return result data size statistics of tasks;

[0092] When managing tasks, first configure the task, then decompose the task into multiple subtasks, schedule the execution of each subtask, and finally monitor the status of each task.

[0093] Specifically, the rule storage module includes an atomic rule storage module, which supports storing rule information such as equal value matching, string matching, regular matching, and combination matching in a relational database.

[0094] The rule storage module also includes a task rule storage module, which supports storing user-configured task rules and real-time screening rules and offline screening rule information formed by time decomposition in a relational database.

[0095] Specifically, the data retention module includes an atomic rule screening result data retention module, which supports caching real-time screening result data in the form of Kafka messages and retaining offline screening data in the form of HIVE database tables;

[0096] The data retention module also includes a task rule screening result data retention module, which supports caching real-time distribution data in the form of Kafka messages and retaining offline distribution data in the form of HIVE database tables;

[0097] Specifically, the common computing module includes a real-time screening module, which provides a resident Flink real-time screening task, supports dynamic reception of data screening atomic rules and takes effect immediately, and writes the screening result data back to the Kafka data bus;

[0098] The common computing module also includes an offline screening module, which provides periodic startup of batch Spark offline screening tasks, obtains the latest full data screening rules before each startup, and writes the screening result data back to the Kafka data bus;

[0099] The common computing module also includes a data enrichment module, which provides a resident memory Flink real-time enrichment task, supports enriching the result data of real-time screening and offline screening according to the agreed data enrichment rules, and writes the enriched result data back to the Kafka data bus;

[0100] The common computing module also includes a data classification module, which provides a real-time classification task of Flink in memory, supports the classification of enriched result data according to the agreed data classification rules, and writes the classification result data back to the Kafka data bus. Different levels of data have different retention strategies corresponding to the data retention module, and high-level data has a longer retention time;

[0101] When performing common calculations, first perform real-time and offline screening as needed, enrich the screening result data, and classify the data. Finally, distribute the enriched result data in real time and offline as needed.

[0102] Specifically, the common computing module includes a real-time distribution module, which provides a resident Flink real-time distribution task, supports filtering the final result data from the cross-task data filtering result data according to the data filtering rules set by the user, and writes it back to the specified Kafka data bus topic;

[0103] The common computing module also includes an offline distribution module, which provides offline distribution tasks that reside in memory, supports filtering the final result data from the cross-task data filtering result data according to the data filtering rules set by the user, and distributes it to the specified data directory in the form of files.

[0104] like Figure 2 As shown, a flowchart of a task-driven method for efficiently screening and distributing massive network logs designed by the present invention is constructed, and the specific construction steps are as follows:

[0105] Step 1: Configure data filtering rules. Users configure data filtering rules through the API interface or visual interface. The filtering rules include filtering data types, matching rules, time range and other conditions;

[0106] Step 2: Time-based rule decomposition. According to the time range conditions in the filtering rules, determine whether the data filtering conditions include historical data and new data; based on time, classify into: real-time filtering rules, offline filtering rules, real-time and offline filtering rules;

[0107] Decompose, deduplicate, and distribute real-time screening rules, and filter out new data from the message queue;

[0108] a) Based on the matching rules, the rules are decomposed into atomic rules: equal value matching, string matching, regular matching and combination rules;

[0109] b) For each atomic rule, determine whether it is repeated based on the maintained and effective atomic rules. If it is repeated, it will not be issued repeatedly;

[0110] c) For atomic rules that are not in effect, send the rule to the common computing engine; at the same time, add a new atomic rule to the effective atomic rules.

[0111] d) The real-time screening task resident in the common computing engine dynamically queries the atomic rule list and automatically takes effect.

[0112] Decompose, deduplicate, and distribute offline screening rules to filter out historical data from distributed data warehouses;

[0113] a) Based on the matching rules, the rules are decomposed into atomic rules: equal value matching, string matching, regular matching and combination rules;

[0114] b) For each atomic rule, determine whether it is repeated based on the maintained and effective atomic rules. If it is repeated, it will not be issued repeatedly;

[0115] c) For atomic rules that are not in effect, send the rule to the common computing engine; at the same time, add a new atomic rule to the effective atomic rules.

[0116] d) The periodic offline screening and distribution tasks in the common computing engine query the atomic rule list before each startup and automatically take effect.

[0117] Step 3: Data distribution: Based on the data screening rules before decomposition, secondary screening is performed from the real-time screening results and offline screening results, and the result data is distributed on demand.

[0118] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A task-driven method for efficiently screening and distributing massive network logs, characterized by: The steps include: Step 1: Configure data filtering rules Users configure data filtering rules through the API interface or visual interface; filtering rules include filtering data types, matching rules and time range conditions; Step 2: Time-based rule decomposition According to the time range conditions in the filtering rules, determine whether the data filtering conditions include historical data and new data; Based on time, the rules are classified into: real-time filtering rules, offline filtering rules, and real-time and offline filtering rules; the time-based rule decomposition includes: decomposing, deduplicating, and issuing real-time filtering rules, filtering out new data from the message queue, and decomposing, deduplicating, and issuing offline filtering rules, filtering out historical data from the distributed data warehouse; Step 3: Data Distribution Based on the data screening rules before decomposition, secondary screening is performed from the real-time screening results and offline screening results, and the result data is distributed on demand.

2. According to claim 1, a task-driven method for efficiently screening and distributing massive network logs is characterized by: The specific steps of decomposing, deduplicating, and issuing the real-time screening rules in step 2 are: a) Based on the matching rules, the rules are decomposed into atomic rules: equal value matching, string matching, regular matching and combination rules; b) For each atomic rule, determine whether it is repeated based on the maintained and effective atomic rules. If it is repeated, it will not be issued repeatedly; c) For atomic rules that are not in effect, send the rule to the common computing engine; at the same time, add a new atomic rule to the effective atomic rules. d) The real-time screening task resident in the common computing engine dynamically queries the atomic rule list and automatically takes effect.

3. According to claim 1, a task-driven method for efficiently screening and distributing massive network logs is characterized by: Decompose, deduplicate, and distribute the offline screening rules described in step 2 to filter out historical data from the distributed data warehouse; the specific steps are as follows: a) Based on the matching rules, the rules are decomposed into atomic rules: equal value matching, string matching, regular matching and combination rules; b) For each atomic rule, determine whether it is repeated based on the maintained and effective atomic rules. If it is repeated, it will not be issued repeatedly; c) For atomic rules that are not in effect, send the rule to the common computing engine; at the same time, add a new atomic rule to the effective atomic rules. HDP240180CNPage 2 / 4 d) The periodic offline screening and distribution tasks in the common computing engine query the atomic rule list before each startup and automatically take effect.

4. A device for efficiently screening and distributing a large amount of network logs using a task-driven method according to claims 1-3, characterized in that: include: Rule management module, task management module, rule storage module, data retention module, common calculation module, including: The rule management module provides full life cycle management of adding, deleting, modifying, checking, approving and status tracking of data screening rules based on both interface and visual interface; The rule storage module provides a retention function for rule data based on a relational database; it includes task rules defined by users through interfaces and interfaces, real-time screening rules and offline screening rules formed based on time decomposition, and principle rules formed based on matching method decomposition. The task management module provides two modes: API interface and visual interface, supports configuration of data screening tasks, and decomposes data screening tasks according to task association pairs. Different screening subtasks after decomposition are scheduled using different scheduling strategies and computing resource queues. In addition, it also provides a status monitoring function for data screening tasks, which can periodically monitor the execution status of tasks and return; The common computing module combines the common computing logic units of different data screening tasks through real-time screening tasks, offline screening tasks, data enrichment tasks, data classification tasks, real-time distribution tasks and offline file sharing tasks in the form of resident services, so as to realize that the same data is read only once, reducing disk IO; the same screening logic is executed only once, avoiding repeated calculations, saving computing resources and improving task execution efficiency; The data stream storage module is based on the Kafka data bus and the distributed data warehouse Hive, and caches the real-time screening, offline screening, data enrichment, and data classification result data of the common computing module in the form of real-time messages and database tables. In addition, when the bus or offline storage resources are sufficient, it also supports short-term caching of the final result data of the user's data screening task; The rule data generated by the rule management module’s distribution, review, decomposition, and deduplication operations are actually stored and managed through the rule storage module; The task management module configures tasks based on the rules stored in the rule storage module. The configured tasks are decomposed and scheduled and transformed into subtasks of the common computing module. The pre-screened data formed by the common calculation module through real-time and offline data screening, enrichment and other operations are retained by the data retention module; The common computing module will also use the data reserved by the data retention module for on-demand distribution; The task monitoring module monitors various tasks of the common computing module.

5. The device of the task-driven method for efficiently screening and distributing massive network logs according to claim 4 is characterized in that: The rule management module includes a rule issuing module, which supports users to configure data screening rules into the device through an API interface or a visual interface; The rule management module also includes a rule review module, which supports the administrator to review the rules issued by the user. Then conduct an audit to eliminate illegal data screening rules that exceed processing capacity; The rule management module also includes a rule decomposition module, which supports decomposing the data screening rules issued by the user into real-time screening rules and offline screening rules according to the time range; supports decomposing the screening rules into equal value matching rules, string matching rules, regular matching rules and combination rules according to the data field matching method; The rule management module also includes a rule deduplication module, which supports deduplication of the decomposed data screening rules to avoid sending duplicate data screening rules to the common calculation module; When managing rules, first issue the rules, review the issued rules, and then decompose and deduplicate the rules that pass the review.

6. The device of the task-driven method for efficiently screening and distributing massive network logs according to claim 4 is characterized in that: The task management module includes a task configuration module, which supports users to configure the matching rules, time range, data type and other information associated with the screening task through an interface or screen; The task management module also includes a task decomposition module, which supports decomposing the screening task into a real-time screening task and an offline screening task based on the decomposition of the task rule to obtain the real-time screening rule and the offline screening rule; Decompose the task into new tasks or regular running tasks according to whether the atomic screening rules have been issued; The task management module also includes an execution scheduling module, which supports filtering the computing resource queue, scheduling strategy, dependency order and other information of the task; The task management module also includes a status monitoring module, which supports monitoring the scheduling status, execution status, and return result data size statistics of the task; When managing tasks, first configure the task, then decompose the task into multiple subtasks, schedule the execution of each subtask, and finally monitor the status of each task.

7. The device of the task-driven method for efficiently screening and distributing massive network logs according to claim 4 is characterized in that: The rule storage module includes an atomic rule storage module, which supports storing rule information such as equal value matching, string matching, regular matching, and combination matching in a relational database. The rule storage module also includes a task rule storage module, which supports storing user-configured task rules and real-time screening rules and offline screening rule information formed by time decomposition in a relational database.

8. The device of the task-driven method for efficiently screening and distributing massive network logs according to claim 4 is characterized in that: The data retention module includes an atomic rule screening result data retention module, which supports caching real-time screening result data in the form of Kafka messages; and retaining offline screening data in the form of HIVE database tables; The data retention module also includes a task rule screening result data retention module, which supports caching real-time distribution data in the form of Kafka messages and retaining offline distribution data in the form of HIVE database tables.

9. The device of the task-driven method for efficiently screening and distributing massive network logs according to claim 4 is characterized in that: The common computing module includes a real-time screening module, which provides a resident Flink real-time screening task, supports dynamic reception of data screening atomic rules and takes effect immediately, and writes the screening result data back to the Kafka data bus; The common computing module also includes an offline screening module, which provides periodic startup of batch Spark offline screening tasks, obtains the latest full data screening rules before each startup, and writes the screening result data back to the Kafka data bus; The common computing module also includes a data enrichment module, which provides a resident Flink real-time enrichment task, supports enriching the result data of real-time screening and offline screening according to the agreed data enrichment rules, and writes the enriched result data back to the Kafka data bus; The common computing module also includes a data classification module, which provides a resident Flink real-time classification task, supports classification of enriched result data according to the agreed data classification rules, and writes the classification result data back to the Kafka data bus. Different levels of data have different retention strategies corresponding to the data retention module, and high-level data has a longer retention time; When performing common calculations, first perform real-time and offline screening as needed, enrich the screening result data, and classify the data. Finally, distribute the enriched result data in real time and offline as needed.

10. The device of the task-driven method for efficiently screening and distributing massive network logs according to claim 4 is characterized in that: The common computing module includes a real-time distribution module, which provides a resident Flink real-time distribution task, supports filtering the final result data from the cross-task data filtering result data again according to the data filtering rules set by the user, and writes it back to the specified Kafka data bus topic; The common computing module also includes an offline distribution module, which provides offline distribution tasks resident in memory, supports re-screening the final result data from the cross-task data screening result data according to the data screening rules set by the user, and distributes it to the specified data directory in the form of files.

Images