A network data analysis method and system

Abstract

The present invention provides a network data analysis method and system, and relates to the technical field of network data processing. The present invention realizes comprehensive, efficient, flexible and targeted analysis and management of network data by combining multi-level dynamic acquisition frequency, real-time monitoring and level judgment, priority division, real-time warning and execution, and emergency response process; through multi-level dynamic acquisition frequency and priority division, resources can be allocated more reasonably, analysis efficiency and accuracy can be improved, and when security threats are discovered, responses can be made quickly and risks can be controlled in time, thereby optimizing resource allocation, significantly improving the security and stability of the network, facilitating continuous adaptation to changes in the network environment, and ensuring long-term and effective security protection.

Description

Technical Field

[0001] The present invention relates to the technical field of network data processing, and in particular to a network data analysis method and system. Background Art

[0002] In today's highly information-based society, networks have become the fundamental infrastructure for the operation of various industries, carrying massive amounts of data flows and complex interactive operations. However, with the continuous expansion of network scale and the increasing diversification of applications, network data analysis and management face unprecedented challenges, including the following problems:

[0003] With the popularization of technologies such as the Internet of Things, cloud computing, and big data, the amount of data in the network is showing an explosive growth trend. However, traditional data collection and analysis methods are single and can no longer meet the needs of analyzing different data volumes, resulting in low data processing efficiency.

[0004] When faced with massive amounts of data and complex threats, how to reasonably allocate and utilize limited resources has become a key issue. Traditional data analysis and security protection methods often lack intelligent resource scheduling mechanisms, resulting in inefficient resource utilization and an inability to meet the needs of efficient, real-time data processing and security protection.

[0005] Therefore, it is necessary to provide a network data analysis method and system to solve the above technical problems. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides a network data analysis method and system for solving the problems of traditional data collection and analysis methods being single in approach, low in data processing efficiency, and low in resource utilization efficiency.

[0007] The present invention provides a method for analyzing network data, which includes the following steps:

[0008] Based on the total amount of network data and the importance of the data during the regular collection period, the data volume level standards are set, including low, medium and high levels;

[0009] Based on the level standard of the total amount of data, set the multi-level dynamic acquisition frequency trigger logic for each data type;

[0010] Regularly acquire network data in real time through sensors and data collection points and conduct real-time monitoring to determine the level standard for regular real-time acquisition of network data;

[0011] According to the grade standard of real-time acquisition of network data, the characteristics of each data type are collected through multi-level dynamic collection frequency trigger logic;

[0012] Based on the preset security threshold segments and the characteristics of each data type collected, each data type is prioritized and security analysis is performed on them in sequence to obtain security analysis data results;

[0013] Based on the security analysis data results, corresponding warning measures are triggered for real-time execution.

[0014] Preferably, the specific steps of setting the level standard of the total amount of data based on the total amount of network data and the importance of the data within the periodic collection time are:

[0015] Regularly collect all relevant network data types within a time period, including but not limited to traffic data, user behavior data, and system logs;

[0016] Multiple level standards are set based on the system resource usage and processing complexity of all relevant data types in the network data within a time period, including low, medium and high levels.

[0017] Preferably, the specific steps of setting the multi-level dynamic acquisition frequency trigger logic for each data type based on the level standard of the total amount of data are:

[0018] Based on historical data statistics, for the total amount of data at the three levels of low, medium and high standards, a corresponding dynamic collection frequency is set for each data type within each level standard;

[0019] A multi-level dynamic acquisition frequency trigger logic is constructed based on the set dynamic acquisition frequency. Specifically, when the total amount of data reaches different level standards, the acquisition frequency of each data type in the total amount of data of the corresponding level standard is automatically adjusted.

[0020] Preferably, the specific steps of regularly acquiring network data in real time and performing real-time monitoring through sensors and data collection points and determining the level standard of regularly acquiring network data in real time are:

[0021] Use sensors and data collection points to regularly obtain network data in real time;

[0022] Monitor and clean the real-time acquired network data in real time, remove duplicate, erroneous or invalid data, and obtain pre-processed network data;

[0023] The current total amount of pre-processed network data is calculated using real-time analysis technology, and the calculated current total amount of pre-processed network data is compared with the preset grade standard to determine the grade standard of the total amount of data to which it belongs.

[0024] Preferably, the specific steps of acquiring the characteristics of each data type according to the grade standard of real-time acquisition of network data through multi-level dynamic acquisition frequency triggering logic are:

[0025] According to the level standard of real-time acquisition of network data, the corresponding acquisition frequency is assigned to each data type through multi-level dynamic acquisition frequency trigger logic;

[0026] Based on allocating corresponding collection frequencies to each data type, features of each data type are collected respectively, and feature values ​​of each data type are obtained.

[0027] Preferably, the specific steps of prioritizing each data type based on the preset security threshold segments and the characteristics of each data type collected, and performing security analysis on them in sequence to obtain the security analysis data results are:

[0028] Set multiple corresponding security threshold segments for each data type, and define corresponding priorities for the multiple security threshold segments;

[0029] Determine the security threshold segment to which the characteristic values ​​of each data type belong, and prioritize each data type based on the security threshold segment;

[0030] Based on the divided priorities, security analysis is performed on the characteristics of each data type in descending order.

[0031] Preferably, the specific steps of triggering corresponding warning measures for real-time execution based on the security analysis data results are:

[0032] The risk level of the security analysis data results is divided into dangerous, urgent and routine risk levels according to the preset risk level threshold range;

[0033] Based on the risk levels, emergency response measures corresponding to the danger level and emergency level are automatically activated;

[0034] Monitor emergency response measures until the risk level of the safety analysis data results is re-assessed as normal.

[0035] A network data analysis system, comprising:

[0036] The level definition module is used to set the level standards of the total amount of data based on the total amount and importance of network data within the regular collection time, including low, medium and high levels;

[0037] Dynamic definition module, used to set multi-level dynamic acquisition frequency trigger logic for each data type based on the level standard of the total amount of data;

[0038] The data monitoring module is used to regularly acquire and monitor network data in real time through sensors and data collection points, and to determine the level standard of regularly acquiring network data in real time;

[0039] Dynamic collection module, used to collect characteristics of various data types based on the level standard of real-time network data acquisition through multi-level dynamic collection frequency trigger logic;

[0040] The priority classification module is used to prioritize each data type based on the preset security threshold segments and the characteristics of each collected data type, and perform security analysis on them in sequence to obtain security analysis data results;

[0041] The warning control module is used to trigger corresponding warning measures for real-time execution based on the security analysis data results.

[0042] Compared with related technologies, the network data analysis method and system provided by the present invention have the following beneficial effects:

[0043] The present invention realizes comprehensive, efficient, flexible and targeted analysis and management of network data by combining multi-level dynamic collection frequency, real-time monitoring and level judgment, priority division, real-time warning and execution, and emergency response process; through multi-level dynamic collection frequency and priority division, resources can be allocated more reasonably, analysis efficiency and accuracy can be improved, and when security threats are discovered, a rapid response can be made and risks can be controlled in time, thereby optimizing resource allocation, significantly improving the security and stability of the network, facilitating continuous adaptation to changes in the network environment, and ensuring long-term and effective security protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a flow chart of a network data analysis method provided by the present invention;

[0045] Figure 2 This is a system block diagram of a network data analysis system provided by the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] Example 1

[0048] like Figure 1 As shown, a method for analyzing network data includes the following steps:

[0049] S100, based on the total amount of network data and the importance of the data within the periodic collection time, set a level standard for the total amount of data, specifically including low, medium and high levels;

[0050] In the specific implementation process, step S100 specifically includes the following steps:

[0051] S101. Regularly collect all relevant network data types within a time period, including but not limited to traffic data, user behavior data, and system logs;

[0052] Specifically, set the collection cycle and preset a specific time period, and use the data collector, specifically any one of Flume or Logstash, to collect data from different data sources in the network, including but not limited to routers, switches, servers, databases, and applications; and for data sources that support API access, use the existing API client library to regularly call the API interface to obtain all required relevant data types of network data, including but not limited to traffic data, user behavior data, and system logs.

[0053] For example, in this embodiment, 1:00 a.m. to 2:00 a.m. every day can be set as a collection time period. By using Flume as a data collector, it is configured to connect to routers, switches, and key servers in the network; Flume will regularly capture traffic data, system logs, and user behavior data from these devices.

[0054] S102. Setting multiple level standards based on the system resource usage and processing complexity of all relevant data types and network data within a time period, specifically including low, medium and high levels.

[0055] Specifically, the system resource usage of each data type of network data, such as CPU, memory, disk space, etc., during the collection, storage and processing process is evaluated. Based on historical data, the processing difficulty and complexity of each data type in subsequent analysis are estimated, and the time required for calculation is estimated. The estimated processing difficulty and complexity of each data type in subsequent analysis and the system resource usage are comprehensively considered, and a comprehensive value is determined by multiplication. Based on the comprehensive value, multiple level standards of network data are set, where the level standards specifically include low, medium and high levels.

[0056] S200, based on the level standard of the total amount of data, set the multi-level dynamic acquisition frequency trigger logic for each data type;

[0057] In the specific implementation process, step S200 specifically includes the following steps:

[0058] S201. Based on historical data statistics, for the total amount of data of the three level standards of low, medium, and high, set a corresponding dynamic collection frequency for each data type within each level standard;

[0059] Specifically, use Python (Pandas library) and R language data analysis tools to perform statistical analysis on historical data, calculate the average data volume and peak data volume indicators of each data type within each level standard, and combine the carrying capacity of system resources to set an initial dynamic collection frequency for each data type within each level standard.

[0060] For example, a dynamic collection frequency is set for the "non-critical application log" data type in the low-level standard. Through historical data analysis, the average data volume of this data type is calculated to be 1GB per day, the peak data volume is 3GB per day, and the data growth trend is relatively stable. Combined with the carrying capacity of system resources, an initial dynamic collection frequency can be set for this data type to be once per hour.

[0061] S202. Construct a multi-level dynamic acquisition frequency trigger logic based on the set dynamic acquisition frequency, wherein the multi-level dynamic acquisition frequency trigger logic is specifically to automatically adjust the acquisition frequency of each data type in the total amount of data corresponding to the level standard when the total amount of data reaches different level standards.

[0062] Specifically, monitoring items and alarm thresholds for each data type are set in the monitoring tool to correspond to the dynamic collection frequency set in step S201, and the Zabbix monitoring tool is used to monitor the changes in the data volume of each data type in real time, and a multi-level dynamic collection frequency trigger logic of an automated script is written. Specifically, when the total amount of data reaches different level standards, the collection frequency of each data type in the total amount of data corresponding to the level standard is automatically adjusted.

[0063] S300, regularly acquiring network data in real time through sensors and data collection points and performing real-time monitoring, and determining the level standard of regularly acquiring network data in real time;

[0064] In the specific implementation process, step S300 specifically includes the following steps:

[0065] S301, using sensors and data collection points to regularly obtain network data in real time;

[0066] Specifically, real-time data is captured through sensors and data collection points at key nodes of the network. Sensors include but are not limited to temperature sensors, pressure sensors, and flow sensors. At the same time, data collection points capture network data such as network traffic, bandwidth usage, user behavior, etc. from network devices, and finally obtain network data in real time on a regular basis.

[0067] S302: Monitor and clean the real-time acquired network data in real time to remove duplicate, erroneous, or invalid data to obtain pre-processed network data;

[0068] Use the Grafana tool to monitor and clean real-time data streams, remove duplicate, erroneous, or invalid data processing, and finally obtain pre-processed network data.

[0069] S303: Calculate the current total amount of pre-processed network data using real-time analysis technology, and compare the calculated current total amount of pre-processed network data with a preset grade standard to determine the grade standard of the total amount of data to which it belongs.

[0070] S400, according to the level standard of real-time acquisition of network data, through multi-level dynamic acquisition frequency trigger logic, collect characteristics of each data type;

[0071] In the specific implementation process, step S400 specifically includes the following steps:

[0072] S401, according to the level standard of real-time acquisition of network data, assign corresponding acquisition frequencies to each data type through multi-level dynamic acquisition frequency trigger logic;

[0073] Specifically, according to the level standard for real-time acquisition of network data, the preset multi-level dynamic collection frequency trigger logic is immediately used to assign a corresponding collection frequency to each data type. For example, if a data type is classified as "advanced", its collection frequency will be immediately adjusted to once per second; if another type of data is evaluated as "intermediate", it will be collected once every ten minutes.

[0074] S402 : Based on the corresponding collection frequency assigned to each data type, collect features for each data type respectively, and obtain feature values ​​for each data type.

[0075] Specifically, according to the corresponding collection frequency assigned to each data type, data is collected from the network data obtained in real time, and the characteristics of each data type are extracted to calculate the corresponding characteristic values.

[0076] S500: Prioritize each data type based on the preset security threshold segments and the characteristics of each collected data type, and perform security analysis on them in order to obtain security analysis data results;

[0077] In the specific implementation process, step S500 specifically includes the following steps:

[0078] S501, setting a plurality of corresponding security threshold segments for each data type, and defining corresponding priorities for the plurality of security threshold segments;

[0079] Specifically, for each data type, multiple security threshold segments are set according to the industry standards of its historical data. In this embodiment, for the data type of network traffic, four segments can be set: "normal", "mild abnormality", "moderate abnormality" and "serious abnormality". Each segment corresponds to a different traffic threshold range, and a corresponding priority is assigned to each security threshold segment.

[0080] For example, in network traffic data, the "severe anomaly" segment (such as exceeding 5 times the normal traffic peak) is assigned the highest priority 3; the "moderate anomaly" segment (such as exceeding 2-5 times the normal traffic peak) is assigned the second highest priority 2; the "mild anomaly" segment (such as exceeding the normal traffic peak but not exceeding 2 times) is assigned priority 1; and the "normal" segment defaults to the lowest priority 0.

[0081] S502: Determine the security threshold segment to which the characteristic values ​​of each data type belong, and prioritize each data type based on the security threshold segment;

[0082] Specifically, for each data type, its key features are extracted. In this example, the peak value, mean value, standard deviation, etc. of the network traffic are extracted and compared with the security threshold segment set in step S501 to determine which segment these feature values ​​belong to. Based on the security threshold segment to which the feature values ​​belong, a corresponding priority is assigned to each data type.

[0083] For example, if the network traffic data in a certain time period shows that its peak value exceeds the set "serious abnormality" threshold, the current status of the data type is classified as the highest priority 3;

[0084] If the number of login failures in user behavior data exceeds the threshold for "mild anomaly" but does not reach the threshold for "moderate anomaly", the data type is classified as priority 1.

[0085] S503: Based on the divided priorities, perform security analysis on the features of each data type in descending order.

[0086] Specifically, according to the priority divided in step S502, all data types are sorted in order of priority from high to low, and security analysis is performed one by one starting from the data type with the highest priority, including checking detailed data records, analyzing abnormal patterns, identifying potential threats, etc. It should be noted that the analysis tools used in the security analysis process are priority technologies and will not be described in detail.

[0087] S600: Based on the security analysis data results, trigger corresponding warning measures for real-time execution.

[0088] In the specific implementation process, step S600 specifically includes the following steps:

[0089] S601. Classify the risk level of the security analysis data results according to a preset risk level threshold range, wherein the risk level specifically includes dangerous, urgent, and routine;

[0090] Specifically, clarify the classification of risk levels, set specific threshold ranges for each risk level based on industry standards, compare the data results of security analysis, such as anomaly detection and threat identification, with the preset threshold ranges, and determine the risk level of each result.

[0091] For example, if an abnormal pattern detected in network traffic highly matches a known DDoS attack pattern, and the traffic level far exceeds the preset "emergency" level threshold, the risk level of the data result is classified as "emergency".

[0092] S602: Automatically activate emergency response measures corresponding to the danger level and emergency level based on the classified risk levels;

[0093] Specifically, an emergency response strategy library is established, which contains preset response measures for different risk levels (especially dangerous and emergency levels). Once the risk level of the safety analysis data results is classified as dangerous or emergency, the corresponding response measures are automatically retrieved and activated from the emergency response strategy library.

[0094] For example, when the risk level of network traffic data is determined to be "urgent", the system automatically triggers DDoS mitigation strategies, including increasing bandwidth, enabling traffic cleaning services, etc.

[0095] S603. Monitor emergency response measures until the risk level of the safety analysis data results is re-determined to be at a normal level.

[0096] Specifically, during the execution of emergency response measures, continuously monitor the status of the affected systems, networks or applications, as well as the latest changes in security analysis data results; regularly evaluate the effectiveness of emergency response measures and analyze new security data; based on the new security analysis data results, re-evaluate the risk level. If the risk level has dropped to the normal level, it indicates that the emergency response measures have been successfully implemented, and monitoring can be terminated and normal business operations can be resumed.

[0097] Example 2

[0098] like Figure 2 As shown, a network data analysis system applied to a network data analysis method includes:

[0099] The level definition module is used to set the level standards of the total amount of data based on the total amount and importance of network data within the regular collection time, including low, medium and high levels;

[0100] Dynamic definition module, used to set multi-level dynamic acquisition frequency trigger logic for each data type based on the level standard of the total amount of data;

[0101] The data monitoring module is used to regularly acquire and monitor network data in real time through sensors and data collection points, and to determine the level standard of regularly acquiring network data in real time;

[0102] Dynamic collection module, used to collect characteristics of various data types based on the level standard of real-time network data acquisition through multi-level dynamic collection frequency trigger logic;

[0103] The priority classification module is used to prioritize each data type based on the preset security threshold segments and the characteristics of each collected data type, and perform security analysis on them in sequence to obtain security analysis data results;

[0104] The warning control module is used to trigger corresponding warning measures for real-time execution based on the security analysis data results.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0107] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

Claims

1. A method for analyzing network data, characterized in that: The analytical method comprises the following steps: Based on the total amount of network data and the importance of data during the periodic collection period, the level standards of total amount of data are set, including low, medium and high levels; Based on the level standard of the total amount of data, set the multi-level dynamic acquisition frequency trigger logic for each data type. The specific steps are as follows: Based on the statistics of historical data, for the total amount of data of the three level standards of low, medium and high, the corresponding dynamic collection frequency is set for each data type in each level standard; Construct a multi-level dynamic acquisition frequency trigger logic based on the set dynamic acquisition frequency, wherein the multi-level dynamic acquisition frequency trigger logic is specifically to automatically adjust the acquisition frequency of each data type in the total amount of data corresponding to the level standard when the total amount of data reaches different level standards; The specific steps for obtaining network data in real time and monitoring it regularly through sensors and data collection points are as follows: Use sensors and data collection points to obtain network data regularly and in real time; Monitor and clean the real-time acquired network data, remove duplicate, erroneous or invalid data, and obtain pre-processed network data; The current total amount of pre-processed network data is calculated using real-time analysis technology, and the calculated current total amount of pre-processed network data is compared with the preset grade standard to determine the grade standard of the total amount of data to which it belongs; According to the level standard of real-time acquisition of network data, the characteristics of each data type are collected through multi-level dynamic collection frequency trigger logic; According to the preset security threshold segments and the characteristics of each data type collected, each data type is prioritized and security analysis is performed on them in order to obtain security analysis data results. The specific steps are as follows: Set a plurality of corresponding security threshold segments for each data type, and define corresponding priorities for the plurality of security threshold segments; Determine the security threshold segment to which the characteristic values ​​of each data type belong, and prioritize each data type based on the security threshold segment; Based on the divided priorities, security analysis is performed on the characteristics of each data type in descending order; Based on the security analysis data results, the corresponding warning measures are triggered for real-time execution.

2. A method for analyzing network data according to claim 1, characterized in that: The specific steps of setting the level standard of the total amount of data based on the total amount of network data and the importance of data within the periodic collection time are as follows: Regularly collect all relevant network data types within a time period, including but not limited to traffic data, user behavior data and system logs; Multiple level standards are set according to the system resource usage and processing complexity of all relevant data types in the network data within a time period, including low, medium and high levels.

3. A method for analyzing network data according to claim 1, characterized in that: The specific steps of collecting the characteristics of each data type according to the level standard of real-time acquisition of network data through multi-level dynamic acquisition frequency trigger logic are as follows: According to the level standard of real-time acquisition of network data, the corresponding acquisition frequency is assigned to each data type through multi-level dynamic acquisition frequency trigger logic; Based on allocating corresponding collection frequencies to each data type, features are collected for each data type respectively, and feature values ​​of each data type are obtained.

4. A method for analyzing network data according to claim 1, characterized in that: The specific steps of triggering corresponding warning measures for real-time execution based on the security analysis data results are: The risk level of the safety analysis data results is divided according to the preset risk level threshold range, where the risk level specifically includes dangerous, urgent and routine; Automatically activate emergency response measures corresponding to the danger level and emergency level based on the risk level; Monitor emergency response measures until the risk level of the safety analysis data results is re-determined to be at the normal level.

5. A network data analysis system, applied to a network data analysis method according to any one of claims 1 to 4, characterized in that: The analysis system comprises: The level definition module is used to set the level standard of the total amount of data based on the total amount of network data and the importance of data within the periodic collection time, including low level, medium level and high level; The dynamic definition module is used to set the multi-level dynamic acquisition frequency trigger logic for each data type based on the level standard of the total amount of data. The specific steps are as follows: Based on the statistics of historical data, for the total amount of data of the three level standards of low, medium and high, the corresponding dynamic collection frequency is set for each data type in each level standard; Construct a multi-level dynamic acquisition frequency trigger logic based on the set dynamic acquisition frequency, wherein the multi-level dynamic acquisition frequency trigger logic is specifically to automatically adjust the acquisition frequency of each data type in the total amount of data corresponding to the level standard when the total amount of data reaches different level standards; The data monitoring module is used to obtain network data in real time and monitor it regularly through sensors and data collection points, and to determine the level standard of regular real-time acquisition of network data. The specific steps are as follows: Use sensors and data collection points to regularly obtain network data in real time; Monitor and clean the real-time acquired network data, remove duplicate, erroneous or invalid data, and obtain pre-processed network data; The current total amount of pre-processed network data is calculated using real-time analysis technology, and the calculated current total amount of pre-processed network data is compared with the preset grade standard to determine the grade standard of the total amount of data to which it belongs; Dynamic collection module, used to collect characteristics of various data types according to the level standard of real-time acquisition of network data through multi-level dynamic collection frequency trigger logic; The priority classification module is used to prioritize each data type according to the preset security threshold segment and the characteristics of each data type collected, and perform security analysis on them in sequence to obtain security analysis data results. The specific steps are as follows: Set a plurality of corresponding security threshold segments for each data type, and define corresponding priorities for the plurality of security threshold segments; Determine the security threshold segment to which the characteristic values ​​of each data type belong, and prioritize each data type based on the security threshold segment; Based on the divided priorities, security analysis is performed on the characteristics of each data type in descending order; The warning control module is used to trigger corresponding warning measures for real-time execution based on the security analysis data results.

Images