Abnormal traffic security assessment management system and method based on big data

Abstract

The invention discloses an abnormal traffic safety assessment management system and method based on big data, and relates to the technical field of abnormal traffic detection, and the method comprises the steps: collecting traffic data in real time, and classifying the collected traffic data; extracting hierarchical traffic features based on the classified traffic data; formulating a layered traffic filtering mechanism based on layered traffic characteristics, and screening potential abnormal traffic of each layer; inputting the potential abnormal traffic of each layer into an abnormal traffic evaluation model, and performing abnormal traffic safety evaluation; and generating a security assessment report according to an assessment result, and optimizing the abnormal traffic assessment model in real time. According to the method, the abnormal traffic evaluation model is constructed, so that automatic identification and quantitative evaluation of the abnormal traffic are realized, the accuracy and automation level of evaluation are improved, and a more intelligent network security defense system can be constructed.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal traffic detection, and particularly to a big data-based abnormal traffic security evaluation management system and method. Background Art

[0002] With the rapid development of Internet technology and the continuous expansion of network scale, the complexity and diversity of network traffic are also increasing day by day. Especially in the big data era, the amount of traffic data processed by network devices every day shows an exponential growth, which brings new challenges to network security detection and management. Traditional network security management methods mainly rely on rule matching and the identification of specific attack patterns, such as static rule filtering based on firewalls or signature-based intrusion detection systems. However, these methods have significant limitations when facing increasingly complex and changeable network attacks. Especially for distributed denial of service attacks (DDoS), packet manipulation attacks, and other new network threats, traditional security protection means often cannot detect and respond in a timely and effective manner. In addition, as the means of network attackers continue to evolve, it is increasingly difficult to identify attack patterns through simple rule matching, which makes traditional network security methods appear powerless in real-time detection and response to abnormal traffic.

[0003] Although existing big data security evaluation technologies have tried to use methods such as machine learning or behavior baselines to improve detection efficiency, there are still some significant deficiencies. First, there are efficiency bottlenecks in traffic data cleaning and feature extraction in existing technologies, and the redundancy and noise of traffic data will interfere with subsequent anomaly detection. Second, existing abnormal traffic evaluation models usually analyze only based on single-level traffic features, ignoring the interaction effects between the physical layer, network layer, transport layer, and application layer, which results in insufficient recognition ability of the model for complex cross-layer attacks. In addition, existing technologies often lack a dynamic optimization mechanism and cannot adjust and optimize the detection model in real time according to the latest network threats, resulting in a high response lag when facing emerging attack means. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a big data-based abnormal traffic security evaluation management method to solve the problems of low efficiency of traffic data cleaning, insufficient cross-layer feature analysis, and lack of a dynamic optimization mechanism in the prior art.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for anomaly traffic security assessment and management based on big data, which includes: collecting traffic data in real time, classifying the collected traffic data; extracting hierarchical traffic features based on the classified traffic data; formulating a hierarchical traffic filtering mechanism based on the hierarchical traffic features to screen potential anomaly traffic at each layer; inputting the potential anomaly traffic at each layer into an anomaly traffic assessment model for anomaly traffic security assessment; generating a security assessment report according to the assessment results, and optimizing the anomaly traffic assessment model in real time.

[0008] As a preferred solution of the method for anomaly traffic security assessment and management based on big data according to the present invention, before collecting traffic data in real time, it further includes: constructing an anomaly traffic assessment model, and the specific steps are as follows:

[0009] Using HMM as the basic model, utilizing the time series analysis ability of HMM to capture the state changes of traffic in the time series, judging whether the traffic is in an abnormal state, training the state transition matrix and observation probability matrix of the model through the Baum-Welch algorithm, and performing interactive analysis on the features of each layer, and finally constructing an anomaly traffic assessment model;

[0010] The traffic data includes IP address, port number, protocol type, packet length, and timestamp.

[0011] As a preferred solution of the method for anomaly traffic security assessment and management based on big data according to the present invention, the classification of the collected traffic data is specifically as follows:

[0012] Clean and format the collected data, remove redundant data and error data, and unify the timestamp;

[0013] Parse layer by layer according to the OSI model, classify the cleaned traffic data into the physical layer, network layer, transport layer, and application layer, and store the classified traffic data separately.

[0014] As a preferred solution of the method for anomaly traffic security assessment and management based on big data according to the present invention, the extraction of hierarchical traffic features based on the classified traffic data is specifically as follows:

[0015] Monitor the signal status through physical layer devices, and extract link status and signal quality;

[0016] By parsing the IP protocol header in the network layer, extract the source IP address, destination IP address, TTL value, IP packet length, and protocol type;

[0017] By parsing the TCP / UDP protocol header in the transport layer, extract the source port number, destination port number, TCP flag bits, and window size;

[0018] By parsing the application layer protocol packets, extract the request method, domain name resolution information, and URI path.

[0019] As a preferred solution of the big data-based abnormal traffic security assessment and management method of the present invention, wherein: a hierarchical traffic filtering mechanism is formulated based on hierarchical traffic characteristics to screen potential abnormal traffic at each layer, and the specific steps are as follows:

[0020] Formulate a physical layer filtering mechanism based on the physical link state and signal quality to detect signal anomalies and monitor abnormal link behaviors at the same time;

[0021] Formulate a network layer filtering mechanism based on the source IP address, destination IP address, TTL value, IP packet length, and protocol type to screen traffic with potential abnormal transmission behaviors;

[0022] Formulate a transport layer filtering mechanism based on the source port number, destination port number, TCP flag bits, and window size to screen potential transport layer attacks;

[0023] Formulate an application layer filtering mechanism based on the request method, domain name resolution information, and URI path to screen potential anomalies at the application layer;

[0024] Based on the hierarchical filtering mechanism, perform feature matching, statistical analysis, and behavior baselines to detect and screen potential abnormal traffic at each layer in real time.

[0025] As a preferred solution of the big data-based abnormal traffic security assessment and management method of the present invention, wherein: input the potential abnormal traffic at each layer into the abnormal traffic assessment model for abnormal traffic security assessment, and the specific steps are as follows:

[0026] Input the potential abnormal traffic at each layer into the abnormal traffic assessment model to predict the security score of the potential abnormal traffic. The expression is:

[0027]

[0028] Among them, S represents the comprehensive security score of the potential abnormal traffic, T is the time window length, and Q P (t,F P ) represents the feature score of the physical layer at time t, and Q N (t,F N ) represents the feature score of the network layer at time t, and Q T (t,F T ) represents the feature score of the transport layer at time t, and Q A (t,F A ) represents the feature score of the application layer at time t, and F P represents the potential abnormal traffic of the physical layer, and F N represents the potential abnormal traffic of the network layer,T represents potential abnormal traffic in the transport layer, F A represents potential abnormal traffic in the application layer, α is the weight coefficient of the physical layer, and β is the weight coefficient when the application layer and the transport layer interact;

[0029] Based on historical traffic data and business requirements, define a low-risk threshold θ1 and a high-risk threshold θ2;

[0030] When S ≤ θ1, the traffic security is in a normal state;

[0031] When θ1 < S ≤ θ2, the traffic security is in a mild abnormal state;

[0032] When S > θ2, the traffic security is in a severe abnormal state.

[0033] As a preferred solution of the abnormal traffic security assessment and management method based on big data according to the present invention, wherein: the real-time optimization of the abnormal traffic assessment model is specifically as follows:

[0034] Utilize the latest traffic characteristics and feedback data to continuously train the abnormal traffic assessment model, update the state transition matrix and the observation probability matrix, and dynamically adjust the feature weights at different levels.

[0035] In a second aspect, the present invention provides an abnormal traffic security assessment and management system based on big data, including a data classification module, a feature extraction module, an abnormal traffic screening module, a security assessment module, and a report generation module; the data classification module: used to collect traffic data in real time and classify the collected traffic data; the feature extraction module: used to extract hierarchical traffic characteristics based on the classified traffic data; the abnormal traffic screening module: used to formulate a hierarchical traffic filtering mechanism based on the hierarchical traffic characteristics and screen the potential abnormal traffic at each layer; the security assessment module: used to input the potential abnormal traffic at each layer into the abnormal traffic assessment model for abnormal traffic security assessment; the report generation module: used to generate a security assessment report according to the assessment results and real-time optimize the abnormal traffic assessment model.

[0036] In a third aspect, the present invention provides a computer device, including a memory and a processor, wherein: the computer program stored in the memory, wherein: when the computer program is executed by the processor, it realizes any step of the abnormal traffic security assessment and management method based on big data according to the first aspect of the present invention.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, wherein: when the computer program is executed by the processor, it realizes any step of the abnormal traffic security assessment and management method based on big data according to the first aspect of the present invention.

[0038] The beneficial effects of the present invention are as follows: By collecting and classifying traffic data in real time and extracting hierarchical traffic features, the accuracy and consistency of the data are achieved. Based on the hierarchical traffic features, a hierarchical traffic filtering mechanism is formulated to realize the refined management and efficient identification of abnormal traffic, enhancing the system security. Further, an abnormal traffic assessment model is constructed, and by utilizing the time series analysis ability of HMM, the automatic identification and quantitative assessment of abnormal traffic are realized. A security assessment report is generated according to the assessment results, and the model is optimized in real time, improving the accuracy and timeliness of detection and significantly enhancing the network security protection level. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0040] Figure 1 It is a flowchart of the method for abnormal traffic security assessment and management based on big data in Embodiment 1.

[0041] Figure 2 It is a schematic diagram of the system for abnormal traffic security assessment and management based on big data in Embodiment 1. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention will be described in detail below with reference to the drawings of the specification.

[0043] Many specific details are set forth in the following description in order to provide a thorough understanding of the present invention, but the present invention may be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0044] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation manner of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that excludes other embodiments.

[0045] Embodiment 1, referring to Figure 1 and Figure 2 , is the first embodiment of the present invention. This embodiment provides a method for abnormal traffic security assessment and management based on big data, including the following steps:

[0046] S1: Collect traffic data in real time, classify the collected traffic data, specifically including:

[0047] S1.1: Before collecting traffic data in real time, it also includes: constructing an abnormal traffic evaluation model, and the specific steps are as follows:

[0048] Use HMM as the basic model, utilize the time series analysis ability of HMM to capture the state changes of traffic in the time series, judge whether the traffic is in an abnormal state, train the state transition matrix and observation probability matrix of the model through the Baum-Welch algorithm, conduct interactive analysis on each layer of features, and finally construct an abnormal traffic evaluation model.

[0049] Further, the state transition probability matrix: represents the probability of transferring from one traffic state (such as normal) to another state (such as abnormal), and the expression is:

[0050]

[0051] a 11 : Represents the probability of transferring from the normal state to the normal state.

[0052] a 12 : Represents the probability of transferring from the normal state to the mildly abnormal state.

[0053] a 13 : Represents the probability of transferring from the normal state to the severely abnormal state.

[0054] a 21 : Represents the probability of transferring from the mildly abnormal state to the normal state.

[0055] a 22 : Represents the probability of transferring from the mildly abnormal state to the mildly abnormal state.

[0056] a 23 : Represents the probability of transferring from the mildly abnormal state to the severely abnormal state.

[0057] a 31 : Represents the probability of transferring from the severely abnormal state to the normal state.

[0058] a 32 : Represents the probability of transferring from the severely abnormal state to the mildly abnormal state.

[0059] a 33 : Represents the probability of transferring from the severely abnormal state to the severely abnormal state.

[0060] The observation probability matrix: represents the probability of observing a specific eigenvalue in a certain state, and the expression is:

[0061]

[0062] b 11 : Represents the probability of observing eigenvalue 1 in the normal state.

[0063] b 12 : Represents the probability of observing eigenvalue 2 in the normal state.

[0064] b 1n : Represents the probability of observing eigenvalue n in the normal state.

[0065] b 21 : Represents the probability of observing eigenvalue 1 in the mildly abnormal state.

[0066] b 22 : Represents the probability of observing eigenvalue 2 in the mildly abnormal state.

[0067] b 2n : Represents the probability of observing eigenvalue n in the mildly abnormal state.

[0068] b 31 : Represents the probability of observing eigenvalue 1 in the severely abnormal state.

[0069] b 32 : Represents the probability of observing eigenvalue 2 in the severely abnormal state.

[0070] b 3n : Represents the probability of observing eigenvalue n in the severely abnormal state.

[0071] Traffic data includes IP address, port number, protocol type, packet length, and timestamp.

[0072] It should be noted that collecting the above data can comprehensively analyze the details of network communication, accurately identify and evaluate potential abnormal traffic.

[0073] S1.2: Clean and format the collected data, remove redundant and error data, and unify the timestamp.

[0074] Specifically, identify and remove duplicate records and error entries to ensure the accuracy and consistency of the data. At the same time, convert the time stamps of all data to the same format and time zone to unify the timestamp and ensure the standardization of time information.

[0075] S1.3: Parse layer by layer according to the OSI model, classify the cleaned traffic data into the physical layer, network layer, transport layer, and application layer, and store the classified traffic data separately.

[0076] Furthermore, first analyze the traffic data layer by layer according to the seven-layer structure of the OSI model, classify the data into the physical layer, network layer, transport layer, and application layer, and then store the classified traffic data separately to ensure the independence and orderliness of the data for each layer.

[0077] The specific process of analyzing the traffic data layer by layer is as follows: First, at the physical layer, extract information related to the network link status, such as link connection status, signal strength, link rate, etc., to ensure the stability of the physical link. Next, at the network layer, parse the IP protocol header and extract key information such as source IP address, destination IP address, TTL value, IP packet length, protocol type, etc., to identify the routing and transmission path of the data packet. Then, at the transport layer, parse the TCP / UDP protocol header and extract parameters such as source port number, destination port number, TCP flag bits (such as SYN, ACK, FIN), and window size, etc., to analyze the communication status and connection behavior at the transport layer. Finally, at the application layer, parse the application layer protocol messages (such as HTTP, DNS, etc.) and extract request methods, URI paths, domain name resolution information, etc., to deeply analyze the data transmission behavior at the application layer. Through this layer-by-layer analysis process, ensure the effective extraction of traffic characteristics for each layer and store the classified data separately, making subsequent anomaly detection and analysis more accurate.

[0078] It should also be noted that although the OSI model has a seven-layer structure, in network traffic analysis, only the physical layer, network layer, transport layer, and application layer are focused on because these four layers contain the key information most relevant to traffic control and data transmission.

[0079] S2: Extract hierarchical traffic characteristics based on the classified traffic data, specifically including:

[0080] S2.1: Monitor the signal status through physical layer devices and extract the link status and signal quality.

[0081] Specifically, use physical layer devices (such as the physical interfaces of switches and routers) to monitor the signal status of the network link. The signal status of the network link includes the connection status of the link (such as whether the connection is normal, whether there is a disconnection) and signal quality (such as signal strength, noise ratio, etc.). For example, by monitoring the CRC error rate and signal attenuation of the link, it can be determined whether there are physical faults or interference on the link.

[0082] S2.2: Extract the source IP address, destination IP address, TTL value, IP packet length, and protocol type by parsing the IP protocol header in the network layer.

[0083] Specifically, analyze the header information of IP data packets and extract key fields, such as source IP address, destination IP address, TTL value (time to live), IP packet length, and protocol type (such as TCP, UDP, ICMP, etc.). For example, by checking the TTL value, the number of hops of the data packet in the network can be judged. A too low TTL value may indicate the risk of data packet loops or being tampered with.

[0084] S2.3: By parsing the TCP / UDP protocol headers at the transport layer, extract the source port number, destination port number, TCP flag bits (such as SYN, ACK, FIN), and window size.

[0085] Specifically, analyze the header information of the TCP or UDP protocol and extract the source port number, destination port number, TCP flag bits (such as SYN, ACK, FIN, etc.), and window size. For example, by checking the TCP flag bits, specific network behaviors can be identified, such as SYN Flood attacks (a large number of SYN packets causing server resource exhaustion) or the normal three-way handshake process.

[0086] S2.4: By parsing application layer protocol messages (such as HTTP, DNS, etc.), extract the request method, domain name resolution information, and URI path.

[0087] Specifically, analyze the data messages of the application layer protocol and extract the specific request method (such as GET, POST in HTTP), domain name resolution information (such as DNS query response), and URI path (such as the URL path of an HTTP request). For example, by analyzing the method and URI path of an HTTP request, potential Web attacks, such as SQL injection or cross-site scripting attacks, can be identified.

[0088] S3: Develop a hierarchical traffic filtering mechanism based on hierarchical traffic characteristics to screen potential abnormal traffic at each layer.

[0089] S3.1: Develop a physical layer filtering mechanism based on the physical link state and signal quality to detect signal anomalies and monitor abnormal link behaviors simultaneously.

[0090] Specifically, monitor the connection state and signal quality of the link through physical layer devices, detect signal anomalies such as link interruptions and frequent fluctuations, and monitor abnormal link behaviors such as device hardware failures and malicious interference on the physical link. Once an anomaly is detected, immediately trigger the filtering mechanism to isolate or mark suspicious traffic to ensure network stability and security.

[0091] It should be noted that the physical layer filtering mechanism specifically refers to monitoring the connection status and signal quality of the link through physical layer devices (such as the physical interfaces of switches and routers), detecting signal anomalies such as link interruptions and frequent fluctuations, and simultaneously monitoring abnormal link behaviors such as device hardware failures and malicious interference on the physical link. Once an anomaly is detected, the filtering mechanism is immediately triggered to isolate or mark suspicious traffic, ensuring the stability and security of the network. For example, by monitoring the CRC error rate and signal attenuation of the link, it is possible to determine whether there are physical faults or interference on the link.

[0092] S3.2: Develop a network layer filtering mechanism based on the source IP address, destination IP address, TTL value, IP packet length, and protocol type to screen traffic with potential abnormal transmission behaviors.

[0093] For example, use the source IP and destination IP to match firewall rules. A too-low TTL value may indicate an abnormal transmission path of the data packet, and an abnormal IP packet length may be a sign of fragmentation attacks or data packet manipulation.

[0094] It should be noted that the network layer filtering mechanism specifically refers to formulating filtering rules based on the source IP address, destination IP address, TTL value, IP packet length, and protocol type to screen traffic with potential abnormal transmission behaviors. By parsing the header information of the IP data packet, extracting key fields such as the source IP address, destination IP address, TTL value (time to live), IP packet length, and protocol type (such as TCP, UDP, ICMP, etc.), filtering rules are set to detect potential abnormal behaviors. For example, use the source IP and destination IP to match firewall rules. A too-low TTL value may indicate an abnormal transmission path of the data packet, and an abnormal IP packet length may be a sign of fragmentation attacks or data packet manipulation. Once abnormal traffic is detected, the filtering mechanism is immediately triggered to block or mark these potential attack behaviors, ensuring the security and stability of network transmission.

[0095] S3.3: Develop a transport layer filtering mechanism based on the source port number, destination port number, TCP flag bits, and window size to screen potential transport layer attacks.

[0096] For example, identify port scanning behaviors by checking the source port number and destination port number, detect SYN Flood attacks by analyzing TCP flag bits (such as a large number of SYN packets), identify traffic manipulation attacks by changes in the window size. Once abnormal traffic is detected, the filtering mechanism is immediately triggered to block or mark these potential attack behaviors, ensuring the security and stability of network transmission.

[0097] It should be noted that typical filtering rules include detecting port scanning, SYN Flood attacks (abnormal SYN flag bits), or abnormal window sizes (which may lead to traffic manipulation attacks).

[0098] S3.4: Develop an application layer filtering mechanism based on the request method, domain name resolution information, and URI path to screen for potential anomalies at the application layer.

[0099] For example, identify potential application layer attacks by detecting abnormal URI lengths, excessive request frequencies, or abnormal behaviors in domain name resolution (such as DNS amplification attacks, malicious website redirects, etc.). Once an anomaly is detected, immediately trigger the filtering mechanism to block or mark this suspicious traffic to ensure the security and normal operation of the application layer.

[0100] It should be noted that the filtering rules include detecting abnormal URI lengths, excessive request frequencies, or abnormal behaviors in domain name resolution (such as DNS amplification attacks, malicious website redirects, etc.).

[0101] S3.5: Based on the hierarchical filtering mechanism, perform signature matching, statistical analysis, and behavior baselining to detect and screen potential abnormal traffic in each layer in real time.

[0102] It should be noted that for rule matching: Use signature-based detection methods to match traffic characteristics with known attack patterns (such as IP blacklists, port scan patterns, etc.) layer by layer. If the characteristics of a certain layer match the known attack pattern, mark this traffic as potentially abnormal.

[0103] Statistical analysis: Through statistical analysis of characteristics, find traffic behaviors that exceed the normal threshold. For example, excessive TCP connection attempts (SYN Flood), abnormally large packet lengths (network layer attacks), or abnormal HTTP request frequencies (application layer DDoS) are all potential abnormal traffic.

[0104] Behavior baselining: Establish a baseline for normal traffic through machine learning or historical data, and any behavior that deviates from the baseline is regarded as potentially abnormal. For example, normally the number of ICMP packets of a certain IP address is low, but a sudden large amount of ICMP traffic may be a sign of a PingFlood attack.

[0105] S4: Input the potential abnormal traffic in each layer into the abnormal traffic assessment model for abnormal traffic security assessment.

[0106] S4.1: Input the potential abnormal traffic in each layer into the abnormal traffic assessment model to predict the security score of the potential abnormal traffic. The expression is:

[0107]

[0108] where S represents the comprehensive security score of the potential abnormal traffic, T is the time window length, and Q P (t,F P ) represents the feature score of the physical layer at time t, and Q N (t,FN ) represents the feature score of the network layer at time t, Q T (t, F T ) represents the feature score of the transport layer at time t, Q A (t, F A ) represents the feature score of the application layer at time t, F P represents the potential abnormal traffic feature of the physical layer, F N represents the potential abnormal traffic feature of the network layer, F T represents the potential abnormal traffic feature of the transport layer, F A represents the potential abnormal traffic feature of the application layer. α is the weight coefficient of the physical layer, and β is the weight coefficient when the application layer and the transport layer interact.

[0109] It should be noted that a comprehensive score reflecting the traffic security status is obtained through this expression. The feature score of the physical layer enhances its influence through an exponential function, the feature score of the network layer is smoothed through a logarithmic function, and the feature scores of the transport layer and the application layer are comprehensively evaluated through a square root function and the weighting coefficient β. The comprehensive score S can comprehensively reflect the abnormal degree of traffic at multiple levels and provide a scientific basis for security assessment.

[0110] S4.2: Define the low-risk threshold θ1 and the high-risk threshold θ2 based on historical traffic data and business requirements.

[0111] When S ≤ θ1, the traffic security is in a normal state.

[0112] When θ1 < S ≤ θ2, the traffic security is in a mild abnormal state.

[0113] When S > θ2, the traffic security is in a severe abnormal state.

[0114] It should be noted that by analyzing historical traffic data, statistically analyzing the feature distributions of normal traffic and known abnormal traffic, determining the typical range of normal traffic and the feature values of abnormal traffic. Then, combined with business requirements and security policies, select appropriate percentiles or standard deviations as reference points for the thresholds. Usually, θ1 is set as the upper limit of the normal traffic feature value, such as the highest value of 95% of the normal traffic data; θ2 is set as the lower limit of the high-risk abnormal traffic feature value, such as the lowest value of 99% of the abnormal traffic data.

[0115] S5: Generate a security assessment report based on the evaluation results and optimize the abnormal traffic assessment model in real time.

[0116] The security assessment report includes an overview of the overall traffic status, specific information about abnormal traffic, the feature analysis results of each layer, the detected types of potential threats, and the corresponding risk levels.

[0117] It should be noted that the specific information of abnormal traffic includes the abnormal IP address, port number, and protocol type.

[0118] Potential threat types include DDoS attacks, port scans, and SYN Floods, etc.

[0119] The real-time optimization of the abnormal traffic assessment model is as follows:

[0120] Utilize the latest traffic characteristics and feedback data to continuously train the abnormal traffic assessment model, update the state transition matrix and observation probability matrix, and dynamically adjust the feature weights at different levels.

[0121] Specifically, collect the latest traffic data in the network and the feedback data of the detection results in real time, use these data to retrain the model, update the state transition matrix to reflect the new state transition probability, update the observation probability matrix to reflect the new feature observation probability, and adjust the weights of each layer of features according to the new data to ensure that the model can adapt to the changes in the network environment in real time and improve the accuracy and timeliness of abnormal traffic detection.

[0122] It should also be noted that the feedback data comes from the continuous monitoring and analysis of the latest traffic characteristics and detection results.

[0123] This embodiment also provides a big data-based abnormal traffic security assessment and management system, including: a data classification module, a feature extraction module, an abnormal traffic screening module, a security assessment module, and a report generation module;

[0124] Data classification module: used to collect traffic data in real time and classify the collected traffic data;

[0125] Feature extraction module: used to extract hierarchical traffic features based on the classified traffic data;

[0126] Abnormal traffic screening module: used to formulate a hierarchical traffic filtering mechanism based on the hierarchical traffic features and screen the potential abnormal traffic at each layer;

[0127] Security assessment module: used to input the potential abnormal traffic at each layer into the abnormal traffic assessment model for abnormal traffic security assessment;

[0128] Report generation module: used to generate a security assessment report according to the assessment results and optimize the abnormal traffic assessment model in real time.

[0129] This embodiment also provides a computer device, applicable to the situation of the big data-based abnormal traffic security assessment and management method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the big data-based abnormal traffic security assessment and management method proposed in the above embodiment.

[0130] The computer device can be a terminal, which includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0131] This embodiment also provides a storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the method for realizing the security assessment and management of abnormal traffic based on big data as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM for short), Electrically Erasable Programmable Read-Only Memory (EEPROM for short), Erasable Programmable Read-Only Memory (EPROM for short), Programmable Read-Only Memory (PROM for short), Read-Only Memory (ROM for short), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0132] In summary, through the following steps, the present invention: collects and classifies traffic data in real time, extracts hierarchical traffic features, and realizes the accuracy and consistency of data. Based on the hierarchical traffic features, a hierarchical traffic filtering mechanism is formulated, realizing the refined management and efficient identification of abnormal traffic, and enhancing the system security. Further constructs an abnormal traffic assessment model, utilizes the time series analysis ability of HMM, and realizes the automatic identification and quantitative assessment of abnormal traffic. Generates a security assessment report according to the assessment results and optimizes the model in real time, improving the accuracy and timeliness of detection, and significantly enhancing the network security protection level.

[0133] Example 2. Referring to Table 1, this is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the abnormal traffic security assessment management method based on big data are given.

[0134] To verify the effectiveness and innovation of the abnormal traffic security assessment management method based on big data, an enterprise internal network environment is selected as the test object in this example. This network environment contains multiple subnets and generates a large amount of traffic data every day, including normal traffic and various types of abnormal traffic (such as DDoS attacks, SYN Flood attacks, port scans, etc.).

[0135] First, use network traffic monitoring devices to collect traffic data in real time, and clean and format the data through Python scripts to remove redundant and error data, unify the timestamp format, and ensure the accuracy and consistency of the data.

[0136] Secondly, in the data classification and feature extraction stage, parse the traffic data layer by layer according to the seven-layer structure of the OSI model, classify the data into the physical layer, network layer, transport layer, and application layer, and extract the feature information of each layer, such as link status, IP address, port number, request method, etc.

[0137] Then, formulate a hierarchical traffic filtering mechanism based on the hierarchical traffic characteristics. Through the filtering rules of the physical layer, network layer, transport layer, and application layer, detect and screen out potential abnormal traffic to ensure the effective implementation of security protection measures for each layer.

[0138] Finally, build an abnormal traffic assessment model, use the HMM model and the Baum-Welch algorithm to train the state transition matrix and the observation probability matrix, input the potential abnormal traffic into the model for security scoring, generate a security report according to the evaluation results, and continuously optimize the model through the latest traffic characteristics and feedback data to ensure its accuracy and timeliness.

[0139] Specifically, it is shown in Table 1 below:

[0140] Table 1 Multilayer Security Feature Analysis Table of Traffic and Attack Samples

[0141]

[0142] Through the data analysis of the above table, it can be clearly seen that the present invention has significant advantages in improving network security and detection accuracy. For example, the comprehensive security scores of normal traffic are generally above 0.88, while the comprehensive security scores of various types of attack traffic (such as DDoS attacks, SYN Flood attacks, and port scans) are all below 0.30, far lower than that of normal traffic. In addition, the feature scores of the present invention at each level (physical layer, network layer, transport layer, and application layer) are also significantly better than those of the prior art. Especially at the transport layer and application layer, the feature scores of attack traffic are significantly reduced, indicating that the present invention can more effectively identify and resist various network attacks.

[0143] Through the method for abnormal traffic security assessment and management based on big data of the present invention, not only can the ability to distinguish normal traffic from attack traffic be significantly improved, but also the detection accuracy of features at each layer can be greatly enhanced. This reflects the high efficiency and reliability of the present invention in the field of network security, especially the advantages in terms of comprehensive security scores and multi-level feature analysis. For example, the comprehensive security scores of normal traffic are generally above 0.88, while the comprehensive security scores of various types of attack traffic (such as DDoS attacks, SYN Flood attacks, and port scans) are all below 0.30, far lower than that of normal traffic. At the same time, the feature scores of the present invention at the physical layer, network layer, transport layer, and application layer are all significantly better than those of the prior art. Especially at the transport layer and application layer, the feature scores of attack traffic are significantly reduced, indicating that the present invention can more effectively identify and resist various network attacks.

[0144] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not restrictive. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A method for abnormal traffic security assessment management based on big data, characterized in that: include: Collect traffic data in real time and classify the collected traffic data; Extracting hierarchical traffic features based on classified traffic data; Develop a layered traffic filtering mechanism based on layered traffic characteristics to filter out potential abnormal traffic at each layer; Input the potential abnormal traffic of each layer into the abnormal traffic assessment model to conduct abnormal traffic security assessment; Generate a security assessment report based on the assessment results and optimize the abnormal traffic assessment model in real time.

2. The abnormal traffic safety assessment management method based on big data according to claim 1 is characterized in that: Before collecting traffic data in real time, it also includes: building an abnormal traffic evaluation model. The specific steps are as follows: HMM is used as the basic model. The time series analysis capability of HMM is used to capture the state changes of traffic in the time series and determine whether the traffic is in an abnormal state. The state transition matrix and observation probability matrix of the model are trained by the Baum-Welch algorithm. The features of each layer are interactively analyzed to finally form an abnormal traffic evaluation model. The traffic data includes IP address, port number, protocol type, data packet length and timestamp.

3. The abnormal traffic safety assessment management method based on big data according to claim 2 is characterized in that: The collected traffic data is classified, and the specific steps are as follows: Clean and format the collected data, remove redundant and erroneous data, and unify the timestamps; According to the OSI model, the cleaned traffic data is analyzed layer by layer, classified into the physical layer, network layer, transport layer and application layer, and the classified traffic data is stored separately.

4. The abnormal traffic safety assessment management method based on big data according to claim 3 is characterized in that: The specific steps of extracting layered traffic features based on classified traffic data are as follows: Monitor signal status through physical layer devices to extract link status and signal quality; By parsing the IP protocol header in the network layer, the source IP address, destination IP address, TTL value, IP packet length and protocol type are extracted; By parsing the TCP / UDP protocol header of the transport layer, the source port number, destination port number, TCP flag bit and window size are extracted; By parsing the application layer protocol message, the request method, domain name resolution information and URI path are extracted.

5. The abnormal traffic safety assessment management method based on big data according to claim 4 is characterized in that: The layered traffic filtering mechanism is formulated based on the layered traffic characteristics to filter out potential abnormal traffic at each layer. The specific steps are as follows: Develop physical layer filtering mechanisms based on physical link status and signal quality to detect signal anomalies and monitor abnormal link behavior; Develop network layer filtering mechanisms based on source IP address, destination IP address, TTL value, IP packet length, and protocol type to filter traffic with potential abnormal transmission behavior; Develop a transport layer filtering mechanism based on the source port number, destination port number, TCP flag bit and window size to screen out potential transport layer attacks; Develop application layer filtering mechanisms based on request methods, domain name resolution information, and URI paths to screen for potential anomalies at the application layer; Based on the layered filtering mechanism, feature matching, statistical analysis and behavioral baseline are performed to detect and filter potential abnormal traffic at each layer in real time.

6. The abnormal traffic safety assessment management method based on big data according to claim 5 is characterized in that: The potential abnormal traffic of each layer is input into the abnormal traffic assessment model to perform abnormal traffic safety assessment. The specific steps are as follows: The potential abnormal traffic of each layer is input into the abnormal traffic assessment model to predict the safety score of the potential abnormal traffic. The expression is: Among them, S represents the comprehensive security score of potential abnormal traffic, T is the time window length, and Q P (t,F P ) represents the feature score of the physical layer at time t, Q N (t,F N ) represents the feature score of the network layer at time t, Q T (t,F T ) represents the feature score of the transmission layer at time t, Q A (t,F A ) represents the feature score of the application layer at time t, F P Indicates potential abnormal traffic at the physical layer, F N Indicates the potential abnormal traffic at the network layer, F T Indicates the potential abnormal traffic at the transport layer, F A It represents the potential abnormal traffic of the application layer, α is the weight coefficient of the physical layer, and β is the weight coefficient when the application layer and the transport layer interact; Based on historical traffic data and business requirements, define low risk threshold θ1 and high risk threshold θ2; When S≤θ1, traffic safety is in normal state; When θ1<S≤θ2, traffic safety is in a slightly abnormal state; When S>θ2, traffic safety is in a severely abnormal state.

7. The abnormal traffic safety assessment management method based on big data according to claim 6 is characterized in that: The specific steps of the real-time optimization abnormal traffic evaluation model are as follows: Utilizing the latest traffic characteristics and feedback data, the abnormal traffic assessment model is continuously trained, the state transfer matrix and observation probability matrix are updated, and the feature weights at different levels are dynamically adjusted.

8. An abnormal traffic safety assessment management system based on big data, based on the abnormal traffic safety assessment management method based on big data according to any one of claims 1 to 7, characterized in that: include: Data classification module, feature extraction module, abnormal traffic screening module, security assessment module and report generation module; Data classification module: used to collect traffic data in real time and classify the collected traffic data; Feature extraction module: used to extract hierarchical traffic features based on classified traffic data; Abnormal traffic screening module: used to formulate a layered traffic filtering mechanism based on layered traffic characteristics and screen potential abnormal traffic at each layer; Security assessment module: used to input the potential abnormal traffic of each layer into the abnormal traffic assessment model to perform abnormal traffic security assessment; Report generation module: used to generate security assessment reports based on assessment results and optimize abnormal traffic assessment models in real time.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the abnormal traffic security assessment management method based on big data are implemented in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the abnormal traffic security assessment management method based on big data described in any one of claims 1 to 7 are implemented.

Images