Network security monitoring and analyzing system based on machine learning
Through a machine learning-based network security monitoring and analysis system, network data is collected and analyzed in real time and abnormal threats are identified, the problems of false alarms and poor adaptability of existing systems are solved, and efficient security monitoring and response are achieved.
Patent Information
- Application Number
- CN202411073166.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-26
AI Technical Summary
Existing network security monitoring systems are prone to false alarms, difficult to detect emerging threats, unable to monitor in real time, rely on professional knowledge, poor adaptability, resulting in low efficiency and high cost.
Design a network security monitoring and analysis system based on machine learning, which can identify abnormal IP addresses and traffic through data collection, feature extraction, cleaning and analysis, generate threat indicators, and achieve real-time monitoring and adaptability improvement.
Reduce false alarms, improve security defense capabilities, promptly detect vulnerabilities, reduce risks, improve monitoring efficiency and adaptability, and reduce costs.
Smart Images

Figure CN119030762A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of network security, and in particular to a network security monitoring and analysis system based on machine learning. Background Art
[0002] Machine learning refers to the process of machines learning large amounts of historical data through statistical algorithms, and then using the generated empirical models to guide business.
[0003] Network security monitoring refers to the continuous monitoring of activities in networks and systems, as well as the detection and identification of potential security threats and vulnerabilities; network security monitoring aims to detect and prevent potential network attacks and intrusions in advance, and take necessary response measures in a timely manner.
[0004] There are some common shortcomings in existing network security monitoring and analysis systems, including that many network security monitoring systems generate a large number of false alarms, causing analysts to spend a lot of time and resources to process and investigate these false alarms, thereby reducing the efficiency of the system; traditional network security monitoring systems are mainly based on known threat and vulnerability databases for analysis, and it is difficult to detect and identify emerging security threats and unknown attack technologies; some network security monitoring systems adopt batch processing or static analysis for monitoring, which cannot meet the monitoring needs of real-time network activities, which may result in security threats not being detected or responded to for a considerable period of time; many network security monitoring systems cannot adapt to the ever-changing network environment and attack technologies, which means that their monitoring models and algorithms cannot be automatically adjusted and updated, and it is difficult to effectively respond to emerging security threats; some network security monitoring systems require highly professional personnel to set up and manage, and rely heavily on professional knowledge; this limits the popularity and use of the system and increases the cost and complexity of the enterprise; therefore, analyzing network security monitoring based on machine learning has important theoretical and practical significance.
[0005] How to use machine learning to collect the original data packets of the computer, extract and clean the features of the original data packets, obtain key features and standard data, analyze the key features and standard data, obtain threat information, and issue warnings for the threat information is a problem we need to solve; for this purpose, a network security monitoring and analysis system based on machine learning is now provided. Summary of the invention
[0006] The purpose of the present invention can be achieved through the following technical solutions:
[0007] A network security monitoring and analysis system based on machine learning includes a monitoring center, wherein the monitoring center is connected to a data acquisition module, a data processing module, an analysis module, and a threat warning module;
[0008] The process of the data acquisition module acquiring the original data packets of the computer includes:
[0009] A flow capture point is set, and original data packets of the computer are captured through the flow capture point.
[0010] The data processing module includes a feature extraction unit and a cleaning unit. The feature extraction unit is used to extract features from the original data packets obtained to obtain key features. The key features include IP addresses, the number of data packets, the average value of data packet traffic, the number of data packets of the same communication protocol type, and the proportion of protocol traffic;
[0011] The cleaning unit is used to clean the obtained key features to obtain standard data, and the standard data includes a standard IP address type, a flow sample mean, and a protocol data packet mean.
[0012] The process of extracting features from the original data packet obtained by the extraction unit includes:
[0013] Decomposing the obtained original data packet into a plurality of data fields;
[0014] The data field includes data packet, data packet size, data packet flow, timestamp, IP address, port number, communication protocol and transmission rate;
[0015] Obtaining a data packet mean value according to the obtained data packet size;
[0016] A window time period is set according to the obtained data packet flow and timestamp, and the length of the window time period is marked. Based on the window time period, the data packets passing through are counted to obtain the number of data packets.
[0017] Obtaining a packet flow average value according to the obtained window time period duration, the number of packets, the packet average value, and the packet size;
[0018] According to the obtained communication protocol and port number, the port number corresponding to the communication protocol is obtained, and the communication protocols with the same port number are recorded as the same type of communication protocols;
[0019] Counting the number of data packets of the same communication protocol type, and obtaining the data flow of the same communication protocol type according to the obtained number of data packets of the same communication protocol type and the data packet flow;
[0020] The protocol traffic ratio is obtained based on the data traffic of the same communication protocol type and the communication protocol type.
[0021] The process of the cleaning unit cleaning the obtained key features includes:
[0022] Performing base-based conversion on the obtained IP addresses, mapping the values of the base-based converted IP addresses to the same range, and marking the base-based converted IP addresses as standard IP address types;
[0023] Set the flow normalization range;
[0024] Mark the maximum and minimum values of the obtained data packet flow averages;
[0025] The flow sample mean is obtained according to the obtained data packet flow mean, the maximum value among the data packet flow mean and the minimum value among the data packet flow mean, and the flow sample mean is satisfied to be within the flow normalization range;
[0026] Set the protocol data packet normalization range;
[0027] The maximum and minimum values of the number of data packets of the same communication protocol type obtained are marked, and the mean value of the protocol data packets is obtained according to the number of data packets of the same communication protocol type obtained and the maximum and minimum values of the number of data packets of the same communication protocol type, and the mean value of the protocol data packets is within the normalization range of the protocol data packets.
[0028] The process of analyzing the obtained standard data and obtaining abnormal IP addresses includes:
[0029] Obtaining a standard IP address characteristic value according to the obtained number of data packets of the same communication protocol type, the transmission rate, and the standard IP address, wherein the standard IP address characteristic value includes the transmission rate and the number of data packets of the same communication protocol type;
[0030] Obtaining a rate coefficient according to the obtained transmission rate, setting a first transmission rate threshold and a second transmission rate threshold according to the obtained rate coefficient, and marking the obtained first transmission rate threshold and the second transmission rate threshold respectively;
[0031] Obtain a first quantity threshold and a second quantity threshold according to the obtained protocol data packet mean, and mark the obtained first quantity threshold and second quantity threshold respectively;
[0032] The obtained IP address is traversed through the standard IP address characteristic values. If the transmission rate and the number of data packets corresponding to the IP address simultaneously satisfy the first transmission rate threshold ≤ transmission rate ≤ second transmission rate threshold and the first number threshold ≤ number of data packets ≤ second number threshold, then the obtained IP address is a normal IP address. Otherwise, the obtained IP address is an abnormal IP address.
[0033] The analysis module analyzes the obtained standard data to obtain abnormal threat information, wherein the abnormal threat information includes abnormal IP addresses and abnormal traffic;
[0034] Obtaining a traffic prediction value according to the obtained protocol traffic proportion and the average value of the data packet traffic, and setting an actual traffic value according to the obtained traffic prediction value and the average value of the data packet traffic;
[0035] Obtain a prediction error value according to the obtained flow prediction value and the actual flow value, and obtain an average error according to the obtained prediction error value;
[0036] If the prediction error value satisfies that the prediction error value is greater than the prediction error threshold, the actual flow value corresponding to the prediction error value is abnormal flow.
[0037] The process of the threat warning module warning the acquired abnormal threat information includes:
[0038] Convert the obtained abnormal IP addresses and abnormal traffic into threat indications, upload the obtained threat indications to the monitoring center, build a visualization window based on the obtained threat indications, and control the visualization window to display the abnormal IP addresses and abnormal traffic through the monitoring center;
[0039] According to the obtained abnormal traffic, a window time period of the abnormal traffic and a corresponding communication protocol type are obtained, and the obtained window time period of the abnormal traffic and the corresponding communication protocol type are displayed on a visualization window;
[0040] The visualization window displays the abnormal IP address, abnormal traffic window time period, and corresponding communication protocol type. When the abnormal traffic window time period is reached, a threat alarm is issued through the control center.
[0041] Compared with the prior art, the beneficial effects of the present invention are as follows: by extracting features from the original data packets of the collected computer, the IP address, the number of data packets, the mean value of data packet flow, the number of data packets of the same communication protocol type, and the proportion of protocol flow are obtained; the obtained IP address is converted into binary system and range-planned to obtain the standard IP address type; the obtained data packet flow mean value and the number of data packets of the same communication protocol type are range-planned respectively to obtain the flow sample mean value and the protocol data packet mean value;
[0042] Obtain a standard IP address characteristic value according to the number of data packets of the same communication protocol type, the transmission rate, and the standard IP address, compare the transmission rate and number of data packets of the obtained IP address with the threshold of the standard IP address characteristic value, and obtain an abnormal IP address;
[0043] The traffic prediction value is obtained according to the obtained protocol traffic proportion and the average value of the data packet traffic, the prediction error value is obtained by setting the actual traffic value, the average error is obtained according to the prediction error value, the obtained prediction error value is compared with the prediction error threshold, and the abnormal traffic is obtained;
[0044] Threat warnings are issued for abnormal IP addresses and abnormal traffic, which greatly improves security defense capabilities, timely discovers security holes and weaknesses, reduces security risks, better monitors and responds to network security incidents and threats, improves the adaptability and flexibility of network security monitoring, and effectively improves the work efficiency of security monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0046] Figure 1 It is a schematic diagram of the present invention. DETAILED DESCRIPTION
[0047] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] like Figure 1 As shown, a network security monitoring and analysis system based on machine learning includes a monitoring center, and the monitoring center is connected to a data acquisition module, a data processing module, an analysis module, and a threat warning module;
[0049] The data acquisition module is used to collect the original data packets of the computer, and the specific acquisition process includes:
[0050] Setting a traffic capture point, and capturing original data packets of the computer through the traffic capture point;
[0051] It should be further explained that, in the specific implementation process, the traffic capture point can be set at a network card interface, a virtual network interface, a monitoring port of a router or switch, a proxy server, and a network mirror port;
[0052] The data processing module includes a feature extraction unit and a cleaning unit. The feature extraction unit is used to extract features from the original data packets obtained to obtain key features. The key features include IP addresses, the number of data packets, the average value of data packet traffic, the number of data packets of the same communication protocol type, and the proportion of protocol traffic;
[0053] The cleaning unit is used to clean the obtained key features to obtain standard data, wherein the standard data includes a standard IP address type, a flow sample mean, and a protocol data packet mean;
[0054] The process of extracting features from the original data packet obtained by the extraction unit includes:
[0055] Decomposing the obtained original data packet into a plurality of data fields and data information corresponding to the data fields;
[0056] It should be further explained that, in the specific implementation process, the data information is the content corresponding to the data field. For example, if the data field is the HTTP port number, the corresponding data information is 80;
[0057] The data field includes data packet, data packet size, data packet flow, timestamp, IP address, port number, communication protocol and transmission rate; the IP address includes source IP address and destination IP address; it should be further explained that, in the specific implementation process, the unit conversion is performed on the obtained data packet size, and the unit of the obtained data packet size is converted into bytes;
[0058] Mark the obtained data packet as i, where i=1, 2, 3, ..., n1, and n1 is a positive integer;
[0059] The obtained data packet size, data packet flow, and transmission rate are marked as DXi, Li, and CSi respectively;
[0060] The mean value of the data packets is obtained according to the obtained data packet size, and the obtained data packet mean value is marked as in And f1, f2, ..., f i All are weight coefficients;
[0061] A window time period is set according to the obtained data packet flow and timestamp, and the obtained window time period is marked as j, where j=1, 2, 3, ..., n2, and n2 is a positive integer; the length of the window time period is recorded as Tj, and the data packets passing through are counted based on the window time period to obtain the number of data packets, and the number of obtained data packets is marked as aj;
[0062] The average value of packet flow is obtained based on the obtained window time period, the number of packets, the average value of packets, and the size of packets. The obtained average value of packet flow is marked as in
[0063] According to the obtained communication protocol and port number, the port number corresponding to the communication protocol is obtained, the communication protocols with the same port number are recorded as the same type of communication protocol, and the obtained communication protocol type is marked as k, where k=1, 2, 3, ..., n3, n3 is a positive integer; for example, the communication protocol types include HTTP, TCP, UDP, DNS, etc.;
[0064] Count the number of data packets of the same communication protocol type, and mark the obtained number of data packets of the same communication protocol type as km, where m=1, 2, 3, ..., n4, and n4 is a positive integer;
[0065] The data flow of the same communication protocol type is obtained according to the number of data packets of the same communication protocol type and the data packet flow, and the obtained data flow of the same communication protocol type is marked as TXk, where
[0066] The protocol flow ratio is obtained according to the obtained data flow of the same communication protocol type and the communication protocol type, and the obtained protocol flow ratio is marked as XZk, where
[0067] It should be further explained that, in the specific implementation process, the protocol traffic ratio is expressed as a percentage. For example, if the traffic of the HTTP protocol is 1000MB and the total traffic is 5000MB, the traffic ratio of the HTTP protocol is 1000MB / 5000MB=0.2, i.e. 20%;
[0068] The obtained key features are uploaded to a cleaning unit to clean the key features. The cleaning process of the cleaning unit for cleaning the obtained key features includes:
[0069] The obtained IP address is converted into a binary system; it should be further explained that, in the specific implementation process, there may be conflicts between IPv4 and IPv6 addresses for the collected IP addresses, resulting in the misuse of IPv4 addresses; therefore, when cleaning the obtained IP address data, it is necessary to identify and distinguish IPv4 and IPv6 addresses. For IPv4 addresses, they are used directly, and for IPv6 addresses, they are converted, such as binary to decimal;
[0070] The IP addresses and data packet flow averages after base conversion are range planned respectively. The range planning for the IP addresses after base conversion is to map the values of the IP addresses after base conversion into the same range, mark the obtained same range as [h1, h2], and mark the IP addresses after base conversion as standard IP address types;
[0071] For example, map the values of IPv4 and IPv6 addresses to the same range, such as [0, 255];
[0072] The process of performing range planning on the mean value of data packet flow includes:
[0073] Set the flow normalization range to [e1, g1], where e1≥-1, g1≤1;
[0074] The maximum value among the obtained packet flow averages is marked as The minimum value among the obtained packet flow means is marked as
[0075] The traffic sample mean is obtained according to the obtained data packet flow mean, the maximum value among the data packet flow mean and the minimum value among the data packet flow mean, and the obtained traffic sample mean is marked as in And meet Within the flow normalization range;
[0076] Similarly, the protocol data packet normalization range is set to [e2, g2], where e2 ≥ -1, g2 ≤ 1;
[0077] The maximum and minimum values of the number of packets of the same communication protocol type are marked as km large and km small respectively. The mean value of the protocol packets is obtained according to the number of packets of the same communication protocol type, the maximum and minimum values of the number of packets of the same communication protocol type, and the mean value of the protocol packets is marked as in And meet Within the protocol data packet normalization range;
[0078] The analysis module is used to analyze the obtained standard data to obtain abnormal threat information, where the abnormal threat information includes abnormal IP addresses and abnormal traffic. The specific process includes:
[0079] Obtaining a standard IP address characteristic value according to the obtained number of data packets of the same communication protocol type, the transmission rate, and the standard IP address, wherein the standard IP address characteristic value includes the transmission rate and the number of data packets of the same communication protocol type;
[0080] A rate coefficient is obtained according to the obtained transmission rate, and the obtained rate coefficient is marked as SL, where A first transmission rate threshold and a second transmission rate threshold are set according to the obtained rate coefficient, and the obtained first transmission rate threshold and the second transmission rate threshold are marked as Z1 and Z2 respectively, wherein Z1=SL*α1, Z2=SL*α2, and 0<α1<1, 1<α2<2;
[0081] The first quantity threshold and the second quantity threshold are obtained according to the obtained protocol data packet mean, and the obtained first quantity threshold and the second quantity threshold are marked as S1 and S2 respectively, where And 0<β1<1, 1<β2<2;
[0082] The obtained IP address is traversed through the standard IP address characteristic values, and the transmission rate and the number of data packets corresponding to the obtained IP address are marked as Y0 and S0. If the transmission rate and the number of data packets corresponding to the IP address satisfy Y1≤Y0≤Y2 and S1≤S0≤S2 at the same time, the obtained IP address is a normal IP address, otherwise, the obtained IP address is an abnormal IP address;
[0083] The traffic prediction value is obtained according to the obtained protocol traffic proportion and the mean value of the data packet traffic, and the obtained traffic prediction value is marked as Yj-k, where Yj-k represents the kth communication protocol type corresponding to the jth window time period. b0, b1, and b2 are flow prediction coefficients;
[0084] The actual traffic value is set according to the obtained traffic prediction value and the average traffic value of the data packet, and the obtained actual traffic value is marked as Y1j-k, where
[0085] A prediction error value is obtained according to the obtained flow prediction value and the actual flow value, and the obtained prediction error value is marked as ΔYj-k, where ΔYj-k=Y1j-k-Yj-k;
[0086] The average error is obtained according to the obtained prediction error value, and the obtained average error is marked as ΔY', where Where c is the number of prediction error values;
[0087] Set the prediction error threshold, denoted as ΔY0. If the prediction error value satisfies ΔYj-k>ΔY0, where Then the actual flow value corresponding to the predicted error value is the abnormal flow;
[0088] The threat warning module is used to warn of the acquired abnormal threat information, and the specific process includes:
[0089] Convert the obtained abnormal IP addresses and abnormal traffic into threat indications, upload the obtained threat indications to the monitoring center, build a visualization window based on the obtained threat indications, and control the visualization window to display the abnormal IP addresses and abnormal traffic through the monitoring center;
[0090] According to the obtained abnormal traffic, a window time period of the abnormal traffic and a corresponding communication protocol type are obtained, and the obtained window time period of the abnormal traffic and the corresponding communication protocol type are displayed on a visualization window;
[0091] The abnormal IP address, abnormal traffic window time period and corresponding communication protocol type are displayed in the visualization window. When the abnormal traffic window time period is reached, the visualization window is displayed through the control center to issue a threat warning.
[0092] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. A network security monitoring and analysis system based on machine learning, comprising a monitoring center, characterized in that: The monitoring center is connected to a data acquisition module, a data processing module, an analysis module and a threat warning module; The data acquisition module is used to collect original data packets from the computer; The data processing module is used to process the obtained original data packets to obtain key features and standard data; The analysis module is used to analyze the obtained key features and standard data to obtain abnormal threat information; The threat warning module is used to generate corresponding security warnings according to the obtained abnormal threat information.
2. A network security monitoring and analysis system based on machine learning according to claim 1, characterized in that: The data processing module includes a feature extraction unit and a cleaning unit, wherein the feature extraction unit is used to extract features from the obtained original data packets to obtain key features, wherein the key features include IP addresses, the number of data packets, the average value of data packet traffic, the number of data packets of the same communication protocol type, and the proportion of protocol traffic; The cleaning unit is used to clean the obtained key features to obtain standard data, and the standard data includes a standard IP address type, a flow sample mean, and a protocol data packet mean.
3. A network security monitoring and analysis system based on machine learning according to claim 2, characterized in that: The process of extracting features from the original data packet obtained by the extraction unit includes: Decomposing the obtained original data packet into a plurality of data fields; The data field includes data packet, data packet size, data packet flow, timestamp, IP address, port number, communication protocol and transmission rate; Obtaining a data packet mean value according to the obtained data packet size; A window time period is set according to the obtained data packet flow and timestamp, and the length of the window time period is marked. Based on the window time period, the data packets passing through are counted to obtain the number of data packets. Obtaining a packet flow average value according to the obtained window time period duration, the number of packets, the packet average value, and the packet size; According to the obtained communication protocol and port number, the port number corresponding to the communication protocol is obtained, and the communication protocols with the same port number are recorded as the same type of communication protocols; Counting the number of data packets of the same communication protocol type, and obtaining the data flow of the same communication protocol type according to the obtained number of data packets of the same communication protocol type and the data packet flow; The protocol traffic ratio is obtained based on the data traffic of the same communication protocol type and the communication protocol type.
4. A network security monitoring and analysis system based on machine learning according to claim 3, characterized in that: The process of the cleaning unit cleaning the obtained key features includes: Performing base-to-base conversion on the obtained IP addresses, mapping the values of the base-to-base converted IP addresses to the same range, and marking the base-to-base converted IP addresses as standard IP address types; Set the flow normalization range; Mark the maximum and minimum values of the obtained data packet flow averages; The flow sample mean is obtained according to the obtained data packet flow mean, the maximum value among the data packet flow mean and the minimum value among the data packet flow mean, and the flow sample mean is satisfied to be within the flow normalization range; Set the protocol data packet normalization range; The maximum and minimum values of the number of data packets of the same communication protocol type obtained are marked, and the mean value of the protocol data packets is obtained according to the number of data packets of the same communication protocol type obtained and the maximum and minimum values of the number of data packets of the same communication protocol type, and the mean value of the protocol data packets is within the normalization range of the protocol data packets.
5. A network security monitoring and analysis system based on machine learning according to claim 4, characterized in that: The process of analyzing the obtained standard data and obtaining abnormal IP addresses includes: Obtaining a standard IP address characteristic value according to the obtained number of data packets of the same communication protocol type, the transmission rate, and the standard IP address, wherein the standard IP address characteristic value includes the transmission rate and the number of data packets of the same communication protocol type; Obtaining a rate coefficient according to the obtained transmission rate, setting a first transmission rate threshold and a second transmission rate threshold according to the obtained rate coefficient, and marking the obtained first transmission rate threshold and the second transmission rate threshold respectively; Obtain a first quantity threshold and a second quantity threshold according to the obtained protocol data packet average, and mark the obtained first quantity threshold and second quantity threshold respectively; The obtained IP address is traversed through the standard IP address characteristic values. If the transmission rate and the number of data packets corresponding to the IP address simultaneously satisfy the first transmission rate threshold ≤ transmission rate ≤ second transmission rate threshold and the first number threshold ≤ number of data packets ≤ second number threshold, then the obtained IP address is a normal IP address. Otherwise, the obtained IP address is an abnormal IP address.
6. A network security monitoring and analysis system based on machine learning according to claim 5, characterized in that: The analysis module analyzes the obtained standard data to obtain abnormal threat information, wherein the abnormal threat information includes abnormal IP addresses and abnormal traffic; A traffic prediction value is obtained according to the obtained protocol traffic proportion and the average value of the data packet traffic, and an actual traffic value is set according to the obtained traffic prediction value and the average value of the data packet traffic; Obtain a prediction error value according to the obtained flow prediction value and the actual flow value, and obtain an average error according to the obtained prediction error value; If the prediction error value satisfies that the prediction error value is greater than the prediction error threshold, the actual flow value corresponding to the prediction error value is abnormal flow.
7. A network security monitoring and analysis system based on machine learning according to claim 6, characterized in that: The process of the threat warning module warning the acquired abnormal threat information includes: Convert the obtained abnormal IP addresses and abnormal traffic into threat indications, upload the obtained threat indications to the monitoring center, build a visualization window based on the obtained threat indications, and control the visualization window to display the abnormal IP addresses and abnormal traffic through the monitoring center; According to the obtained abnormal traffic, a window time period of the abnormal traffic and a corresponding communication protocol type are obtained, and the obtained window time period of the abnormal traffic and the corresponding communication protocol type are displayed on a visualization window; The visualization window displays the abnormal IP address, abnormal traffic window time period, and corresponding communication protocol type. When the abnormal traffic window time period is reached, a threat alarm is issued through the control center.
