Network security incident detection method and system based on machine learning

Abstract

The present invention belongs to the field of network security technology, and specifically discloses a network security incident detection method and system based on machine learning. By collecting network traffic data of the same network data stream for feature extraction and classification prediction, the classification real value corresponding to each eigenvector is determined, and then each eigenvector and its corresponding classification real value are used for mapping calculation to determine the classification mapping value corresponding to each eigenvector. Finally, a comprehensive early warning assessment is performed based on the classification mapping value corresponding to each eigenvector to determine whether a network security incident has occurred, and then a network security early warning is performed. The present invention uses a network data security assessment method that integrates the classification mapping results of multiple eigenvectors to more accurately assess the security status of network data and reduce the false alarm rate of network security monitoring. Moreover, through a specific feature classification result mapping calculation and comprehensive assessment method, the timeliness and scalability of network security monitoring can be improved, and it can adapt to large-scale network environments.

Description

Technical Field

[0001] The present invention belongs to the field of network security technology, and specifically relates to a network security incident detection method and system based on machine learning. Background Art

[0002] The development of network technology has brought great convenience to people's lives and work, but illegal network access and malicious attacks also pose a serious threat to network security. Therefore, the supervision of malicious network behavior is crucial. However, current security detection of malicious network behavior mainly relies on signature libraries and rule matching methods. It lacks the means to dynamically detect and process traffic interaction information, and lacks real-time and dynamic response capabilities. In large-scale and complex network environments, it is difficult to detect potential network attacks in a timely and accurate manner, resulting in insufficient monitoring and judgment capabilities for network access and traffic-related behavior. Summary of the Invention

[0003] The purpose of the present invention is to provide a network security incident detection method and system based on machine learning to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] First, a network security incident detection method based on machine learning is provided, including:

[0006] Collect the original network traffic data of the same network data flow;

[0007] Preprocess each original network traffic data to obtain corresponding normalized data;

[0008] Perform feature extraction on each normalized data after preprocessing to obtain the corresponding feature vector;

[0009] Input each eigenvector into a preset classification model to perform eigenvector classification processing to obtain the corresponding classification real value;

[0010] Calculate the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value;

[0011] The comprehensive early warning assessment parameters are calculated using the classification mapping values ​​corresponding to each eigenvector;

[0012] Determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

[0013] In one possible design, the preprocessing of each raw network traffic data to obtain corresponding normalized data includes:

[0014] Perform data cleaning on each original network traffic data to obtain cleaned network traffic data;

[0015] The cleaned network traffic data is normalized to obtain the corresponding normalized data.

[0016] In one possible design, the data cleaning of each raw network traffic data includes:

[0017] Deleting duplicate data from the original network traffic data and converting each character data in the deduplicated original network traffic data into corresponding numerical data;

[0018] The interquartile range method is used to identify outliers in numerical data, and outliers in numerical data are deleted to obtain cleaned network traffic data.

[0019] In a possible design, the feature extraction is performed on each normalized data after preprocessing to obtain a corresponding feature vector, including:

[0020] The principal component analysis method is used to extract features from the preprocessed normalized data to obtain the corresponding feature vectors.

[0021] In one possible design, before inputting each feature vector into a preset classification model for feature vector classification processing, the method further includes:

[0022] An initialized support vector machine model is constructed, and the initialized support vector machine model is trained using a training set until the set training conditions are met to obtain a trained classification model, wherein the training set contains feature vector samples, and each feature vector sample is marked with a corresponding classification label.

[0023] In one possible design, calculating the classification mapping value corresponding to each eigenvector based on each eigenvector and its corresponding classification real value includes:

[0024] Substitute each eigenvector and its corresponding classification real value into the preset classification mapping formula for calculation to obtain the classification mapping value corresponding to each eigenvector. The classification mapping formula is:

[0025]

[0026] Among them, i represents the eigenvector number, X i Characterize the eigenvector numbered i, Z i Characterizing the feature vector X i The corresponding classification mapping value, Y i Characterizing the feature vector X i The corresponding categorical real value, X ois the set sample vector, σ is the set characteristic length scale parameter, b is the set bias parameter, Sigmoid represents the Sigmoid function operation, exp represents the natural exponential function operation, and ‖‖ represents the norm operation.

[0027] In one possible design, the calculation of comprehensive early warning assessment parameters using the classification mapping values ​​corresponding to each eigenvector includes:

[0028] Extract the largest values ​​from the classification mapping values ​​corresponding to each eigenvector as evaluation values;

[0029] The average value of all evaluation values ​​is used as the comprehensive early warning evaluation parameter.

[0030] In one possible design, determining whether a cybersecurity incident has occurred based on comprehensive early warning assessment parameters includes:

[0031] When the comprehensive early warning assessment parameter is greater than the set security threshold, it is determined that a network security incident has occurred; otherwise, it is determined that no network security incident has occurred.

[0032] In a second aspect, a network security incident detection system based on machine learning is provided, comprising a data acquisition unit, a data processing unit, a feature extraction unit, a classification processing unit, a mapping calculation unit, an evaluation calculation unit, and a detection and determination unit, wherein:

[0033] A data collection unit, used for collecting original network flow data of the same network data flow;

[0034] A data processing unit, used to pre-process each original network flow data to obtain corresponding normalized data;

[0035] A feature extraction unit is used to extract features from each normalized data after preprocessing to obtain corresponding feature vectors;

[0036] A classification processing unit is used to input each feature vector into a preset classification model to perform feature vector classification processing to obtain a corresponding classification real value;

[0037] A mapping calculation unit, configured to calculate a classification mapping value corresponding to each eigenvector based on each eigenvector and its corresponding classification real value;

[0038] An evaluation calculation unit, used to calculate comprehensive warning evaluation parameters using the classification mapping values ​​corresponding to each eigenvector;

[0039] The detection and judgment unit is used to determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

[0040] Third, we provide a network security incident detection system based on machine learning, including:

[0041] a memory for storing instructions;

[0042] A processor is used to read the instructions stored in the memory and execute any one of the methods described in the first aspect according to the instructions.

[0043] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform any one of the methods described in the first aspect. Furthermore, a computer program product is provided that, when executed on a computer, performs any one of the methods described in the first aspect.

[0044] Beneficial effects: The present invention collects network traffic data of the same network data stream for feature extraction and classification prediction, determines the classification real value corresponding to each eigenvector, then uses each eigenvector and its corresponding classification real value to perform mapping calculation, determines the classification mapping value corresponding to each eigenvector, and finally performs a comprehensive early warning assessment based on the classification mapping value corresponding to each eigenvector to determine whether a network security incident has occurred, and then performs a network security early warning, achieving fast and efficient network security incident monitoring and early warning. The present invention uses a network data security assessment method that integrates the classification mapping results of multiple eigenvectors to more accurately assess the security status of network data, improve generalization capabilities, and reduce the false alarm rate of network security monitoring. Moreover, through specific feature classification result mapping calculations and comprehensive assessment methods, it can improve the timeliness and scalability of network security monitoring and adapt to large-scale network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 Schematic diagram of the steps of the method in Example 1 of the present invention;

[0047] Figure 2 Schematic diagram of the system structure in Example 2 of the present invention;

[0048] Figure 3 Schematic diagram of the hardware system in Example 3 of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the description of these embodiments is intended to help understand the present invention, but does not constitute a limitation of the present invention. The specific structural and functional details disclosed herein are merely intended to describe exemplary embodiments of the present invention. However, the present invention may be embodied in a variety of alternative forms, and should not be construed as being limited to the embodiments set forth herein.

[0050] It should be understood that, unless otherwise expressly specified or limited, the corresponding terms should be understood in a broad sense. For example, "connection" can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to an electrical connection, a direct connection, an indirect connection through an intermediate medium, or internal communication between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the embodiments based on specific circumstances.

[0051] In the following description, certain details are provided to facilitate a thorough understanding of the example embodiments. However, one skilled in the art will appreciate that the example embodiments may be practiced without these specific details. For example, devices may be shown in block diagrams to avoid obscuring the examples with unnecessary detail. In other embodiments, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0052] Example 1:

[0053] This embodiment provides a network security event detection method based on machine learning, which can be applied to corresponding network security monitoring terminals, such as Figure 1 As shown, the method includes the following steps:

[0054] S1. Collect the original network traffic data of the same network data flow.

[0055] During implementation, the network security monitoring terminal collects raw network traffic data from the same network data stream in the corresponding channel in real time. This raw network traffic data includes data packets (data packets contain the data to be transmitted and some control information. Each data packet has a source address and a destination address to indicate its origin and destination), protocol information (network traffic is controlled by different protocols, such as TCP / IP, UDP, HTTP, HTTPS, FTP, etc., which define the structure and transmission method of data packets), port information (network traffic usually uses specific port numbers to identify different services or applications), session information (a group of data packets transmitted in the network can constitute a session, representing a communication process from start to finish), and traffic statistics (including the number of unique visitors, the number of repeat visitors, the number of page views, the number of page views per visitor, and statistical indicators of specific files / pages, such as the number of page displays and file downloads).

[0056] S2. Preprocess each original network traffic data to obtain corresponding normalized data.

[0057] In specific implementation, the network security monitoring terminal first cleans the raw network traffic data to obtain cleaned network traffic data. The data cleaning process includes deleting duplicate data from the raw network traffic data and converting each character data in the deduplicated raw network traffic data into corresponding numeric data, such as through one-hot encoding. The interquartile range method is then used to identify and remove outliers from the numeric data, resulting in cleaned network traffic data. The interquartile range (IQR) method calculates the interquartile range (IQR) of each numeric data point (IQR = Q3 - Q1), where Q3 is the third quartile and Q1 is the first quartile. The outlier boundary value (OB) is determined as Q3 - k·IQR, where k is the set outlier range adjustment coefficient. If a numeric data point exceeds the outlier boundary value (OB), it is considered an outlier. The cleaned network traffic data is then normalized to obtain the corresponding normalized data.

[0058] S3. Perform feature extraction on each normalized data after preprocessing to obtain the corresponding feature vector.

[0059] In practice, principal component analysis (PCA) can be used to extract features from the preprocessed, normalized data to obtain corresponding feature vectors. The PCA algorithm can better process high-dimensional, nonlinear network traffic data and extract more discriminative features.

[0060] S4. Input each eigenvector into a preset classification model to perform eigenvector classification processing to obtain the corresponding classification real value.

[0061] In specific implementation, before inputting each feature vector into a pre-set classification model for feature vector classification, the network security monitoring terminal pre-builds an initialized support vector machine (SVM) model. This model is trained using a training set until the pre-set training conditions are met, resulting in a trained classification model. The training set contains feature vector samples, each labeled with a corresponding classification label. During application, the network security monitoring terminal inputs each feature vector into the classification model for feature vector classification prediction and outputs the corresponding classification prediction result, i.e., a classification real value, which is a real number in the interval [-1, 1].

[0062] S5. Calculate the classification mapping value corresponding to each eigenvector based on each eigenvector and its corresponding classification real value.

[0063] In specific implementation, the network security monitoring terminal substitutes each eigenvector and its corresponding classification real value into a preset classification mapping formula for calculation to obtain the classification mapping value corresponding to each eigenvector. The classification mapping formula is:

[0064]

[0065] Among them, i represents the eigenvector number, X i Characterize the eigenvector numbered i, Z i Characterizing the feature vector X i The corresponding classification mapping value, Y i Characterizing the feature vector X i The corresponding categorical real value, X o is the set sample vector, σ is the set characteristic length scale parameter, b is the set bias parameter, Sigmoid represents the Sigmoid function operation, exp represents the natural exponential function operation, and ‖‖ represents the norm operation.

[0066] S6. Calculate comprehensive warning assessment parameters using the classification mapping values ​​corresponding to each eigenvector.

[0067] In specific implementation, the network security monitoring terminal first extracts several largest values ​​from the classification mapping values ​​corresponding to each feature vector as evaluation values, and then uses the average value of all evaluation values ​​as the comprehensive warning evaluation parameter.

[0068] S7. Determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

[0069] In specific implementation, when the comprehensive early warning assessment parameter exceeds the set security threshold, the network security monitoring terminal determines that a network security incident has occurred. If the comprehensive early warning assessment parameter does not exceed the set security threshold, the network security monitoring terminal determines that no network security incident has occurred. When a network security incident is determined to have occurred, the network security monitoring terminal outputs the corresponding network security warning information to issue a network security incident warning.

[0070] This method uses a network data security assessment method that integrates the classification mapping results of multiple feature vectors to more accurately assess the security status of network data, improve generalization capabilities, and reduce the false alarm rate of network security monitoring. In addition, through specific feature classification result mapping calculation and comprehensive evaluation methods, it can improve the timeliness and scalability of network security monitoring and adapt to large-scale network environments.

[0071] Example 2:

[0072] This embodiment provides a network security incident detection system based on machine learning, such as Figure 2As shown, it includes a data acquisition unit, a data processing unit, a feature extraction unit, a classification processing unit, a mapping calculation unit, an evaluation calculation unit and a detection and judgment unit, wherein:

[0073] A data collection unit, used for collecting original network flow data of the same network data flow;

[0074] A data processing unit, used to pre-process each original network flow data to obtain corresponding normalized data;

[0075] A feature extraction unit is used to extract features from each normalized data after preprocessing to obtain corresponding feature vectors;

[0076] A classification processing unit is used to input each feature vector into a preset classification model to perform feature vector classification processing to obtain a corresponding classification real value;

[0077] A mapping calculation unit, configured to calculate a classification mapping value corresponding to each eigenvector based on each eigenvector and its corresponding classification real value;

[0078] An evaluation calculation unit, used to calculate comprehensive warning evaluation parameters using the classification mapping values ​​corresponding to each eigenvector;

[0079] The detection and judgment unit is used to determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

[0080] Example 3:

[0081] This embodiment provides a network security incident detection system based on machine learning, such as Figure 3 As shown, at the hardware level, it includes:

[0082] Data interface, used to establish data connection between the processor and external data source;

[0083] a memory for storing instructions;

[0084] The processor is used to read the instructions stored in the memory and execute the network security incident detection method in Example 1 according to the instructions.

[0085] Optionally, the system further includes an internal bus, through which the processor, memory, and data interface can be interconnected. The internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0086] The memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out memory (FIFO) and / or first-in-last-out memory (FILO), etc. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0087] Example 4:

[0088] This embodiment provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the computer is caused to execute the network security incident detection method of Embodiment 1. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0089] This embodiment further provides a computer program product, which, when executed on a computer, executes the network security incident detection method of embodiment 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0090] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A network security incident detection method based on machine learning, characterized in that: include: Collect the original network traffic data of the same network data flow; Preprocess each original network traffic data to obtain corresponding normalized data; Perform feature extraction on each normalized data after preprocessing to obtain the corresponding feature vector; Input each eigenvector into the preset classification model to perform eigenvector classification prediction, and output the corresponding classification prediction result value, that is, the classification real value, which is a real number in the interval [-1, 1]; Calculating the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value, including: substituting each eigenvector and its corresponding classification real value into a preset classification mapping formula for calculation to obtain the classification mapping value corresponding to each eigenvector, wherein the classification mapping formula is: Among them, i represents the eigenvector number, X i Characterize the eigenvector numbered i, Z i Characterizing the feature vector X i The corresponding classification mapping value, Y i Characterizing the feature vector X i The corresponding categorical real value, X o is the set sample vector, σ is the set characteristic length scale parameter, b is the set bias parameter, Sigmoid represents the Sigmoid function operation, exp represents the natural exponential function operation, and ‖‖ represents the norm operation; The comprehensive early warning assessment parameters are calculated using the classification mapping values ​​corresponding to each eigenvector; Determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

2. The network security incident detection method based on machine learning according to claim 1 is characterized in that: The preprocessing of each original network traffic data to obtain corresponding normalized data includes: Perform data cleaning on each original network traffic data to obtain cleaned network traffic data; The cleaned network traffic data is normalized to obtain the corresponding normalized data.

3. The network security incident detection method based on machine learning according to claim 2 is characterized in that: The data cleaning of each original network traffic data includes: Deleting duplicate data from the original network traffic data and converting each character data in the deduplicated original network traffic data into corresponding numerical data; The interquartile range method is used to identify outliers in numerical data, and outliers in numerical data are deleted to obtain cleaned network traffic data.

4. The network security incident detection method based on machine learning according to claim 1 is characterized in that: The feature extraction of each pre-processed normalized data to obtain a corresponding feature vector includes: The principal component analysis method is used to extract features from the preprocessed normalized data to obtain the corresponding feature vectors.

5. The network security incident detection method based on machine learning according to claim 1 is characterized in that: Before inputting each feature vector into a preset classification model for feature vector classification processing, the method further includes: An initialized support vector machine model is constructed, and the initialized support vector machine model is trained using a training set until the set training conditions are met to obtain a trained classification model, wherein the training set contains feature vector samples, and each feature vector sample is marked with a corresponding classification label.

6. The network security incident detection method based on machine learning according to claim 1 is characterized in that: The calculation of comprehensive early warning assessment parameters using the classification mapping values ​​corresponding to each eigenvector includes: Extract the largest values ​​from the classification mapping values ​​corresponding to each eigenvector as evaluation values; The average value of all evaluation values ​​is used as the comprehensive early warning evaluation parameter.

7. The network security incident detection method based on machine learning according to claim 1 is characterized in that: Determining whether a cybersecurity incident has occurred based on comprehensive early warning assessment parameters includes: When the comprehensive early warning assessment parameter is greater than the set security threshold, it is determined that a network security incident has occurred; otherwise, it is determined that no network security incident has occurred.

8. A network security incident detection system based on machine learning, characterized in that: It includes a data acquisition unit, a data processing unit, a feature extraction unit, a classification processing unit, a mapping calculation unit, an evaluation calculation unit and a detection and judgment unit, wherein: A data collection unit, used for collecting original network flow data of the same network data flow; A data processing unit, used to pre-process each original network flow data to obtain corresponding normalized data; A feature extraction unit is used to extract features from each normalized data after preprocessing to obtain corresponding feature vectors; The classification processing unit is used to input each feature vector into a preset classification model to perform feature vector classification prediction and output the corresponding classification prediction result value, that is, the classification real value, which is a real number in the interval [-1, 1]; A mapping calculation unit is used to calculate the classification mapping value corresponding to each eigenvector based on each eigenvector and its corresponding classification real value, including: substituting each eigenvector and its corresponding classification real value into a preset classification mapping formula for calculation to obtain the classification mapping value corresponding to each eigenvector, wherein the classification mapping formula is: Among them, i represents the eigenvector number, X i Characterize the eigenvector numbered i, Z i Characterizing the feature vector X i The corresponding classification mapping value, Y i Characterizing the feature vector X i The corresponding categorical real value, X o is the set sample vector, σ is the set characteristic length scale parameter, b is the set bias parameter, Sigmoid represents the Sigmoid function operation, exp represents the natural exponential function operation, and ‖‖ represents the norm operation; An evaluation calculation unit, used to calculate comprehensive warning evaluation parameters using the classification mapping values ​​corresponding to each eigenvector; The detection and judgment unit is used to determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

9. A network security incident detection system based on machine learning, characterized in that: include: a memory for storing instructions; A processor is used to read the instructions stored in the memory and execute the network security incident detection method described in any one of claims 1 to 7 according to the instructions.

Images