Network security event detection method and system based on machine learning

Abstract

The invention belongs to the technical field of network security, and particularly discloses a network security event detection method and system based on machine learning, and the method comprises the steps: carrying out the feature extraction and classification prediction through collecting network flow data of the same network data flow, and determining a classification real value corresponding to each feature vector; the method comprises the following steps: firstly, carrying out classification on a plurality of feature vectors, then carrying out mapping calculation by utilizing each feature vector and a corresponding classification real value, determining a classification mapping value corresponding to each feature vector, and finally, carrying out comprehensive early warning evaluation based on the classification mapping value corresponding to each feature vector to judge whether a network security event occurs or not so as to carry out network security early warning. According to the network data security assessment method, the security condition of the network data can be assessed more accurately through a network data security assessment mode of fusion of multi-feature vector classification mapping results, the false alarm rate of network security monitoring is reduced, and the network security monitoring accuracy is improved through a specific feature classification result mapping calculation and comprehensive assessment mode. The timeliness and expandability of network security monitoring can be improved, and the method can adapt to a large-scale network environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to a network security event detection method and system based on machine learning. Background Art

[0002] The development of network technology has brought great convenience to people's lives and work, but illegal network access and malicious attacks are also seriously threatening network security. Therefore, supervision of network malicious behaviors is crucial. At present, security detection of network malicious behaviors mainly relies on feature libraries and rule matching methods, lacking means to dynamically detect and process traffic interaction information, and lacking real-time and dynamic response capabilities. In the face of large-scale complex network environments, it is difficult to timely and accurately discover potential network attack behaviors, resulting in insufficient monitoring and judgment capabilities for network access and traffic-related behaviors. Summary of the invention

[0003] The purpose of the present invention is to provide a network security incident detection method and system based on machine learning to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: First, a network security incident detection method based on machine learning is provided, including: Collect the original network traffic data of the same network data flow; Preprocess each original network traffic data to obtain corresponding normalized data; Perform feature extraction on each normalized data after preprocessing to obtain the corresponding feature vector; Input each feature vector into a preset classification model to perform feature vector classification processing to obtain a corresponding classification real value; Calculate the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value; The comprehensive early warning assessment parameters are calculated using the classification mapping values ​​corresponding to each eigenvector; Determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

[0005] In a possible design, the preprocessing of each original network traffic data to obtain corresponding normalized data includes: Perform data cleaning on each original network flow data to obtain cleaned network flow data; The cleaned network traffic data is normalized to obtain the corresponding normalized data.

[0006] In a possible design, the data cleaning of each raw network traffic data includes: Deleting duplicate data in the original network traffic data, and converting each character data in the original network traffic data after deduplication into corresponding numerical data; The interquartile range method is used to identify outliers in numerical data, and outliers in numerical data are deleted to obtain cleaned network traffic data.

[0007] In a possible design, the feature extraction is performed on each preprocessed normalized data to obtain a corresponding feature vector, including: The principal component analysis method is used to extract features from the preprocessed normalized data to obtain the corresponding feature vectors.

[0008] In a possible design, before inputting each feature vector into a preset classification model for feature vector classification processing, the method further includes: An initialized support vector machine model is constructed, and the initialized support vector machine model is trained using a training set until the set training conditions are met to obtain a trained classification model, wherein the training set includes feature vector samples, and each feature vector sample is annotated with a corresponding classification label.

[0009] In a possible design, the step of calculating the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value includes: Substitute each eigenvector and its corresponding classification real value into the preset classification mapping formula for calculation to obtain the classification mapping value corresponding to each eigenvector. The classification mapping formula is:

[0010] Among them, i represents the feature vector number, X i Characterizes the eigenvector numbered i, Z i Characterizing the feature vector X i The corresponding classification mapping value, Y i Characterizing the feature vector X i The corresponding categorical real value, X o is the set sample vector, σ is the set characteristic length scale parameter, b is the set bias parameter, Sigmoid represents the Sigmoid function operation, exp represents the natural exponential function operation, and ‖‖ represents the norm operation.

[0011] In a possible design, the calculation of the comprehensive warning assessment parameter using the classification mapping value corresponding to each feature vector includes: Extract the largest values ​​from the classification mapping values ​​corresponding to each feature vector as evaluation values; The average of all evaluation values ​​is taken as the comprehensive early warning evaluation parameter.

[0012] In a possible design, judging whether a network security incident occurs according to the comprehensive early warning evaluation parameters includes: When the comprehensive early warning assessment parameter is greater than the set security threshold, it is determined that a network security incident has occurred; otherwise, it is determined that no network security incident has occurred.

[0013] In a second aspect, a network security incident detection system based on machine learning is provided, including a data acquisition unit, a data processing unit, a feature extraction unit, a classification processing unit, a mapping calculation unit, an evaluation calculation unit and a detection and determination unit, wherein: A data collection unit, used for collecting each original network flow data of the same network data flow; A data processing unit, used for preprocessing each original network flow data to obtain corresponding normalized data; A feature extraction unit is used to extract features from each normalized data after preprocessing to obtain a corresponding feature vector; A classification processing unit, used to input each feature vector into a preset classification model to perform feature vector classification processing to obtain a corresponding classification real value; A mapping calculation unit, used for calculating the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value; An evaluation calculation unit, used to calculate comprehensive warning evaluation parameters using the classification mapping values ​​corresponding to each feature vector; The detection and determination unit is used to determine whether a network security incident has occurred based on the comprehensive warning evaluation parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

[0014] Third, a network security incident detection system based on machine learning is provided, including: A memory for storing instructions; A processor is used to read the instructions stored in the memory and execute any one of the methods described in the first aspect according to the instructions.

[0015] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored on the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes any one of the methods described in the first aspect. In addition, a computer program product is provided, and when the computer program product is executed on a computer, the computer executes any one of the methods described in the first aspect.

[0016] Beneficial effects: The present invention collects network traffic data of the same network data stream for feature extraction and classification prediction, determines the classification real value corresponding to each feature vector, and then uses each feature vector and its corresponding classification real value to perform mapping calculations to determine the classification mapping value corresponding to each feature vector. Finally, a comprehensive early warning assessment is performed based on the classification mapping value corresponding to each feature vector to determine whether a network security incident has occurred, and then perform network security early warning to achieve fast and efficient network security incident monitoring and early warning. The present invention can more accurately assess the security status of network data, improve generalization capabilities, and reduce the false alarm rate of network security monitoring through a network data security assessment method that integrates the classification mapping results of multiple feature vectors, and through specific feature classification result mapping calculations and comprehensive assessment methods, it can improve the timeliness and scalability of network security monitoring and can adapt to large-scale network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a schematic diagram of the steps of the method in Example 1 of the present invention; Figure 2 This is a schematic diagram of the structure of the system in Example 2 of the present invention; Figure 3 This is a schematic diagram of the hardware system in Example 3 of the present invention. DETAILED DESCRIPTION

[0019] It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. The specific structures and functional details disclosed herein are only used to describe the exemplary embodiments of the present invention. However, the present invention can be embodied in many alternative forms, and it should not be understood that the present invention is limited to the embodiments set forth herein.

[0020] It should be understood that unless otherwise clearly specified and limited, the corresponding terms should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be an electrical connection, a direct connection, an indirect connection through an intermediate medium, or the internal connection of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments can be understood according to specific circumstances.

[0021] In the following description, certain details are provided to facilitate a complete understanding of the example embodiments. However, it will be appreciated by those of ordinary skill in the art that the example embodiments may be implemented without these certain details. For example, devices may be shown in block diagrams to avoid obscuring the examples with unnecessary details. In other embodiments, well-known processes, structures, and techniques may not be shown in unnecessary detail to avoid obscuring the embodiments.

[0022] Embodiment 1: This embodiment provides a network security event detection method based on machine learning, which can be applied to corresponding network security monitoring terminals, such as Figure 1 As shown, the method comprises the following steps: S1. Collect the original network traffic data of the same network data flow.

[0023] In specific implementation, the network security monitoring terminal collects the original network traffic data of the same network data stream in the corresponding channel in real time. The original network traffic data includes data packets (data packets contain the data to be transmitted and some control information, and each data packet has a source address and a destination address to indicate where it comes from and where it goes), protocol information (network traffic is controlled by different protocols, such as TCP / IP, UDP, HTTP, HTTPS, FTP, etc. These protocols define the structure and transmission method of data packets), port information (network traffic usually identifies different services or applications through specific port numbers), session information (a group of data packets transmitted in the network can form a session, indicating a communication process from start to end), and traffic statistics (including the number of independent visitors, the number of repeat visitors, the number of page views, the number of page views per visitor, and the statistical indicators of specific files / pages, such as the number of page displays, the number of file downloads, etc.).

[0024] S2. Preprocess each original network traffic data to obtain corresponding normalized data.

[0025] In the specific implementation, the network security monitoring terminal first cleans the original network traffic data to obtain the cleaned network traffic data. The data cleaning process includes: deleting duplicate data in the original network traffic data, and converting each character data in the original network traffic data after deduplication into corresponding numerical data, such as converting through unique hot encoding. Then, the interquartile range method is used to identify outliers in the numerical data, and the outliers in the numerical data are deleted to obtain the cleaned network traffic data. The interquartile range (IQR) method is to calculate the interquartile range of each numerical data IQR=Q3-Q1, where Q3 is the third quartile of the numerical data, Q1 is the first quartile of the numerical data, and determine the outlier boundary value OB=Q3-k·IQR, where k is the set outlier range adjustment coefficient. When a certain numerical data is greater than the outlier boundary value OB, it is determined to be an outlier. Then, the cleaned network traffic data is normalized to obtain the corresponding normalized data.

[0026] S3. Perform feature extraction on each preprocessed normalized data to obtain a corresponding feature vector.

[0027] In specific implementation, the principal component analysis (PCA) method can be used to extract features from the preprocessed normalized data to obtain the corresponding feature vector. The principal component analysis (PCA) algorithm can better process high-dimensional, nonlinear network traffic data and extract more discriminative features.

[0028] S4. Input each feature vector into a preset classification model to perform feature vector classification processing to obtain the corresponding classification real value.

[0029] In specific implementation, before inputting each feature vector into a preset classification model for feature vector classification processing, the network security monitoring terminal pre-builds an initialized support vector machine (SVM) model, and uses a training set to train the initialized support vector machine model until the set training conditions are met, and obtains a trained classification model, wherein the training set contains feature vector samples, and each feature vector sample is annotated with a corresponding classification label. When applied, the network security monitoring terminal inputs each feature vector into the classification model for feature vector classification prediction, and outputs the corresponding classification prediction result value, i.e., the classification real value, which is a real number in the interval [-1, 1].

[0030] S5. Calculate the classification mapping value corresponding to each eigenvector based on each eigenvector and its corresponding classification real value.

[0031] In specific implementation, the network security monitoring terminal substitutes each feature vector and its corresponding classification real value into a preset classification mapping formula for calculation to obtain a classification mapping value corresponding to each feature vector. The classification mapping formula is:

[0032] Among them, i represents the feature vector number, X i Characterizes the eigenvector numbered i, Z i Characterizing the feature vector X i The corresponding classification mapping value, Y i Characterizing the feature vector X i The corresponding categorical real value, X o is the set sample vector, σ is the set characteristic length scale parameter, b is the set bias parameter, Sigmoid represents the Sigmoid function operation, exp represents the natural exponential function operation, and ‖‖ represents the norm operation.

[0033] S6. Calculate comprehensive warning assessment parameters using the classification mapping values ​​corresponding to each eigenvector.

[0034] In specific implementation, the network security monitoring terminal first extracts several largest values ​​from the classification mapping values ​​corresponding to each feature vector as evaluation values, and then uses the average value of all evaluation values ​​as the comprehensive warning evaluation parameter.

[0035] S7. Determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

[0036] In specific implementation, when the comprehensive warning evaluation parameter is greater than the set security threshold, the network security monitoring terminal determines that a network security incident has occurred. If the comprehensive warning evaluation parameter does not exceed the set security threshold, the network security monitoring terminal determines that no network security incident has occurred. When it is determined that a network security incident has occurred, the network security monitoring terminal outputs the corresponding network security warning information to issue a network security incident warning.

[0037] This method uses a network data security assessment method that fuses the results of multi-feature vector classification mapping to more accurately assess the security status of network data, improve generalization ability, and reduce the false alarm rate of network security monitoring. In addition, through specific feature classification result mapping calculation and comprehensive evaluation methods, it can improve the timeliness and scalability of network security monitoring and can adapt to large-scale network environments.

[0038] Embodiment 2: This embodiment provides a network security incident detection system based on machine learning, such as Figure 2 As shown, it includes a data acquisition unit, a data processing unit, a feature extraction unit, a classification processing unit, a mapping calculation unit, an evaluation calculation unit and a detection and determination unit, wherein: A data collection unit, used for collecting each original network flow data of the same network data flow; A data processing unit, used for preprocessing each original network flow data to obtain corresponding normalized data; A feature extraction unit is used to extract features from each normalized data after preprocessing to obtain a corresponding feature vector; A classification processing unit, used to input each feature vector into a preset classification model to perform feature vector classification processing to obtain a corresponding classification real value; A mapping calculation unit, used for calculating the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value; An evaluation calculation unit, used to calculate comprehensive warning evaluation parameters using the classification mapping values ​​corresponding to each feature vector; The detection and determination unit is used to determine whether a network security incident has occurred based on the comprehensive warning evaluation parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

[0039] Embodiment 3: This embodiment provides a network security incident detection system based on machine learning, such as Figure 3 As shown, at the hardware level, it includes: Data interface, used to establish data connection between the processor and external data source; A memory for storing instructions; The processor is used to read the instructions stored in the memory and execute the network security event detection method in Example 1 according to the instructions.

[0040] Optionally, the system further includes an internal bus, through which the processor, the memory and the data interface can be interconnected, and the internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0041] The memory may include, but is not limited to, random access memory (RAM), read only memory (ROM), flash memory, first input first output (FIFO) and / or first in last out (FILO), etc. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0042] Embodiment 4: This embodiment provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the network security incident detection method in Embodiment 1. The computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0043] This embodiment also provides a computer program product, which, when running on a computer, executes the network security incident detection method in Embodiment 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0044] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A network security incident detection method based on machine learning, characterized in that: include: Collect the original network traffic data of the same network data flow; Preprocess each original network traffic data to obtain corresponding normalized data; Perform feature extraction on each normalized data after preprocessing to obtain the corresponding feature vector; Input each feature vector into a preset classification model to perform feature vector classification processing to obtain a corresponding classification real value; Calculate the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value; The comprehensive early warning assessment parameters are calculated using the classification mapping values ​​corresponding to each eigenvector; Determine whether a network security incident has occurred based on the comprehensive warning assessment parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

2. The network security incident detection method based on machine learning according to claim 1 is characterized in that: The preprocessing of each original network flow data to obtain corresponding normalized data includes: Perform data cleaning on each original network flow data to obtain cleaned network flow data; The cleaned network traffic data is normalized to obtain the corresponding normalized data.

3. The network security incident detection method based on machine learning according to claim 2 is characterized in that: The data cleaning of each original network flow data includes: Deleting duplicate data in the original network traffic data, and converting each character data in the original network traffic data after deduplication into corresponding numerical data; The interquartile range method is used to identify outliers in numerical data, and outliers in numerical data are deleted to obtain cleaned network traffic data.

4. The network security incident detection method based on machine learning according to claim 1 is characterized in that: The feature extraction of each preprocessed normalized data to obtain a corresponding feature vector includes: The principal component analysis method is used to extract features from the preprocessed normalized data to obtain the corresponding feature vectors.

5. The network security incident detection method based on machine learning according to claim 1 is characterized in that: Before inputting each feature vector into a preset classification model for feature vector classification processing, the method further includes: An initialized support vector machine model is constructed, and the initialized support vector machine model is trained using a training set until the set training conditions are met to obtain a trained classification model, wherein the training set includes feature vector samples, and each feature vector sample is annotated with a corresponding classification label.

6. The network security incident detection method based on machine learning according to claim 1 is characterized in that: The calculating the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value includes: Substitute each eigenvector and its corresponding classification real value into the preset classification mapping formula for calculation to obtain the classification mapping value corresponding to each eigenvector. The classification mapping formula is: Among them, i represents the feature vector number, X i Characterizes the eigenvector numbered i, Z i Characterizing the feature vector X i The corresponding classification mapping value, Y i Characterizing the feature vector X i The corresponding categorical real value, X o is the set sample vector, σ is the set characteristic length scale parameter, b is the set bias parameter, Sigmoid represents the Sigmoid function operation, exp represents the natural exponential function operation, and ‖‖ represents the norm operation.

7. The network security incident detection method based on machine learning according to claim 1 is characterized in that: The method of calculating the comprehensive early warning assessment parameters using the classification mapping values ​​corresponding to each feature vector includes: Extract the largest values ​​from the classification mapping values ​​corresponding to each feature vector as evaluation values; The average of all evaluation values ​​is taken as the comprehensive early warning evaluation parameter.

8. The network security incident detection method based on machine learning according to claim 1 is characterized in that: The determination of whether a network security incident has occurred based on the comprehensive early warning assessment parameters includes: When the comprehensive early warning assessment parameter is greater than the set security threshold, it is determined that a network security incident has occurred; otherwise, it is determined that no network security incident has occurred.

9. A network security incident detection system based on machine learning, characterized in that: It includes a data acquisition unit, a data processing unit, a feature extraction unit, a classification processing unit, a mapping calculation unit, an evaluation calculation unit and a detection and judgment unit, wherein: A data collection unit, used for collecting each original network flow data of the same network data flow; A data processing unit, used for preprocessing each original network flow data to obtain corresponding normalized data; A feature extraction unit is used to extract features from each normalized data after preprocessing to obtain a corresponding feature vector; A classification processing unit, used to input each feature vector into a preset classification model to perform feature vector classification processing to obtain a corresponding classification real value; A mapping calculation unit, used for calculating the classification mapping value corresponding to each eigenvector according to each eigenvector and its corresponding classification real value; An evaluation calculation unit, used to calculate comprehensive warning evaluation parameters using the classification mapping values ​​corresponding to each feature vector; The detection and determination unit is used to determine whether a network security incident has occurred based on the comprehensive warning evaluation parameters, and output corresponding network security warning information when it is determined that a network security incident has occurred.

10. A network security incident detection system based on machine learning, characterized in that: include: A memory for storing instructions; A processor is used to read the instructions stored in the memory and execute the network security incident detection method described in any one of claims 1-8 according to the instructions.

Images