A method and system for detecting abnormal network data

Abstract

The present invention provides a method and system for detecting abnormal data in a network, and belongs to the technical field of abnormal data detection. The abnormal data detection method comprises: adding a penalty term of the complexity of a decision subtree to a focal loss function, minimizing the focal loss function with the added penalty term, assigning weights to a plurality of pre-trained weak learners based on decision subtrees, linearly combining each weak learner based on the corresponding weight to obtain a strong learner, and obtaining network abnormal data based on the strong learner and the pre-processed network data to be detected. The present invention utilizes the ability of the focal loss function to process unbalanced data, adds a penalty term to suppress the complexity of the tree, improves the recognition ability and generalization ability of abnormal data, and the focal loss function can better handle the situation of data imbalance, and can more accurately perform network abnormal data detection under the premise that the overall accuracy level is comparable to other integrated models or single models.

Description

Technical Field

[0001] The present invention relates to the technical field of abnormal data detection, and in particular to a network abnormal data detection method, a network abnormal data detection device, a computer device, a computer-readable storage medium and a computer program product. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] As the use of computer networks and the number of applications running on them grows, network security becomes increasingly important. All computer systems have security vulnerabilities that are difficult and economically expensive for manufacturers to resolve from a technical point of view. Therefore, the importance of intrusion detection systems (IDS) as specialized equipment to detect anomalies and attacks in networks is increasing, and anomaly detection has long been the main focus of research in the field of intrusion detection.

[0004] The inventors have found that the existing methods of outlier detection using ensemble learning are to identify outliers through voting ensemble or stacking ensemble method, but outlier detection often has an extremely unbalanced positive and negative ratio. The use of voting method or other ensemble learning methods does not make good use of the condition of data imbalance. Summary of the invention

[0005] In order to address the shortcomings of the prior art, the present invention provides a network abnormal data detection method and system, which utilizes the ability of the focal loss function to process unbalanced data, adds a penalty term to suppress the complexity of the tree, and improves the recognition and generalization capabilities of abnormal data. The focal loss function can better handle data imbalance situations, and can more accurately detect network abnormal data under the premise that the overall accuracy level is comparable to that of other integrated models or single models.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for detecting abnormal network data.

[0008] A method for detecting abnormal network data includes the following steps:

[0009] Preprocess the network data to be detected;

[0010] A penalty term for the complexity of the decision subtree is added to the focal loss function, the focal loss function with the added penalty term is minimized, weights are assigned to multiple pre-trained weak learners based on decision subtrees, and each weak learner is linearly combined based on the corresponding weight to obtain a strong learner. According to the strong learner and the preprocessed network data to be detected, the network abnormal data is obtained.

[0011] As a further limitation of the first aspect of the present invention, the preprocessing includes: data cleaning, normalization and feature extraction.

[0012] As a further limitation of the first aspect of the present invention, training a decision subtree as a base model of a weak learner includes: training each weak learner using a bootstrap method to obtain training data, and randomly selecting a portion of extracted features from the preprocessed training data to construct a decision subtree.

[0013] As a further limitation of the first aspect of the present invention, when each node of the decision subtree is split, multiple feature subsets are randomly selected from all the features corresponding to the training data, and the feature sample with the largest information gain is used as the splitting node to generate the next subtree until the feature sample is completely distinguished or the restriction condition is reached, thereby obtaining a complete decision subtree.

[0014] As a further limitation of the first aspect of the present invention, the penalty term of the complexity of the decision subtree is: Among them, focal_loss is the original focal loss function, N is the number of acquired data, P nn (w) is an n×n matrix based on the weak learner weights w.

[0015] As a further limitation of the first aspect of the present invention, grid search or Bayesian optimization is used to adjust the hyperparameters of the focal loss function.

[0016] In a second aspect, the present invention provides a network abnormal data detection device.

[0017] A network abnormal data detection device, comprising:

[0018] The data preprocessing unit is configured to: preprocess the network data to be detected;

[0019] The network abnormal data generating unit is configured to: add a penalty term of the complexity of the decision subtree to the focal loss function, minimize the focal loss function with the added penalty term, assign weights to multiple pre-trained weak learners based on decision subtrees, linearly combine each weak learner based on the corresponding weight to obtain a strong learner, and obtain network abnormal data according to the strong learner and the preprocessed network data to be detected.

[0020] In a third aspect, the present invention provides a computer device, comprising: a processor and a computer-readable storage medium;

[0021] a processor adapted to execute a computer program;

[0022] A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the network abnormal data detection method as described in the first aspect of the present invention is implemented.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is suitable for being loaded by a processor and executing the network abnormal data detection method as described in the first aspect of the present invention.

[0024] In a fifth aspect, the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the network abnormal data detection method as described in the first aspect of the present invention.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. The present invention innovatively proposes a method for detecting abnormal network data, which utilizes the ability of the focal loss function to process unbalanced data, adds a penalty term to suppress the complexity of the tree, and improves the recognition and generalization capabilities of abnormal data. The focal loss function can better handle data imbalance situations, and can more accurately detect abnormal network data while maintaining the overall accuracy level comparable to other integrated models or single models.

[0027] 2. The present invention innovatively proposes a method for detecting abnormal network data. Through a specially designed penalty term for the complexity of the decision subtree, the complexity of the model is controlled to a certain extent, the possible overfitting of the model is suppressed, and the generalization ability of the network abnormal data detection problem is improved.

[0028] 3. The present invention innovatively proposes a method for detecting abnormal network data, which minimizes the focal loss function with a penalty term added, assigns weights to multiple pre-trained weak learners based on decision subtrees, and linearly combines each weak learner based on the corresponding weights to obtain a strong model, thereby further improving the accuracy of abnormal network data detection.

[0029] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 A schematic diagram of a flow chart of a method for detecting abnormal network data provided in Embodiment 1 of the present invention;

[0032] Figure 2 A schematic diagram of the structure of a network abnormal data detection device provided in Embodiment 2 of the present invention;

[0033] Figure 3 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0036] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0037] Embodiment 1:

[0038] In the embodiment, a method for detecting abnormal network data is proposed. The following is a brief introduction to the technical terms and related concepts involved in this processing solution, wherein:

[0039] Network abnormal data may be caused by a variety of reasons, including but not limited to hardware failure, software problems, network setting errors, firewall configuration problems, network congestion, network attacks, device not connected to the network, router setting problems, network cable problems, server interruptions, device system problems, poor network signals, confusing mobile phone system network settings, too many mobile phone running programs, mobile phone software problems and server problems. Although the present invention takes network attack data as an example, it may also include any other abnormal data that can be trained to be normally identified.

[0040] Random forest is a classifier that uses multiple decision trees to train and predict samples. Its subtree construction method includes: using N to represent the number of training cases (samples) and M to represent the number of features; inputting the number of features m to determine the decision result of a node on the decision tree; where m should be much smaller than M; sampling N times with replacement from the N training cases (samples) to form a training set (i.e. bootstrap sampling), and using the unsampled cases (samples) for prediction to evaluate its error; for each node, randomly selecting m features, the decision of each node on the decision tree is determined based on these features, and calculating the best splitting method based on these m features; each tree will grow completely without pruning, which may be adopted after a normal tree classifier is built.

[0041] Specifically, this embodiment uses the KDDCUP network intrusion dataset, which is a publicly available dataset in the field of anomaly detection and contains 125,970 data items, each of which contains 41 features. 1,069 data items are proportionally extracted for training, of which the proportion of outliers is 2.9%, to verify the effectiveness of the method proposed in the present invention, which specifically includes the following process:

[0042] S1. Clean, normalize and extract features from the collected data. Data preprocessing can facilitate model training and evaluation.

[0043] S2. On the collected labeled data, a list of weak learners is trained using the bootstrap method; the decision tree of the random forest is trained as the base model. Each training first uses the bootstrap method to obtain data, and then selects some features to construct a decision tree. Through these methods, a list of base models can be constructed;

[0044] S3. Minimize the focal loss function with special regularization terms to assign weights to weak learners. The focal loss function can improve the adaptability to outlier problems. The special regularization terms can suppress the complexity of the model and improve the generalization ability of outlier detection problems.

[0045] S4. Linearly combine the weak learners with the weights to obtain a set of strong learners, so that the strong learners can be used for outlier detection tasks, thereby completing the modeling and prediction of outlier detection problems.

[0046] Specifically, in one embodiment of the present invention, after obtaining labeled anomaly detection data, the data needs to be preprocessed, including data cleaning, normalization, and feature extraction operations. These operations can facilitate data input and improve the accuracy of the model, thereby better building an outlier detection model, which specifically includes the following sub-steps:

[0047] S1.1. Data cleaning.

[0048] After obtaining the data, first check whether there are missing values ​​in the data. If there are missing values ​​in the data, consider deleting the missing value samples or using interpolation, regression and other methods to process the missing values ​​according to the missing value ratio. In addition, there are categorical variables in the network intrusion data set, which are converted into numerical variables using label encoding.

[0049] S1.2. Data normalization.

[0050] Data normalization can map data to a certain range, which is convenient for putting data into the model for training. In the network intrusion data set, the variable value ranges of different attributes are very different, so normalization operation is required. Normalization is usually achieved using minimum and maximum normalization, Z-score standardization and logarithmic transformation. In this embodiment, the Z-score method is used to normalize the data.

[0051] Minimum and maximum normalization is a simple and intuitive method that scales the data to a fixed range to ensure that all data has exactly the same scale. Specifically, for the eigenvalue x, the minimum and maximum normalization uses the following formula for transformation:

[0052]

[0053] Among them, x' is the normalized sample, max(x) and min(x) are the maximum and minimum values ​​of the sample respectively. This method gives different features the same scale and is suitable for situations where there is no obvious difference between different features of the data.

[0054] Z-score standardization is a common standardization method. It standardizes data based on the mean and standard deviation of the original data. The processed data will obey the standard normal distribution.

[0055] Specifically, for the eigenvalue x, the Z-score standardization is transformed by the following formula:

[0056]

[0057] Among them, x' is the standardized sample, u is the mean of the sample, and σ is the standard deviation of the sample. This method is not affected by the extreme values ​​in the original data. It is suitable for situations where the maximum and minimum values ​​of the features are unknown and there are outliers in the data.

[0058] Logarithmic standardization is a data transformation technique that transforms data by taking the logarithm of each value in order to reduce the skewness of the data and reduce the impact of outliers. Specifically, for the eigenvalue x, the logarithmic standardization method is to standardize the data by transforming the logarithmic function:

[0059]

[0060] Among them, x' is the sample after standardization, and max(X) is the maximum value in the sample. This method can reduce the impact of outliers and is suitable for processing long-tail or skewed distribution data.

[0061] S1.3. Feature extraction.

[0062] After data standardization, before putting the data into the base model for training, feature selection is required. Feature selection is to select the attributes in the original data that have a significant impact on the classification results, thereby reducing the complexity of the model. Feature extraction is achieved using methods such as Fourier transform and principal component analysis. Extracting some features from all the features in the abnormal data can simplify the complexity of the model and improve the accuracy of the model.

[0063] In one embodiment of the present invention, a bootstrap method is used to train a series of weak learners on the extremely unbalanced labeled data obtained. By training the weak learners, a basis can be provided for subsequent model averaging, which specifically includes the following sub-steps:

[0064] S2.1. Bootstrap sampling.

[0065] Before starting to train the decision tree, in order to maintain the differences between the trees, it is necessary to use sampling with replacement as the training data of the tree.

[0066] S2.2. Construct a decision subtree.

[0067] Constructing a decision subtree is the basis for constructing a weak learner. The process of constructing a decision subtree is divided into two parts: randomly extracting attributes and constructing a decision tree.

[0068] Random extraction attributes: When each node of the decision tree is sorted, instead of considering all possible features, m feature subsets are randomly selected from all features. m is usually less than the total number of features M, so as to reduce the correlation between trees. In this embodiment, 7 features are selected to construct a sub-decision tree, and the above process is repeated to obtain 50 decision sub-trees.

[0069] Calculate information gain: Calculate information gain can be used to analyze the best split point. The calculation method of information gain is as follows:

[0070]

[0071] Among them, Gain is the information gain, H(t) is the entropy of the parent node, N tL 、N tR and N tThey are the number of samples in the left subtree, the right subtree and the tree with the parent node as the root node. Calculating the information gain can determine the amount of information obtained after selecting the node.

[0072] Calculate the split point: Use the sample with the largest information gain as the split node to generate the next subtree until the sample is completely distinguished or the restriction condition is reached, thus forming a complete decision subtree.

[0073] In one embodiment of the present invention, the focal loss function with a special regularization term is minimized, and weights are assigned to weak learners. Through these operations, the problem caused by extreme data imbalance can be balanced, while suppressing the complexity of the model, assigning corresponding weights to weak learners with different training effects, and improving the effect of the overall outlier detection model. Specifically, the following sub-steps are included:

[0074] S3.1. Construct a focal loss function with a special regularization term.

[0075] On the basis of the focal loss function, a penalty term for the complexity of the tree is added to balance the complexity of the tree. Specifically, the target loss function is given step by step in the following way. First, the form of the focal loss function is given:

[0076]

[0077] Among them, α t is the weight to balance positive and negative samples, p t is the predicted probability of the model for the true category, and γ is the focusing parameter, which is used to reduce the weight of easy-to-classify samples. By minimizing the focal loss function with a penalty term added, the classification difficulty caused by extreme data imbalance can be alleviated.

[0078] In order to give the penalty term for the complexity of the tree structure, refer to the penalty term construction method in the mallow-type regression method: Assuming that N data are obtained, after the subtree is built, the data will fall into a leaf node. If there are l data points in the leaf node, construct a one-dimensional vector V of size N×1 N1 , if the i-th node appears in the leaf node, its form is:

[0079]

[0080] If the i-th node does not appear in the leaf node, it is assigned a value of 0, so that the length of the vector can be maintained at N, and then the P matrix is ​​constructed using the V vector:

[0081]

[0082] We get an N×N matrix P. The P matrix can be regarded as a measure of data specificity. If the data finally falls on only one node, then the corresponding vector has only one 1, and the rest of the data are 0, indicating that the data is completely distinguished. The depth or complexity corresponding to this node is high. Then the objective function to be minimized is given in the following form:

[0083]

[0084] Here, w is the weight assigned to each weak learner, which gives the form of the objective function to be minimized.

[0085] S3.2. Adjust the hyperparameters of the focal loss function.

[0086] The hyperparameters are selected so as to fit the data when minimizing the focal loss. The method for adjusting the hyperparameters includes: grid search and Bayesian optimization. Specifically, the three hyperparameters to be optimized are: α and β of the focal loss function and the initial value w_start of the weight.

[0087] The grid search method for optimizing hyperparameters is: set a series of values ​​for each hyperparameter in the model, the combination of these values ​​will form a network, and then traverse and evaluate the parameter combination to select the optimal parameter combination. In this embodiment, the grid search method is used to optimize the hyperparameters, and the optimal parameter combinations α and β and the initial value w_start of the weight are 0.95, 2, and 0.05 respectively;

[0088] Bayesian optimization selects hyperparameters: First, through the Gaussian process, consider y N and N+1 The relationship between them is then calculated, and the posterior distribution is then calculated to sample the area where the global optimal solution is most likely to appear, thereby searching for the optimal parameters.

[0089] S3.3. Minimize the loss function.

[0090] The loss function is minimized by an optimization method to obtain a weight vector corresponding to the minimization of the loss function. The method for minimizing the loss function includes: sequential least squares method.

[0091] Weak learners are linearly combined with weights to obtain a set of strong learners, which can then be used for outlier detection tasks. Linearly combining the obtained weights with weak learners can improve the effect of outlier detection.

[0092] In the anomaly detection problem, the effect of a model can often be measured by the model's evaluation indicators. The performance evaluation indicators used in the experiment of this invention include accuracy, AUC index, recall rate and adjusted Rand index. The definition of the accuracy evaluation index is as follows:

[0093]

[0094] Among them, TP stands for true positive, TN stands for true negative, FP stands for false positive, and FN stands for false negative.

[0095] To calculate the AUC index, you first need to calculate the ROC curve. The ROC curve calculation first calculates the confusion matrix, then sorts the model's prediction probabilities from high to low, uses each probability value as the threshold, calculates TP and FP as the y-axis and x-axis respectively, and obtains the ROC curve. Then, the AUC index can be obtained by calculating the area under the ROC curve.

[0096] As an important indicator of anomaly detection, recall rate can measure the ability to correctly identify outliers. The definition of recall rate is as follows:

[0097]

[0098] Among them, TP stands for true positive and FN stands for false negative.

[0099] The adjusted Rand index is also a commonly used evaluation indicator. First, the definition of the Rand index is given:

[0100]

[0101] Where n is the total number of samples, k is the number of clusters, and a, b, c, and d represent the number of sample pairs in two clusters that are simultaneously assigned to the same cluster or different clusters.

[0102] Calculate the adjusted Rand index:

[0103]

[0104] Among them, E(RI) represents the expected Rand index. The value range of the adjusted Rand index is [-1, 1]. The closer the value is to 1, the higher the similarity between the two clusters. If the two clusters are completely consistent, the ARI value is 1; if the two clusters are completely inconsistent, the ARI value is -1.

[0105] In order to verify the effectiveness of the model proposed in this invention, this example selects some commonly used outlier detection models for comparison, including:

[0106] Isolation forest method: Isolation forest method is an algorithm for anomaly detection in structured data. In the isolation forest, anomalies are defined as those outliers that are easily isolated, that is, sparsely distributed points.

[0107] Logistic regression method: Logistic regression is a binary classification model that can convert the degree of abnormality of data points into probability value output, thereby determining the abnormality of the data.

[0108] Random forest method: The random forest method uses a voting method to detect outliers to determine whether the data is an outlier.

[0109] A model averaging method based on minimizing the cross entropy loss. In this method, each model is assigned a corresponding weight according to the cross entropy loss of its own classification, and then an integrated model is used to determine whether the data is an outlier.

[0110] Each model and the method of the present invention were run for 60 rounds and the average value was taken. The comparison of the methods is shown in Table 1.

[0111] Table 1: Comparison of the effects of the methods.

[0112]

[0113] It can be seen from Table 1 that compared with the classic anomaly detection and classification methods, the method proposed in the present invention performs better in all indicators, among which it shows more significant advantages in the modified Rand indicator, recall rate and AUC indicator.

[0114] Embodiment 2:

[0115] like Figure 2 As shown, Embodiment 2 of the present invention provides a network abnormal data detection device, including:

[0116] The data preprocessing unit is configured to: preprocess the network data to be detected, more specifically, the preprocessing includes: data cleaning, normalization and feature extraction;

[0117] The network abnormal data generating unit is configured to: add a penalty term of the complexity of the decision subtree to the focal loss function, minimize the focal loss function with the added penalty term, assign weights to multiple pre-trained weak learners based on decision subtrees, linearly combine each weak learner based on the corresponding weight to obtain a strong learner, and obtain network abnormal data according to the strong learner and the preprocessed network data to be detected.

[0118] The network abnormal data generating unit includes:

[0119] First, the form of the objective function to be minimized is given:

[0120]

[0121] Here, w is the weight assigned to each weak learner, which gives the form of the objective function to be minimized.

[0122] Then, hyperparameters are selected to match the data while minimizing the focal loss. The method for adjusting the hyperparameters includes: grid search and Bayesian optimization. The three hyperparameters to be optimized are: α and β of the focal loss function and the initial value w_start of the weight.

[0123] Specifically, the grid search method for optimizing hyperparameters is: set a series of values ​​for each hyperparameter in the model, the combination of these values ​​will form a network, and then traverse and evaluate the parameter combination to select the optimal parameter combination. In this embodiment, the grid search method is used to optimize the hyperparameters, and the optimal parameter combinations α and β and the initial value w_start of the weight are 0.95, 2, and 0.05 respectively;

[0124] Specifically, Bayesian optimization selects hyperparameters: First, through the Gaussian process, consider y N and N+1 The relationship between them is then calculated, and the posterior distribution is then calculated to sample the area where the global optimal solution is most likely to appear, thereby searching for the optimal parameters.

[0125] Thirdly, an optimization method is used to minimize the loss function to obtain a weight vector corresponding to when the loss function is minimized. The method for minimizing the loss function includes: sequential least squares method.

[0126] Finally, the weak learners are linearly combined with the weights to obtain a set of strong learners, so that the strong learners can be used for outlier detection tasks. The obtained weights are linearly combined with the weak learners to improve the effect of outlier detection.

[0127] It is understandable that the above-mentioned modules can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0128] According to another embodiment of the present application, the system described in this embodiment can be constructed and the image processing method of the embodiment of the present application can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Example 1 on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0129] Embodiment 3:

[0130] like Figure 3 As shown, Embodiment 3 of the present invention provides an electronic device, which includes a processor 1001, a communication interface 1002, and a computer-readable storage medium 1003. The processor 1001, the communication interface 1002, and the computer-readable storage medium 1003 may be connected via a bus or other means.

[0131] Among them, the communication interface 1002 is used to receive and send data, the computer-readable storage medium 1003 can be stored in the memory of the electronic device, the computer-readable storage medium 1003 is used to store a computer program, the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer-readable storage medium 1003.

[0132] Processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method processes or corresponding functions.

[0133] The processor 1001 is configured to execute the processes of data preprocessing and network abnormal data generation, wherein when the processor 1001 executes data preprocessing, it specifically executes the processes of data cleaning, normalization processing and feature extraction in sequence.

[0134] When the processor 1001 generates network abnormal data, it specifically performs the following steps:

[0135] First, the form of the objective function to be minimized is given:

[0136]

[0137] Here, w is the weight assigned to each weak learner, which gives the form of the objective function to be minimized.

[0138] Then, hyperparameters are selected to match the data while minimizing the focal loss. The method for adjusting the hyperparameters includes: grid search and Bayesian optimization. The three hyperparameters to be optimized are: α and β of the focal loss function and the initial value w_start of the weight.

[0139] Specifically, the grid search method for optimizing hyperparameters is: set a series of values ​​for each hyperparameter in the model, the combination of these values ​​will form a network, and then traverse and evaluate the hyperparameter combination to select the optimal parameter combination. In this embodiment, the grid search method is used to optimize the parameters, and the obtained optimal parameter combinations α and β and the initial value w_start of the weight are 0.95, 2, and 0.05 respectively;

[0140] Specifically, Bayesian optimization selects hyperparameters: First, through the Gaussian process, consider y N and N+1 The relationship between them is then calculated, and the posterior distribution is then sampled in the area where the global optimal solution is most likely to appear, thereby searching for the optimal hyperparameters.

[0141] Thirdly, an optimization method is used to minimize the loss function to obtain a weight vector corresponding to when the loss function is minimized. The method for minimizing the loss function includes: sequential least squares method.

[0142] Finally, the weak learners are linearly combined with the weights to obtain a set of strong learners, so that the strong learners can be used for outlier detection tasks. The obtained weights are linearly combined with the weak learners to improve the effect of outlier detection.

[0143] Embodiment 4:

[0144] This embodiment provides a computer-readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer-readable storage medium provides a storage space that stores the processing system of the electronic device.

[0145] In addition, the storage space also stores one or more instructions suitable for being loaded and executed by the processor, and these instructions may be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor.

[0146] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the method embodiment described in the above embodiment 1;

[0147] In a specific implementation, one or more instructions in a computer-readable storage medium are loaded by a processor and execute the process of data preprocessing and network abnormal data generation, wherein one or more instructions in a computer-readable storage medium are loaded by a processor and when executing data preprocessing, specifically execute the process of data cleaning, normalization processing and feature extraction in sequence.

[0148] One or more instructions in the computer-readable storage medium are loaded by the processor and when executing network abnormality data generation, specifically perform the following steps:

[0149] First, the form of the objective function to be minimized is given:

[0150]

[0151] Here, w is the weight assigned to each weak learner, which gives the form of the objective function to be minimized.

[0152] Then, hyperparameters are selected to match the data while minimizing the focal loss. The method for adjusting the hyperparameters includes: grid search and Bayesian optimization. The three hyperparameters to be optimized are: α and β of the focal loss function and the initial value w_start of the weight.

[0153] Specifically, the grid search method for optimizing hyperparameters is: set a series of values ​​for each parameter in the model, the combination of these values ​​will form a network, and then traverse and evaluate the hyperparameter combination to select the optimal parameter combination. In this embodiment, the grid search method is used to optimize the parameters, and the optimal parameter combinations α and β and the initial value w_start of the weight are 0.95, 2, and 0.05 respectively;

[0154] Specifically, Bayesian optimization selects hyperparameters: First, through the Gaussian process, consider y N andN+1 The relationship between them is then calculated, and the posterior distribution is then calculated to sample the area where the global optimal solution is most likely to appear, thereby searching for the optimal parameters.

[0155] Thirdly, an optimization method is used to minimize the loss function to obtain a weight vector corresponding to when the loss function is minimized. The method for minimizing the loss function includes: sequential least squares method.

[0156] Finally, the weak learners are linearly combined with the weights to obtain a set of strong learners, so that the strong learners can be used for outlier detection tasks. The obtained weights are linearly combined with the weak learners to improve the effect of outlier detection.

[0157] Embodiment 5:

[0158] This embodiment provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method described in Embodiment 1.

[0159] A person skilled in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0160] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instruction can be stored in a computer-readable storage medium or transmitted by a computer-readable storage medium. The computer instruction can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that contains one or more available media integration. Available media can be magnetic media (for example, floppy disk, hard disk, tape), optical media (for example, DVD), or semiconductor media (for example, solid-state drive (Solid State Disk, SSD)) and the like.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormal network data, characterized in that: The process includes: Preprocess the network data to be detected; A penalty term of the complexity of the decision subtree is added to the focal loss function, the focal loss function with the added penalty term is minimized, weights are assigned to a plurality of pre-trained weak learners based on decision subtrees, each weak learner is linearly combined based on the corresponding weight to obtain a strong learner, and network abnormal data is obtained according to the strong learner and the pre-processed network data to be detected; The focal loss function adds a penalty term for the complexity of the decision subtree, specifically: The form of the focal loss function is: focal_loss=-a t (1-p t ) γ log(p t )(5); Among them, α t is the weight to balance positive and negative samples, p t is the model's predicted probability for the true category, and γ is the focus parameter, which is used to reduce the weight of easy-to-classify samples. By minimizing the focus loss function with a penalty term added, the classification difficulty caused by extreme data imbalance can be alleviated; The penalty term for the complexity of the decision subtree is: Among them, focal_loss is the original focal loss function, N is the number of acquired data, P nn (w) is an n×n matrix based on the weak learner weight w; The focal loss function with the penalty term added is: When constructing the penalty term, assume that N data are obtained. After the subtree is constructed, the data will fall into a leaf node. If there are l data points in the leaf node, a one-dimensional vector V of size N×1 is constructed. N·1 , if the i-th node appears in the leaf node, its form is: If the i-th node does not appear in the leaf node, it is assigned a value of 0 to maintain the length of the vector as N, and then the P matrix is ​​constructed using the V vector: We get an N×N matrix P. The P matrix is ​​regarded as a measure of data specificity. If the data finally falls on only one node, then the corresponding vector has only one 1, and the rest of the data are 0, indicating that the data is completely distinguished, and the depth or complexity corresponding to the node is high. The decision subtree is specifically: The training of each weak learner adopts the bootstrap method to obtain training data, and randomly selects some extracted features from the preprocessed training data to construct a decision subtree. When each node of the decision tree is divided, instead of considering all possible features, m feature subsets are randomly selected from all features, where m is less than the total number of features M. The information gain is calculated to analyze the optimal split point. The information gain is calculated as follows: Among them, Gain is the information gain, H(t) is the entropy of the parent node, N tL 、N tR and N t They are the number of samples in the left subtree, the right subtree and the tree with the parent node as the root node. The information gain is calculated to determine the amount of information obtained after selecting the node; The sample with the largest information gain is used as the split node to generate the next subtree until the sample is completely distinguished or the restriction condition is reached, thus forming a complete decision subtree; Use grid search or Bayesian optimization to adjust the hyperparameters of the focal loss function, and select hyperparameters that fit the data while minimizing the focal loss; An optimization method is used to minimize the loss function to obtain a weight vector corresponding to the minimized loss function; the method for minimizing the loss function includes: sequential least squares method.

2. The network abnormal data detection method according to claim 1, characterized in that: The preprocessing includes: data cleaning, normalization and feature extraction.

3. The network abnormal data detection method according to claim 1, characterized in that: When splitting each node of the decision subtree, multiple feature subsets are randomly selected from all the features corresponding to the training data, and the feature sample with the largest information gain is used as the split node to generate the next subtree until the feature sample is completely distinguished or the restriction condition is reached, and a complete decision subtree is obtained.

4. A network abnormal data detection device, characterized in that: The method for detecting abnormal network data according to any one of claims 1 to 3 comprises: The data preprocessing unit is configured to: preprocess the network data to be detected; The network abnormal data generating unit is configured to: add a penalty term of the complexity of the decision subtree to the focal loss function, minimize the focal loss function with the added penalty term, assign weights to a plurality of pre-trained weak learners based on decision subtrees, linearly combine each weak learner based on the corresponding weight to obtain a strong learner, and obtain network abnormal data according to the strong learner and the pre-processed network data to be detected; The focal loss function adds a penalty term for the complexity of the decision subtree, specifically: The form of the focal loss function is: focal_loss=-a t (1-p t ) γ log(p t ); Among them, α t is the weight to balance positive and negative samples, p t is the model's predicted probability for the true category, and γ is the focus parameter, which is used to reduce the weight of easy-to-classify samples. By minimizing the focus loss function with a penalty term added, the classification difficulty caused by extreme data imbalance can be alleviated; The penalty term for the complexity of the decision subtree is: Among them, focal_loss is the original focal loss function, N is the number of acquired data, P nn (w) is an n×n matrix based on the weak learner weight w; The decision subtree is specifically: The training of each weak learner adopts the bootstrap method to obtain training data, and randomly selects some extracted features from the preprocessed training data to construct a decision subtree; Use grid search or Bayesian optimization to adjust the hyperparameters of the focal loss function, and select hyperparameters that fit the data while minimizing the focal loss; An optimization method is used to minimize the loss function to obtain a weight vector corresponding to the minimized loss function; the method for minimizing the loss function includes: sequential least squares method.

5. A computer device, characterized in that: include: a processor and a computer readable storage medium; a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the network abnormal data detection method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the network abnormal data detection method according to any one of claims 1 to 3.

7. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the network abnormal data detection method according to any one of claims 1 to 3 is implemented.

Images