A method for abnormal data detection and classification based on deep learning

Abstract

The present invention discloses a method for detecting and classifying abnormal data based on deep learning, comprising the following steps: S1, obtaining a target data set; S2, constructing a multi-layer neural network model; S3, using a semi-supervised learning model in Scikit-learn to perform preliminary training on the target data set; S4, constructing an integrated model; S5, training the integrated model by supervised learning, wherein the training data set includes normal data and abnormal data, and optimizing the weight of the integrated model based on a classification loss function; S6, applying the trained integrated model to a target data set to be detected, extracting features from the target data set to be detected, and using the integrated model to determine whether a data point is abnormal data, and outputting a preliminary detection result; S7, further classifying the detected abnormal data, dividing the abnormal data into different types, and generating a classification result; S8, updating the integrated model according to the detection and classification results. The present invention improves the classification capability of multi-category abnormal data and avoids the misclassification phenomenon in traditional methods.

Description

Technical Field

[0001] The present invention relates to the field of data detection technology, and in particular to an abnormal data detection and classification method based on deep learning. Background Art

[0002] In the prior art, the application of abnormal data detection and classification in large-scale complex data sets mainly relies on rule-based algorithms or shallow machine learning models. Traditional rule-based anomaly detection methods usually identify abnormal data through manually set thresholds or statistical rules. However, the limitation of existing methods is that when faced with data sets with high dimensions, nonlinearities or complex patterns, feature extraction is not sufficient and potential abnormal features in the data cannot be effectively captured, resulting in high missed and false alarm rates. In addition, traditional methods lack automated feature extraction capabilities. When dealing with dynamically changing data environments, the setting of rules and thresholds is difficult to adaptively adjust, affecting the accuracy and efficiency of anomaly detection.

[0003] Among the existing anomaly detection technologies based on machine learning, common algorithms include support vector machines and then K-nearest neighbor algorithms. Machine learning algorithms can improve detection accuracy to a certain extent by training on partially labeled data. However, they perform poorly when processing unlabeled data, especially in semi-supervised learning scenarios. Existing models are insufficient in how to effectively utilize unlabeled data.

[0004] In addition, when processing large-scale data sets, existing technologies often find it difficult to strike a balance between detection accuracy and computational efficiency. Shallow models have obvious performance bottlenecks when processing large-scale complex data and are difficult to respond in real time. In addition, due to the lack of dynamic learning and self-update capabilities, existing anomaly detection systems are prone to becoming outdated when faced with a rapidly changing data environment, and are unable to respond to new types of anomalies in a timely manner, resulting in a gradual decrease in detection accuracy and an inability to adapt to complex scenarios in actual applications, such as network security and quality control, which require high-precision anomaly detection.

[0005] In summary, the existing technologies have the following main shortcomings: First, traditional rule-based methods cannot effectively handle potential abnormal features in high-dimensional and complex data, and the detection accuracy is limited; second, existing machine learning methods are insufficient in semi-supervised learning and multi-class anomaly classification, and are prone to misclassification or missed detection problems; third, the existing technologies lack a dynamic update mechanism and cannot adapt to the rapidly changing data environment, resulting in insufficient accuracy and robustness of the anomaly detection system. Summary of the invention

[0006] One object of the present invention is to propose a method for detecting and classifying abnormal data based on deep learning, which improves the classification capability of multi-category abnormal data and avoids the misclassification phenomenon in traditional methods.

[0007] According to an embodiment of the present invention, a method for detecting and classifying abnormal data based on deep learning includes the following steps:

[0008] S1, obtain the target data set and preprocess the target data set;

[0009] S2, construct a multi-layer neural network model, which includes an input layer, several hidden layers and an output layer in sequence;

[0010] S3. Perform preliminary training on the target data set using a semi-supervised learning model in Scikit-learn. The semi-supervised model includes a semi-supervised k-nearest neighbor model and a semi-supervised support vector machine model. The target data set is trained separately to extract preliminary anomaly detection features from the target data sets with partial annotations and without annotations.

[0011] S4, integrating the preliminary anomaly detection features of the Scikit-learn semi-supervised model with the anomaly detection results output by the multi-layer neural network model to build an integrated model;

[0012] S5. Train the integrated model by supervised learning. The training data set contains normal data and abnormal data. The integrated model weight is optimized based on the classification loss function.

[0013] S6. Apply the trained integrated model to the target data set to be detected, extract features from the target data set to be detected, and use the integrated model to determine whether the data point is abnormal data, and output a preliminary detection result;

[0014] S7, further classify the detected abnormal data, divide the abnormal data into different types, and generate classification results;

[0015] S8. Monitor the newly input data stream in real time and update the integrated model based on the detection and classification results.

[0016] Optionally, the S1 includes:

[0017] S11. Obtain target data set:

[0018] D={x1,x2,…,x n};

[0019] Among them, x i is the i-th data point in the target data set, i = 1, 2, ..., n, and n is the number of samples in the data set;

[0020] S12, using a denoising algorithm based on mean filtering to perform data denoising on the target data set D, processing each data point to obtain a denoised target data set D';

[0021] S13, performing missing value filling processing on the denoised target data set D', and filling the values ​​at the positions of the missing values ​​by interpolation to obtain the filled target data set D";

[0022] S14, use the Min-Max normalization method to normalize the filled target data set D" and convert each data point x" i Map to the interval [0,1] to obtain the preprocessed target data set:

[0023]

[0024] Optionally, S2 includes:

[0025] S21, constructing an input layer, the input layer is used to receive the preprocessed target data set, and the input layer extracts preliminary features by mapping each data point to a high-dimensional feature space;

[0026] S22. Construct several hidden layers, where the hidden layers consist of a multi-layer asymmetric neural network. Each hidden layer extracts features from the data and uses a ReLU activation function to perform nonlinear transformation on the input features.

[0027] S23. Introduce a multi-branch structure in the hidden layer to construct multiple branch networks. Each branch network processes the features of different dimensions of the target data set. The output features of each branch network are integrated to obtain the anomaly detection effect. The output calculation formula of the branch network is:

[0028]

[0029]

[0030] in, and are the outputs of the two branches of the i-th layer, and are the weight matrices of the i-th layer, representing the connection weights between the input features and the hidden layer nodes; and are the bias vectors of the i-th layer branch, respectively, which adjust the linear transformation results of the features;

[0031] S24, constructing a self-supervised learning module for processing unlabeled data sets, the self-supervised learning module automatically learning abnormal feature patterns and updating the multi-layer neural network model;

[0032] S25. Construct an output layer for generating anomaly detection results. Use the Softmax function to map the output results of the multi-layer neural network model into multi-category probability distributions for classifying different types of anomaly data. The Softmax function calculates the probability distributions of different anomaly categories based on the characteristics of the input data and generates the final classification results:

[0033]

[0034] Among them, P(y=k|x) represents the probability that the input data x belongs to the kth type of abnormal data, W k is the weight matrix of the k-th anomaly in the output layer, and K is the total number of abnormal data categories.

[0035] Optionally, the S3 includes:

[0036] S31. Use the semi-supervised k-nearest neighbor model in Scikit-learn to train the target dataset D norm Perform preliminary training. The target data set includes partially labeled data and unlabeled data. The weighted distance between data points is calculated by nonlinear weight distance to extract preliminary anomaly detection features:

[0037]

[0038] Among them, d w (x i ,x j ) is the weighted distance, and The data points x i and x j The value of the k-th dimension feature, m is the dimension of the feature, w k is the weight of the k-th dimension feature, α, β, γ are nonlinear coefficients;

[0039] S32. Use the semi-supervised support vector machine model in Scikit-learn to train the target data set and classify the data points by optimizing the decision boundary formula based on multi-scale regularization:

[0040]

[0041] Among them, f(x) is the classification decision function of the input data x, x i is the support vector, K(x i ,x) is the kernel function, α i is the Lagrange multiplier, b is the bias term, n l is the number of labeled data, λ is the regularization coefficient, is the kernel function gradient;

[0042] S33, integrating the preliminary anomaly detection features extracted by the semi-supervised k-nearest neighbor model and the semi-supervised support vector machine model to generate a preliminary anomaly feature set of the target data set:

[0043]

[0044] Among them, Φ(x i ) is the data point x i The comprehensive abnormal features, σ is the nonlinear activation function, P is the number of feature integration branches, ω p is the weight of each integrated branch, Q is the number of classification functions of the support vector machine, μ q is the weight of each classification function, f q (x i ) are the classification decision results output by different support vector machine models.

[0045] Optionally, the building of the integrated model includes extracting the preliminary anomaly detection feature set Φ(x i ) and the output classification result P of the multi-layer neural network model NN (y i =k|x i ) to generate an integrated feature vector:

[0046]

[0047] in, is the final prediction output of the integrated model, σ is the activation function, W ensemble is the weight matrix of the integrated model, α and β are the integration coefficients, corresponding to the weights of the semi-supervised model features and the multi-layer neural network output, respectively, b ensemble is the bias term.

[0048] Optionally, the S5 includes:

[0049] S51, construct a training data set containing normal data and abnormal data, the training data set is:

[0050] D train ={(x1,y1),(x2,y2),…,(x n ,y n )};

[0051] Among them, x i is the i-th data point, y i is the corresponding label, y i =0 indicates normal data, y i = 1 indicates abnormal data. Each data point in the training data set undergoes pre-processing and feature extraction to generate an integrated feature vector ψ(x i );

[0052] S52, training data set D train Input the integrated model and calculate the predicted output of the model through forward propagation The output of the ensemble model represents the data point x i Probability of belonging to the anomaly class:

[0053]

[0054] Among them, σ is a nonlinear activation function, is the weight matrix of the jth class of the ensemble model, ψ (j) (x i ) is the feature output of the jth class, is the bias term, δ j is the regularization coefficient, is the gradient regularization term of the integrated model feature space;

[0055] S53. Define the binary cross entropy loss function to measure the error between the predicted output of the integrated model and the true label. The loss function is defined as:

[0056]

[0057] Where L is the total loss, is the abnormal probability predicted by the integrated model, y i is the true label, λ is the regularization coefficient, It is the L2 regularization term to prevent the model from overfitting;

[0058] S54. Calculate the loss function L for the integrated model weight through the back propagation algorithm and bias The gradient update is:

[0059]

[0060]

[0061] Where η is the learning rate, and are the gradients of the loss function with respect to weights and biases, respectively.

[0062] Optionally, the S6 includes:

[0063] S61, applying the trained integrated model to the target data set D to be detected test , feature extraction is performed on each data point in the target data set to be detected, and the extracted feature is represented as ψ test (x i );

[0064] S63, the eigenvector ψ test (x i ) is input into the integrated model, and the integrated model is used to calculate the data x i Probability of belonging to the anomaly class:

[0065] P ensemble (y i =1|x i )=σ(W ensemble ·ψ test (x i )+b ensemble );

[0066] Among them, P ensemble (y i =1|x i ) represents the data point x i The probability of belonging to the abnormal category, W ensemble is the weight matrix of the integrated model, b ensemble is the bias term, σ is the activation function;

[0067] S64. According to the output results of the integrated model, determine the data point x i Is it abnormal data? Set the threshold τ for abnormal detection. If P ensemble (y i =1|x i )≥τ, then we can judge the data point x i It is abnormal data, otherwise it is normal data, and the final output is the preliminary test result:

[0068]

[0069] in, For data point x i τ is the classification threshold of anomaly detection, which is used to distinguish normal data from abnormal data.

[0070] Optionally, the S7 includes:

[0071] S71. The abnormal data set detected initially Input into the classification model, k is the number of abnormal data points, and the classification model is used to sub-classify the abnormal data;

[0072] S72. Classify the abnormal data using a classification model. The output of the classification model is the probability distribution of each abnormal data point belonging to a specific abnormal type. The classification result is expressed as:

[0073]

[0074] Among them, P class (y i =c|xi ) represents an abnormal data point x i The probability of belonging to the cth type of anomaly, is the weight matrix of the classification model, is the bias term of the cth class, and C is the total number of abnormal types;

[0075] S73, each abnormal data point x is classified according to the output probability of the classification model. i The abnormal category with the highest probability value is assigned to generate the classification result, which is expressed as:

[0076]

[0077] in, is an abnormal data point x i The final classification result indicates the abnormal category to which the data point belongs.

[0078] The beneficial effects of the present invention are:

[0079] (1) The present invention constructs a multi-layer neural network model, and extracts features from high-dimensional and nonlinear data in sequence through an input layer, a plurality of hidden layers, and an output layer. By introducing the ReLU activation function and a multi-branch structure, the model can capture subtle abnormal patterns in complex data. The powerful learning ability of the multi-layer neural network is combined with the semi-supervised learning model of Scikit-learn, which can automatically extract abnormal features from partially annotated and unannotated data, further improving the accuracy of detection, and having a significant recognition effect on subtle abnormal points that are difficult to detect.

[0080] (2) The present invention integrates the preliminary detection results of the multi-layer neural network with the semi-supervised support vector machine and the K nearest neighbor algorithm by constructing an integrated model, thereby effectively improving the classification accuracy of abnormal data. In scenarios with complex abnormal data types, the Softmax function and a multi-category classification algorithm are used to calculate the probability distribution of abnormal points, thereby improving the classification ability of multi-category abnormal data, avoiding the misclassification phenomenon in traditional methods, and being able to more accurately distinguish different types of abnormal data.

[0081] (3) The present invention continuously updates the weight parameters of the integrated model through an online learning mechanism and weight optimization based on the gradient descent method, so that it can adaptively adjust the classification and detection rules when facing a new data environment, thereby ensuring the continuous optimization performance of the model. Unlike the static model in the prior art, the present invention can maintain high performance in rapidly changing data scenarios through the combination of self-supervised learning and semi-supervised learning, respond to new anomaly types in a timely manner, ensure that the detection accuracy of the model does not decay over time, and significantly improve the robustness and adaptability of the system in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0083] Figure 1 A flowchart of an abnormal data detection and classification method based on deep learning proposed by the present invention;

[0084] Figure 2 This is a process diagram of applying the integrated model in the deep learning-based abnormal data detection and classification method proposed in the present invention to the target data set to be detected. DETAILED DESCRIPTION

[0085] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0086] refer to Figure 1-2 , a method for abnormal data detection and classification based on deep learning, comprising the following steps:

[0087] S1, obtain the target data set and preprocess the target data set;

[0088] S2, construct a multi-layer neural network model, which includes an input layer, several hidden layers and an output layer in sequence;

[0089] S3. Use the semi-supervised learning model in Scikit-learn to perform preliminary training on the target data set. The semi-supervised model includes a semi-supervised k-nearest neighbor model and a semi-supervised support vector machine model. The target data set is trained separately to extract preliminary anomaly detection features from the target data set with partial annotations and without annotations.

[0090] S4, integrating the preliminary anomaly detection features of the Scikit-learn semi-supervised model with the anomaly detection results output by the multi-layer neural network model to build an integrated model;

[0091] S5. Train the integrated model by supervised learning. The training data set contains normal data and abnormal data. The integrated model weight is optimized based on the classification loss function.

[0092] S6. Apply the trained integrated model to the target data set to be detected, extract features from the target data set to be detected, and use the integrated model to determine whether the data point is abnormal data, and output a preliminary detection result;

[0093] S7, further classify the detected abnormal data, divide the abnormal data into different types, and generate classification results;

[0094] S8. Monitor the newly input data stream in real time and update the integrated model based on the detection and classification results.

[0095] In this implementation, S1 includes:

[0096] S11. Obtain target data set:

[0097] D={x1,x2,…,x n};

[0098] Among them, x i is the i-th data point in the target data set, i = 1, 2, ..., n, and n is the number of samples in the data set;

[0099] S12, using a denoising algorithm based on mean filtering to perform data denoising on the target data set D, processing each data point to obtain a denoised target data set D';

[0100] S13, performing missing value filling processing on the denoised target data set D', and filling the values ​​at the positions of the missing values ​​by interpolation to obtain the filled target data set D";

[0101] S14, use the Min-Max normalization method to normalize the filled target data set D" and convert each data point x" i Map to the interval [0,1] to obtain the preprocessed target data set:

[0102]

[0103] In this implementation, S2 includes:

[0104] S21, constructing an input layer, the input layer is used to receive the preprocessed target data set, and the input layer extracts preliminary features by mapping each data point to a high-dimensional feature space;

[0105] S22. Construct several hidden layers, where the hidden layers consist of a multi-layer asymmetric neural network. Each hidden layer extracts features from the data and uses a ReLU activation function to perform nonlinear transformation on the input features.

[0106] S23. Introduce a multi-branch structure in the hidden layer to construct multiple branch networks. Each branch network processes the features of different dimensions of the target data set. The output features of each branch network are integrated to obtain the anomaly detection effect. The output calculation formula of the branch network is:

[0107]

[0108]

[0109] in, and are the outputs of the two branches of the i-th layer, and are the weight matrices of the i-th layer, representing the connection weights between the input features and the hidden layer nodes; and are the bias vectors of the i-th layer branch, respectively, which adjust the linear transformation results of the features;

[0110] S24, constructing a self-supervised learning module for processing unlabeled data sets, the self-supervised learning module automatically learning abnormal feature patterns and updating the multi-layer neural network model;

[0111] S25. Construct an output layer for generating anomaly detection results. Use the Softmax function to map the output results of the multi-layer neural network model into multi-category probability distributions for classifying different types of anomaly data. The Softmax function calculates the probability distributions of different anomaly categories based on the characteristics of the input data and generates the final classification results:

[0112]

[0113] Among them, P(y=k|x) represents the probability that the input data x belongs to the kth type of abnormal data, W k is the weight matrix of the k-th anomaly in the output layer, and K is the total number of abnormal data categories.

[0114] In this implementation, S3 includes:

[0115] S31. Use the semi-supervised k-nearest neighbor model in Scikit-learn to train the target dataset D norm Perform preliminary training. The target data set includes partially labeled data and unlabeled data. The weighted distance between data points is calculated by nonlinear weight distance to extract preliminary anomaly detection features:

[0116]

[0117] Among them, d w (x i ,x j ) is the weighted distance, and The data points x i and x j The value of the k-th dimension feature, m is the dimension of the feature, w k is the weight of the k-th dimension feature, α, β, γ are nonlinear coefficients;

[0118] S32. Use the semi-supervised support vector machine model in Scikit-learn to train the target data set and classify the data points by optimizing the decision boundary formula based on multi-scale regularization:

[0119]

[0120] Among them, f(x) is the classification decision function of the input data x, x i is the support vector, K(x i ,x) is the kernel function, α i is the Lagrange multiplier, b is the bias term, n l is the number of labeled data, λ is the regularization coefficient, is the kernel function gradient;

[0121] S33, integrating the preliminary anomaly detection features extracted by the semi-supervised k-nearest neighbor model and the semi-supervised support vector machine model to generate a preliminary anomaly feature set of the target data set:

[0122]

[0123] Among them, Φ(x i ) is the data point x i The comprehensive abnormal features, σ is the nonlinear activation function, P is the number of feature integration branches, ω p is the weight of each integrated branch, Q is the number of classification functions of the support vector machine, μ q is the weight of each classification function, f q (x i ) are the classification decision results output by different support vector machine models.

[0124] In this embodiment, constructing the integrated model includes extracting the initial anomaly detection feature set Φ(x i ) and the output classification result P of the multi-layer neural network model NN (y i =k|x i ) to generate an integrated feature vector:

[0125]

[0126] in, is the final prediction output of the integrated model, σ is the activation function, W ensemble is the weight matrix of the integrated model, α and β are the integration coefficients, corresponding to the weights of the semi-supervised model features and the multi-layer neural network output, respectively, b ensemble is the bias term.

[0127] In this implementation, S5 includes:

[0128] S51, construct a training data set containing normal data and abnormal data, the training data set is:

[0129] D train ={(x1,y1),(x2,y2),…,(x n ,y n )};

[0130] Among them, x i is the i-th data point, y i is the corresponding label, y i =0 indicates normal data, y i = 1 indicates abnormal data. Each data point in the training data set undergoes pre-processing and feature extraction to generate an integrated feature vector ψ(x i );

[0131] S52, training data set D train Input the integrated model and calculate the predicted output of the model through forward propagation The output of the ensemble model represents the data point x i Probability of belonging to the anomaly class:

[0132]

[0133] Among them, σ is a nonlinear activation function, is the weight matrix of the jth class of the ensemble model, ψ (j) (x i ) is the feature output of the jth class, is the bias term, δ j is the regularization coefficient, is the gradient regularization term of the integrated model feature space;

[0134] S53. Define the binary cross entropy loss function to measure the error between the predicted output of the integrated model and the true label. The loss function is defined as:

[0135]

[0136] Where L is the total loss, is the abnormal probability predicted by the integrated model, y i is the true label, λ is the regularization coefficient, It is the L2 regularization term to prevent the model from overfitting;

[0137] S54. Calculate the loss function L for the integrated model weight through the back propagation algorithm and bias The gradient update is:

[0138]

[0139]

[0140] Where η is the learning rate, and are the gradients of the loss function with respect to weights and biases, respectively.

[0141] In this implementation, S6 includes:

[0142] S61, applying the trained integrated model to the target data set D to be detected test , feature extraction is performed on each data point in the target data set to be detected, and the extracted feature is represented as ψ test (x i );

[0143] S63, the eigenvector ψ test (x i ) is input into the integrated model, and the integrated model is used to calculate the data x i Probability of belonging to the anomaly class:

[0144] P ensemble (y i =1|x i )=σ(W ensemble ·ψ test (x i )+b ensemble );

[0145] Among them, P ensemble (y i =1|x i ) represents the data point x i The probability of belonging to the abnormal category, W ensemble is the weight matrix of the integrated model, b ensemble is the bias term, σ is the activation function;

[0146] S64. According to the output results of the integrated model, determine the data point x i Is it abnormal data? Set the threshold τ for abnormal detection. If P ensemble (y i =1|x i )≥τ, then we can judge the data point x i It is abnormal data, otherwise it is normal data, and the final output is the preliminary test result:

[0147]

[0148] in, For data point x i τ is the classification threshold of anomaly detection, which is used to distinguish normal data from abnormal data.

[0149] In this implementation, S7 includes:

[0150] S71. The abnormal data set detected initially Input into the classification model, k is the number of abnormal data points, and the classification model is used to sub-classify the abnormal data;

[0151] S72. Classify the abnormal data using a classification model. The output of the classification model is the probability distribution of each abnormal data point belonging to a specific abnormal type. The classification result is expressed as:

[0152]

[0153] Among them, P class (y i =c|x i ) represents an abnormal data point x i The probability of belonging to the cth type of anomaly, is the weight matrix of the classification model, is the bias term of the cth class, and C is the total number of abnormal types;

[0154] S73, each abnormal data point x is classified according to the output probability of the classification model. i The abnormal category with the highest probability value is assigned to generate the classification result, which is expressed as:

[0155]

[0156] in, is an abnormal data point x i The final classification result indicates the abnormal category to which the data point belongs.

[0157] Embodiment 1:

[0158] In a large Internet company, the focus of the network security department is to detect potential network attack behaviors. Especially when faced with massive user behavior data, how to detect and classify abnormal behaviors in a timely and efficient manner is one of the core tasks. The method of the present invention is verified in a large Internet company.

[0159] Time: August 1-10, 2024;

[0160] Location: Company’s global data center;

[0161] Background: The company receives millions of user requests every second. In daily operations, 80% of the traffic comes from normal user operations, and the remaining 20% ​​of the traffic may be mixed with malicious attacks. In the past period of time, the traditional rule-based detection method has a high false positive rate, especially when dealing with DDoS, SQL injection and other attacks, the false positive problem is serious.

[0162] In this application, the security team used the method of the present invention to perform anomaly detection and classification on user behavior data from August 1 to August 10. The data set included user access logs, IP addresses, packet sizes, response times, request types, source ports, and destination ports, totaling 50 million records.

[0163] The actual abnormal data detection process:

[0164] Case 1: Detection of DDoS attacks:

[0165] At 3:20 p.m. on August 3, 2024, the system discovered a host with a source IP address of "192.168.1.100". Its packet request frequency was abnormal and the packet size was significantly different from normal requests. Through the integrated model, the anomaly detection score exceeded the set threshold of 0.8 (the score was 0.92). Further analysis showed that the host source port was 443, the destination port was 80, and the packet length showed extreme changes. The system generated an attack report, recording the attack source information, time and characteristics, and sent a warning signal to the network administrator in real time. The administrator immediately banned the IP address, preventing further access to the host, and the attack was identified as a DDoS attack.

[0166] Case 2: Detection of SQL injection attacks:

[0167] At 10:05 am on August 5, 2024, the system captured a user sending multiple abnormal SQL query requests to the database. The user's source IP address was "172.16.10.55" and the anomaly score was 0.88, which exceeded the set threshold of 0.8. The abnormal characteristics included continuous sending of complex SQL query statements, and the query content in the data packet contained a large number of special symbols, such as "'OR 1=1--". Through further analysis, the system found that the behavior was consistent with the SQL injection attack mode and generated an attack analysis report. The system quickly issued an alarm to the administrator, who immediately restricted the IP's access rights and cleared the affected database.

[0168] Case 3: Detection of malicious login attempts:

[0169] At 2:40 a.m. on August 7, 2024, the security system detected abnormal behavior of multiple failed logins from the host with source IP address "10.0.0.200". The host attempted more than 100 login requests within 10 minutes, including different username and password combinations. The anomaly score reached 0.95. The system identified this as a malicious login attack, generated an attack report in time, and sent an alert to the administrator. The administrator immediately blocked the IP address and prevented further login attempts.

[0170] To verify the effectiveness of the method of the present invention, the system compared the detection results with the traditional rule-based anomaly detection method. The specific data are as follows:

[0171] Table 1 Comparison data between the method of the present invention and the traditional detection method

[0172]

[0173] As can be seen from Table 1, the method of the present invention has obvious advantages over the traditional method. The detection accuracy is improved to 98.9%, and the false alarm rate and missed alarm rate are significantly reduced. In terms of processing efficiency, the method of the present invention only takes 80 milliseconds to process every 10,000 data, which is much better than the traditional method.

[0174] In the case of DDoS attack detection, the traditional rule-based method is prone to underreporting in high-frequency and low-volume attacks due to its fixed setting of traffic thresholds, which ultimately leads to some attacks not being detected in time. The present invention not only captures traffic anomalies by integrating the feature extraction capabilities in the model, but also further confirms that these behaviors belong to DDoS attacks through feature enhancement. In the detection of SQL injection attacks, although the shallow model has improved the accuracy to a certain extent, false alarms still exist due to the model's insufficient ability to extract complex features. The present invention combines deep neural networks with semi-supervised models to accurately classify SQL injection and malicious login attacks and issue timely warnings.

[0175] The present invention effectively improves the accuracy and efficiency of abnormal data detection in large-scale user behavior data by combining deep learning with semi-supervised learning. The system can not only detect potential security threats in a timely manner, but also help the security team quickly locate the source and type of attacks through further classification analysis, thereby improving the security of the overall network.

[0176] The present invention constructs a multi-layer neural network model, and extracts features of high-dimensional and nonlinear data through an input layer, several hidden layers, and an output layer in sequence. By introducing the ReLU activation function and a multi-branch structure, the model can capture subtle abnormal patterns in complex data. The powerful learning ability of the multi-layer neural network is combined with the semi-supervised learning model of Scikit-learn, which can automatically extract abnormal features from partially labeled and unlabeled data, further improving the accuracy of detection, and has a significant recognition effect on subtle abnormal points that are difficult to detect.

[0177] The present invention integrates the preliminary detection results of the multi-layer neural network with the semi-supervised support vector machine and the K-nearest neighbor algorithm by constructing an integrated model, thereby effectively improving the classification accuracy of abnormal data. In scenarios with complex types of abnormal data, the Softmax function and a multi-category classification algorithm are used to calculate the probability distribution of abnormal points, thereby improving the classification ability of multi-category abnormal data, avoiding the misclassification phenomenon in traditional methods, and being able to more accurately distinguish different types of abnormal data.

[0178] The present invention continuously updates the weight parameters of the integrated model through an online learning mechanism and weight optimization based on the gradient descent method, so that it can adaptively adjust the classification and detection rules when facing a new data environment, ensuring the continuous optimization performance of the model. Unlike the static model in the prior art, the combination of self-supervised learning and semi-supervised learning can maintain high performance in rapidly changing data scenarios, respond to new anomaly types in a timely manner, ensure that the detection accuracy of the model does not decay over time, and significantly improve the robustness and adaptability of the system in practical applications.

[0179] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for abnormal data detection and classification based on deep learning, characterized in that: The steps include: S1. Obtain a target data set and preprocess the target data set, where the target data set includes user access logs, IP addresses, data packet sizes, response time, request types, source ports, and destination ports; S2, construct a multi-layer neural network model, which includes an input layer, several hidden layers and an output layer in sequence; S3. Perform preliminary training on the target data set using a Scikit-learn semi-supervised model, wherein the Scikit-learn semi-supervised model includes a semi-supervised k-nearest neighbor model and a semi-supervised support vector machine model, respectively training the target data set, and extracting preliminary anomaly detection features from the target data set with partial annotations and without annotations; S4, integrating the preliminary anomaly detection features of the Scikit-learn semi-supervised model with the anomaly detection results output by the multi-layer neural network model to build an integrated model; The construction of the integrated model includes extracting the initial anomaly detection feature set Φ(x i ) and the output classification result P of the multi-layer neural network model NN (y i =k|x i ) to generate an integrated feature vector: in, is the final prediction output of the integrated model, σ is the activation function, W ensemble is the weight matrix of the integrated model, α and β are the integration coefficients, corresponding to the weights of the Scikit-learn semi-supervised model features and the multi-layer neural network output, respectively, b ensemble is the bias term; S5. Train the integrated model by supervised learning. The training data set contains normal data and abnormal data. The integrated model weight is optimized based on the classification loss function. S6. Apply the trained integrated model to the target data set to be detected, extract features from the target data set to be detected, and use the integrated model to determine whether the data point is abnormal data, and output a preliminary detection result; S7, further classify the detected abnormal data, divide the abnormal data into different types, and generate classification results; S8. Monitor the newly input data stream in real time and update the integrated model based on the detection and classification results.

2. The method for abnormal data detection and classification based on deep learning according to claim 1, characterized in that: The S1 includes: S11. Obtain target data set: D={x1,x2,…,x n }; Among them, x i is the i-th data point in the target data set, i = 1, 2, ..., n, and n is the number of samples in the data set; S12, using a denoising algorithm based on mean filtering to perform data denoising on the target data set D, processing each data point to obtain a denoised target data set D'; S13, performing missing value filling processing on the denoised target data set D', and filling the values ​​at the positions of the missing values ​​by interpolation to obtain the filled target data set D"; S14, use the Min-Max normalization method to normalize the filled target data set D" and convert each data point x" i Map to the interval [0,1] to obtain the preprocessed target data set:

3. The method for abnormal data detection and classification based on deep learning according to claim 1, characterized in that: The S2 includes: S21, constructing an input layer, the input layer is used to receive the preprocessed target data set, and the input layer extracts preliminary features by mapping each data point to a high-dimensional feature space; S22. Construct several hidden layers, where the hidden layers consist of a multi-layer asymmetric neural network. Each hidden layer extracts features from the data and uses a ReLU activation function to perform nonlinear transformation on the input features. S23. Introduce a multi-branch structure in the hidden layer to construct multiple branch networks. Each branch network processes the features of different dimensions of the target data set. The output features of each branch network are integrated to obtain the anomaly detection effect. The output calculation formula of the branch network is: in, and are the outputs of the two branches of the i-th layer, and are the weight matrices of the i-th layer, representing the connection weights between the input features and the hidden layer nodes; and are the bias vectors of the i-th layer branch, respectively, which adjust the linear transformation results of the features; S24, constructing a self-supervised learning module for processing unlabeled data sets, the self-supervised learning module automatically learning abnormal feature patterns and updating the multi-layer neural network model; S25. Construct an output layer for generating anomaly detection results. Use the Softmax function to map the output results of the multi-layer neural network model into multi-category probability distributions for classifying different types of anomaly data. The Softmax function calculates the probability distributions of different anomaly categories based on the characteristics of the input data and generates the final classification results: Among them, P(y=k|x) represents the probability that the input data x belongs to the kth type of abnormal data, W k is the weight matrix of the k-th anomaly in the output layer, and K is the total number of abnormal data categories.

4. The method for abnormal data detection and classification based on deep learning according to claim 1, characterized in that: The S3 includes: S31. Use the semi-supervised k-nearest neighbor model in Scikit-learn to train the target dataset D norm Perform preliminary training. The target data set includes partially labeled data and unlabeled data. The weighted distance between data points is calculated by nonlinear weight distance to extract preliminary anomaly detection features: Among them, d w (x i ,x j ) is the weighted distance, and The data points x i and x j The value of the k-th dimension feature, m is the dimension of the feature, w k is the weight of the k-th dimension feature, α, β, γ are nonlinear coefficients; S32. Use the semi-supervised support vector machine model in Scikit-learn to train the target data set and classify the data points by optimizing the decision boundary formula based on multi-scale regularization: Among them, f(x) is the classification decision function of the input data x, x i is the support vector, K(x i ,x) is the kernel function, α i is the Lagrange multiplier, b is the bias term, n l is the number of labeled data, λ is the regularization coefficient, is the kernel function gradient; S33, integrating the preliminary anomaly detection features extracted by the semi-supervised k-nearest neighbor model and the semi-supervised support vector machine model to generate a preliminary anomaly feature set of the target data set: Among them, Φ(x i ) is the data point x i The comprehensive abnormal features, σ is the nonlinear activation function, P is the number of feature integration branches, ω p is the weight of each integrated branch, Q is the number of classification functions of the support vector machine, μ q is the weight of each classification function, f q (x i ) are the classification decision results output by different support vector machine models.

5. The method for abnormal data detection and classification based on deep learning according to claim 1, characterized in that: The S5 includes: S51, construct a training data set containing normal data and abnormal data, the training data set is: D train ={(x1,y1),(x2,y2),…,(x n ,y n )}; Among them, x i is the i-th data point, y i is the corresponding label, y i =0 indicates normal data, y i = 1 indicates abnormal data. Each data point in the training data set undergoes pre-processing and feature extraction to generate an integrated feature vector ψ(x i ); S52, training data set D train Input the integrated model and calculate the predicted output of the model through forward propagation The output of the ensemble model represents the data point x i Probability of belonging to the anomaly class: Among them, σ is a nonlinear activation function, is the weight matrix of the jth class of the ensemble model, ψ (j) (x i ) is the feature output of the jth class, is the bias term, δ j is the regularization coefficient, is the gradient regularization term of the integrated model feature space; S53. Define the binary cross entropy loss function to measure the error between the predicted output of the integrated model and the true label. The loss function is defined as: Where L is the total loss, is the abnormal probability predicted by the integrated model, y i is the true label, λ is the regularization coefficient, It is the L2 regularization term to prevent the model from overfitting; S54. Calculate the loss function L for the integrated model weight through the back propagation algorithm and bias The gradient update is: Where η is the learning rate, and are the gradients of the loss function with respect to weights and biases, respectively.

6. The method for abnormal data detection and classification based on deep learning according to claim 1, characterized in that: The S6 includes: S61, applying the trained integrated model to the target data set D to be detected test , feature extraction is performed on each data point in the target data set to be detected, and the extracted feature is represented as ψ test (x i ); S63, the eigenvector ψ test (x i ) is input into the integrated model, and the integrated model is used to calculate the data x i Probability of belonging to the anomaly class: P ensemble (y i =1|x i )=σ(W ensemble ·ψ test (x i )+b ensemble ); Among them, P ensemble (y i =1|x i ) represents the data point x i The probability of belonging to the abnormal category, W ensemble is the weight matrix of the integrated model, b ensemble is the bias term, σ is the activation function; S64. According to the output results of the integrated model, determine the data point x i Is it abnormal data? Set the threshold τ for abnormal detection. If P ensemble (y i =1|x i )≥τ, then we can judge the data point x i It is abnormal data, otherwise it is normal data, and the final output is the preliminary test result: in, For data point x i τ is the classification threshold of anomaly detection, which is used to distinguish normal data from abnormal data.

7. The method for abnormal data detection and classification based on deep learning according to claim 1, characterized in that: The S7 includes: S71. The abnormal data set detected initially Input into the classification model, k is the number of abnormal data points, and the classification model is used to sub-classify the abnormal data; S72. Classify the abnormal data using a classification model. The output of the classification model is the probability distribution of each abnormal data point belonging to a specific abnormal type. The classification result is expressed as: Among them, P class (y i =c|x i ) represents an abnormal data point x i The probability of belonging to the cth type of anomaly, is the weight matrix of the classification model, is the bias term of the cth class, and C is the total number of abnormal types; S73, each abnormal data point x is classified according to the output probability of the classification model. i The abnormal category with the highest probability value is assigned to generate the classification result, which is expressed as: in, is an abnormal data point x i The final classification result indicates the abnormal category to which the data point belongs.

Images