Abnormal behavior safety analysis method based on artificial intelligence
Through an abnormal behavior security analysis method based on artificial intelligence, combined with data cleaning, feature extraction and model training, real-time detection and dynamic adjustment, the problems of high false alarm rate and large false alarm rate of traditional methods are solved, and efficient and accurate abnormalities are achieved Detection and rapid response improve network security and user trust.
Patent Information
- Application Number
- CN202411259193.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-12-13
AI Technical Summary
Traditional abnormal behavior detection methods have high false alarm rates and large false alarm rates, making it difficult to adapt to complex and changeable network environments and cannot effectively protect network security.
Adopt anomaly behavior security analysis method based on artificial intelligence, including data collection, cleaning, feature extraction, model training, real-time detection and dynamic adjustment, and train models through supervised or unsupervised learning algorithms, combining convolutional neural networks and recurrent neural networks, real-time Data is collected for prediction and adapted to environmental changes through dynamic adjustment and multi-model fusion strategies.
Improve the accuracy and real-time nature of abnormal detection, reduce false alarms and missed reports, enhance the robustness and adaptability of the system, reduce computing complexity and maintenance costs, ensure user data and privacy security, and improve users Trust and experience.
Smart Images
Figure CN119128878A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an abnormal behavior security analysis method based on artificial intelligence. Background Art
[0002] Artificial intelligence is an important driving force for a new round of scientific and technological revolution and industrial transformation. It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is an important part of the intelligent discipline. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is a very broad science, including robots, language recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc.
[0003] With the rapid development of network technology, network security issues are becoming increasingly severe, and abnormal behavior detection has become an important means to protect network security. Traditional abnormal behavior detection methods often rely on simple rule matching or statistical analysis, which have problems such as high false positive rate, large false negative rate, and difficulty in adapting to complex and changing network environments. Therefore, an abnormal behavior security analysis method based on artificial intelligence is proposed. Summary of the invention
[0004] The purpose of the present invention is to provide an abnormal behavior security analysis method based on artificial intelligence to solve the problems raised in the above background technology.
[0005] To achieve the above object, the present invention provides the following technical solution: an abnormal behavior security analysis method based on artificial intelligence, comprising the following steps:
[0006] S1. Data collection: Collect network traffic data, including but not limited to IP addresses, port numbers, protocol types, and data packets;
[0007] S2, clean the data, remove invalid or redundant information, and perform denoising, standardization and normalization on the data;
[0008] S3, Feature extraction: Extract key features that help distinguish normal and abnormal behaviors based on the characteristics of network traffic;
[0009] S4. Model training: Use supervised or unsupervised learning algorithms to train the model;
[0010] S5, anomaly detection: input the real-time collected data into the trained model for prediction, and the output result is used to determine whether it is abnormal behavior;
[0011] S6. Dynamic adjustment: Update the model based on new data so that the model can continuously adapt to environmental changes;
[0012] S7. Security response: When abnormal behavior is detected, execute the corresponding security policy.
[0013] Preferably, in S2, the calculation formula of the denoising y(n) is:
[0014] y(n)=median{x(nN),...,x(n),...,x(n+N)}
[0015] Among them, x(n) is the original data, y(n) is the processed data, and N is the size of the window.
[0016] Preferably, the above-mentioned standardized X norm The data calculation formula is:
[0017]
[0018] Among them, X is the original data, u is the mean, and σ is the standard deviation.
[0019] Preferably, the calculation formula of the normalized x' is:
[0020]
[0021] Here, x is the original data.
[0022] Preferably, in S3, the key feature uses orthogonal transformation to convert the correlated high-dimensional variables into a set of linearly uncorrelated variables, and the conversion formula is:
[0023] X n×p =T n×m P p×m +E n×p
[0024] Among them, X is the original data matrix, T is the score matrix, P is the loading matrix, and E is the residual matrix.
[0025] Preferably, in S4, the supervised learning requires a labeled data set; for unsupervised learning, no labeled data is required;
[0026] The model training includes convolutional neural network and recurrent neural network.
[0027] Preferably, the calculation formula of the convolutional neural network y is as follows:
[0028]
[0029] Where b is the bias term, ω i is the convolution kernel weight, x iis a part of the input data, and σ is a nonlinear activation function.
[0030] Preferably, the above-mentioned recurrent neural network is used to process sequence data, and the recurrent neural network h t The calculation formula is as follows:
[0031] h t =σ(Ux t +Wh t-1 )
[0032] Among them, h t is the hidden state at time step t, U and W are weight matrices, and x t is the input.
[0033] Preferably, in S5, the calculation formula of the anomaly score AnomalyScore in the anomaly detection is:
[0034]
[0035] Among them, x j is the value of the current behavior on feature j, u j is the average value of the normal behavior pattern on feature j, and m is the number of features;
[0036] The higher the anomaly score, the more abnormal the behavior.
[0037] Preferably, the above.
[0038] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:
[0039] 1. The quality of basic data for analysis is ensured through comprehensive and accurate data collection, as well as in-depth mining and refined processing. In the process of data cleaning, invalid or redundant information is removed, and denoising, standardization and normalization are performed to further improve the purity and consistency of the data, which helps to reduce false positives and false negatives. In the feature extraction stage, key features are extracted by combining the characteristics of network traffic and professional knowledge, and the data dimension is reduced through methods such as orthogonal transformation, which improves the effectiveness of features and the generalization ability of the model.
[0040] Second, by collecting data in real time and inputting it into the trained model for prediction, it can quickly determine whether it is an abnormal behavior, thereby improving the real-time performance of detection. By adopting a distributed computing framework and efficient data processing algorithms, it can process large-scale data streams and ensure low latency and high throughput. When abnormal behavior is detected, it can immediately trigger the preset security policies, such as blocking abnormal traffic, isolating infected devices, etc., to quickly respond to threats.
[0041] 3. Through the dynamic adjustment mechanism, the model is updated according to the new data, so that the model can continuously adapt to environmental changes, improving the robustness and adaptability of the system. The introduction of dynamic thresholds and multi-model fusion strategies can respond more flexibly to changes in network traffic and improve the accuracy and stability of anomaly detection. Incremental learning and online learning techniques allow the model to gradually update its parameters when receiving new data, without the need to retrain the entire model, reducing training time and computing resource consumption.
[0042] 4. Through feature dimensionality reduction and model optimization, the computational complexity and storage requirements are reduced, resource utilization efficiency is improved, model parameters are updated in real time and dynamic adjustment strategies are implemented, unnecessary comprehensive retraining is avoided, maintenance costs are reduced, anomaly detection and security response mechanisms can reduce business interruptions and losses caused by security incidents, and improve overall cost-effectiveness.
[0043] 5. Through timely anomaly detection and response, user data and privacy security can be protected, and user trust in the system can be improved. Accurate anomaly identification and reasonable security policy execution can reduce the impact of false alarms and erroneous operations on users and improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 It is a schematic diagram of the method of the present invention; DETAILED DESCRIPTION
[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0047] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which this application can be implemented. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in this application without affecting the effects and purposes that can be achieved by this application.
[0048] Example
[0049] See also Figure 1 The present invention provides a technical solution: an abnormal behavior security analysis method based on artificial intelligence, comprising the following steps:
[0050] S1. Data collection: Collect network traffic data, including but not limited to IP addresses, port numbers, protocol types, and data packets;
[0051] Data collection is not limited to the collection of basic information, but also requires in-depth mining and refined processing to ensure the comprehensiveness and accuracy of the data.
[0052] S2, clean the data, remove invalid or redundant information, and perform denoising, standardization and normalization on the data;
[0053] Set reasonable thresholds or rules to identify data records that are obviously erroneous, missing, or illogical, such as incorrect IP address format, port number out of range, abnormal data packet size, etc.
[0054] Check whether there are duplicate records or fields in the dataset, and remove duplicates for identical records; for redundant fields, if they do not affect subsequent analysis, you can choose to delete or merge them.
[0055] Outliers in the data, such as extremely large or small values, can be processed by deleting them, replacing them with the mean / median / mode, or using interpolation.
[0056] The calculation formula for denoising y(n) is:
[0057] y(n)=median{x(nN),...,x(n),...,x(n+N)}
[0058] Among them, x(n) is the original data, y(n) is the processed data, and N is the size of the window. After denoising, the denoising effect should be evaluated to ensure that the original characteristics of the data are retained while removing the noise. This can be achieved by comparing the data distribution before and after denoising, signal quality indicators, or the accuracy of subsequent analysis results.
[0059] StandardizationX norm The data calculation formula is:
[0060]
[0061] Among them, X is the original data, u is the mean, and σ is the standard deviation. It should be ensured that they are statistical results based on the entire data set or a specific subset. For time series data, it may be necessary to consider the local mean and standard deviation within the time window. If the standard deviation of a feature is zero, that is, all values are the same, the standardization formula cannot be directly applied. You can choose to delete the feature, treat it as a constant feature, or use other preprocessing methods.
[0062] The calculation formula for normalized x' is:
[0063]
[0064] Among them, x is the original data. During the normalization process, if there are extreme values in the data, that is, values far away from most data points, they may have a greater impact on the normalization result. You can consider denoising or truncating the data first to reduce the impact of extreme values.
[0065] S3. Feature extraction: Extract key features that help distinguish normal from abnormal behaviors based on the characteristics of network traffic. Identify a preliminary set of features that may help distinguish normal from abnormal behaviors based on the basic attributes and behavior patterns of network traffic, including traffic size, packet rate, protocol type distribution, session duration, IP address pair frequency, etc.
[0066] In addition to statistical and machine learning-based methods, expertise in the field of network security can also be combined to guide feature engineering. Based on common patterns of network attacks, such as scanning, injection, DDoS, etc., feature sets that can capture the characteristics of these patterns can be designed.
[0067] The key feature uses orthogonal transformation to convert correlated high-dimensional variables into a set of linearly unrelated variables, and converts high-dimensional variables in the original data into a set of linearly unrelated variables, namely principal components or independent components, through linear transformation. The conversion formula is:
[0068] X n×p =T n×m P p×m +E n×p
[0069] Among them, X is the original data matrix, T is the score matrix, P is the loading matrix, and E is the residual matrix;
[0070] According to the needs of actual applications and the characteristics of the data, the first few principal components or independent components with the largest variance contribution can be selected as the new feature set, which can significantly reduce the dimension of the data while retaining most of the useful information. The effect of feature dimensionality reduction can be evaluated by comparing the performance of the anomaly detection model built using the feature set before and after dimensionality reduction, such as accuracy, recall rate, F1 score, etc. According to the evaluation results, the feature selection, extraction and dimensionality reduction strategies can be iteratively adjusted to further optimize the quality of the feature set and the performance of the model.
[0071] S4. Model training: Use supervised or unsupervised learning algorithms to train the model;
[0072] For supervised learning, a labeled data set is required; for unsupervised learning, labeled data is not required;
[0073] Model training includes convolutional neural networks and recurrent neural networks. In supervised learning, in addition to convolutional neural networks (CNN) and recurrent neural networks (RNN), other types of models can also be considered, such as decision trees, random forests, support vector machines (SVM) or deep belief networks (DBN). Each model has its own unique advantages and applicable scenarios, so it is necessary to select the most appropriate model based on the characteristics of the data and the requirements of the problem.
[0074] CNN is good at extracting spatial hierarchical features from raw data, and is particularly effective when processing image data. In network traffic analysis, although the data is not in a typical image format, the advantages of CNN can be exploited by converting the traffic data into a two-dimensional matrix, such as a time-frequency graph, a protocol distribution graph, etc. In addition, CNN can also be used to capture patterns or structures inside data packets;
[0075] RNNs are a natural choice for processing sequential data because they are able to capture temporal dependencies and long-term memory effects in the data. In network traffic analysis, RNNs can be used to analyze session data, traffic time series, or packet sequences to identify abnormal behavior patterns.
[0076] In unsupervised learning, commonly used anomaly detection algorithms include clustering algorithms such as K-means, DBSCAN, density estimation algorithms such as LOF, KDE, and autoencoders, which can identify points that are significantly different from most data points as anomalies. By building deep learning models such as autoencoders (AE) or variational autoencoders (VAE), the advantages of these two networks are combined. CNN is used to extract the spatial features of the input data, and then RNN is used to process the time series of these features. Finally, the original data is reconstructed through the decoder. By comparing the differences between the original data and the reconstructed data, anomalies can be identified.
[0077] The calculation formula of the convolutional neural network y is as follows:
[0078]
[0079] Where b is the bias term, ω i is the convolution kernel weight, x i is a part of the input data, and σ is a nonlinear activation function.
[0080] Recurrent neural networks are used to process sequence data. t The calculation formula is as follows:
[0081] h t =σ(Ux t +Wh t-1 )
[0082] Among them, h t is the hidden state at time step t, U and W are weight matrices, and x t is the input.
[0083] When training the model, it is necessary to use appropriate optimization algorithms, such as gradient descent, Adam, RMSprop, etc., to minimize loss functions, such as cross entropy loss, mean square error, etc. At the same time, in order to prevent overfitting, it is also necessary to use regularization techniques, such as L1 / L2 regularization, Dropout, etc., and strategies such as Early Stopping, and find the optimal hyperparameter combination through methods such as cross validation, grid search or Bayesian optimization.
[0084] S5, anomaly detection: input the real-time collected data into the trained model for prediction, and the output result is used to determine whether it is abnormal behavior;
[0085] The calculation formula for the anomaly score AnomalyScore in anomaly detection is:
[0086]
[0087] Among them, x j is the value of the current behavior on feature j, u j is the average value of the normal behavior pattern on feature j, and m is the number of features;
[0088] A higher anomaly score indicates more abnormal behavior.
[0089] In the calculation formula of the anomaly score, all features are considered to have equal weights. In practical applications, some features may reflect abnormal behavior better than other features. It is possible to consider introducing feature weights to reflect the importance of different features in anomaly detection. The weights can be determined by feature selection algorithms, such as information gain, chi-square test, etc., or model-based importance assessment, such as feature importance of random forests.
[0090] The threshold of the anomaly score is usually used to determine whether the behavior is abnormal. The fixed threshold may not be suitable for all situations, especially when the network traffic pattern changes. You can consider using a dynamic threshold, which can be adjusted according to the changing trend of historical data or the statistical characteristics of real-time data. In order to improve the accuracy and robustness of anomaly detection, you can consider using multiple models for fusion. Each model can be trained based on different algorithms or feature sets and generate its own anomaly score. Then, the prediction results of multiple models can be combined through weighted average, voting mechanism or more complex fusion strategies to obtain the final anomaly judgment.
[0091] In real-time anomaly detection, it is necessary to efficiently process the continuously influx of data streams, which requires low latency and high throughput capabilities. To this end, distributed computing frameworks such as Apache Kafka and Apache Spark Streaming can be used to process data streams in parallel and use efficient data structures and algorithms to accelerate the computing process. As new data is continuously added, the model may need to be updated to adapt to new network traffic patterns. Incremental learning is an online learning method that allows the model to gradually update its parameters when new data is received without retraining the entire model, which can significantly improve the adaptability and real-time performance of the model.
[0092] When abnormal behavior is detected, an alarm should be issued to relevant personnel in a timely manner. The alarm information should contain sufficient details to quickly locate the problem and take corresponding measures. In addition, different alarm levels can be set to distinguish different types of abnormal behaviors. In order to continuously improve the anomaly detection system, an effective feedback mechanism should be established, including collecting and analyzing false positives, that is, misjudging normal behavior as abnormal, and missed negatives, that is, failing to detect abnormal behavior, and adjusting model parameters or optimizing algorithms based on feedback results;
[0093] In order to evaluate the performance of the anomaly detection system, it is necessary to select appropriate evaluation indicators. Commonly used indicators include accuracy, recall, F1 score, false alarm rate and false negative rate, which can help us fully understand the performance of the system and find potential room for improvement. In order to verify the effectiveness of the system, it can be benchmarked with other anomaly detection methods or systems. By comparing the performance of different methods or systems on the same data set, their respective advantages and disadvantages can be evaluated, and guidance can be provided for subsequent optimization.
[0094] S6. Dynamic adjustment: Update the model based on new data so that the model can continuously adapt to environmental changes;
[0095] When new data arrives, instead of retraining the entire model, only some parameters or components of the model are updated to quickly adapt to the new data, which can significantly reduce training time and computing resource consumption. The model is trained directly on the new data and updates its internal state in real time. Online learning is particularly suitable for data flow environments and can instantly reflect the latest patterns of network traffic;
[0096] The threshold in the anomaly detection system can be adaptively adjusted according to the statistical characteristics of the new data. A sliding window can be set to calculate the average and standard deviation of normal behavior in the recent period of time, and the threshold of the anomaly score can be dynamically adjusted accordingly;
[0097] In a multi-model system, the most appropriate model can be selected for anomaly detection based on the characteristics of the current network traffic. When it is detected that the model performance has degraded or is no longer suitable for the current environment, it can automatically switch to other models or combine multiple models to improve the detection effect.
[0098] S7, Security response: When abnormal behavior is detected, the corresponding security policy is executed. Once abnormal behavior is detected, the preset security policy is immediately triggered;
[0099] Including blocking abnormal traffic, isolating infected devices, sending alarm notifications, etc., adopting a layered defense strategy, dividing the network into different security areas, and deploying different security control measures for different areas. When abnormal behavior is detected, corresponding defense measures can be taken according to the scope and severity of the behavior.
[0100] In summary, the quality of basic data for analysis is ensured through comprehensive and accurate data collection, as well as deep mining and refined processing. In the process of data cleaning, invalid or redundant information is removed, and denoising, standardization and normalization are performed to further improve the purity and consistency of the data, which helps to reduce false positives and negatives. In the feature extraction stage, key features are extracted by combining the characteristics of network traffic and professional knowledge, and the data dimension is reduced through methods such as orthogonal transformation, which improves the effectiveness of features and the generalization ability of the model.
[0101] By collecting data in real time and inputting it into a trained model for prediction, it is possible to quickly determine whether it is abnormal behavior, thereby improving the real-time nature of detection. By adopting a distributed computing framework and efficient data processing algorithms, it is able to process large-scale data streams, ensuring low latency and high throughput. When abnormal behavior is detected, it can immediately trigger preset security policies, such as blocking abnormal traffic, isolating infected devices, etc., to quickly respond to threats.
[0102] Through the dynamic adjustment mechanism, the model is updated according to the new data, so that the model can continuously adapt to environmental changes, improving the robustness and adaptability of the system. The introduction of dynamic thresholds and multi-model fusion strategies can more flexibly respond to changes in network traffic and improve the accuracy and stability of anomaly detection. Incremental learning and online learning techniques allow the model to gradually update its parameters when receiving new data, without the need to retrain the entire model, reducing training time and computing resource consumption.
[0103] Through feature dimensionality reduction and model optimization, the computational complexity and storage requirements are reduced, resource utilization efficiency is improved, model parameters are updated in real time and strategies are adjusted dynamically, unnecessary comprehensive retraining is avoided, maintenance costs are reduced, anomaly detection and security response mechanisms can reduce business interruptions and losses caused by security incidents, and improve overall cost-effectiveness.
[0104] Through timely anomaly detection and response, user data and privacy security can be protected and user trust in the system can be improved. Accurate anomaly identification and reasonable security policy execution can reduce the impact of false alarms and erroneous operations on users and improve user experience.
[0105] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.
Claims
1. An abnormal behavior security analysis method based on artificial intelligence, characterized in that: The following steps are involved: S1. Data collection: Collect network traffic data, including but not limited to IP addresses, port numbers, protocol types, and data packets; S2, clean the data, remove invalid or redundant information, and perform denoising, standardization and normalization on the data; S3, Feature extraction: Extract key features that help distinguish normal and abnormal behaviors based on the characteristics of network traffic; S4. Model training: Use supervised or unsupervised learning algorithms to train the model; S5, anomaly detection: input the real-time collected data into the trained model for prediction, and the output result is used to determine whether it is abnormal behavior; S6. Dynamic adjustment: Update the model based on new data so that the model can continuously adapt to environmental changes; S7. Security response: When abnormal behavior is detected, execute the corresponding security policy.
2. The abnormal behavior security analysis method based on artificial intelligence according to claim 1 is characterized by: In S2, the calculation formula of the denoising y(n) is: y(n)=median{x(nN),...,x(n),...,x(n+N)} Among them, x(n) is the original data, y(n) is the processed data, and N is the size of the window.
3. The abnormal behavior security analysis method based on artificial intelligence according to claim 2 is characterized by: The standardized X norm The data calculation formula is: Among them, X is the original data, u is the mean, and σ is the standard deviation.
4. The abnormal behavior security analysis method based on artificial intelligence according to claim 3 is characterized by: The calculation formula of the normalized x' is: Here, x is the original data.
5. The abnormal behavior security analysis method based on artificial intelligence according to claim 1 is characterized by: In S3, the key feature uses orthogonal transformation to transform the correlated high-dimensional variables into a set of linearly uncorrelated variables. The transformation formula is: X n×p =T n×m P p×m +E n×p Among them, X is the original data matrix, T is the score matrix, P is the loading matrix, and E is the residual matrix.
6. The abnormal behavior security analysis method based on artificial intelligence according to claim 1 is characterized by: In S4, the supervised learning requires a labeled data set; for unsupervised learning, no labeled data is required; The model training includes convolutional neural network and recurrent neural network.
7. The abnormal behavior security analysis method based on artificial intelligence according to claim 6 is characterized by: The calculation formula of the convolutional neural network y is as follows: Where b is the bias term, ω i is the convolution kernel weight, x i is a part of the input data, and σ is a nonlinear activation function.
8. The abnormal behavior security analysis method based on artificial intelligence according to claim 7 is characterized by: The recurrent neural network is used to process sequence data. t The calculation formula is as follows: h t =σ(Ux t +Wh t-1 ) Among them, h t is the hidden state at time step t, U and W are weight matrices, and x t is the input.
9. The abnormal behavior security analysis method based on artificial intelligence according to claim 1 is characterized by: In S5, the calculation formula of the anomaly score AnomalyScore in the anomaly detection is: Among them, x j is the value of the current behavior on feature j, u j is the average value of the normal behavior pattern on feature j, and m is the number of features; The higher the anomaly score, the more abnormal the behavior.