Abnormal access data behavior monitoring method based on collaborative framework dynamic feature mining

Abstract

The invention discloses an abnormal access data behavior monitoring method based on collaborative framework dynamic feature mining, a collaborative framework machine learning model is established, and the model can fully solve the problems that an abnormal access data rule base is not updated in time and a long-term dependency relationship is difficult to accurately capture. The model integrates the advantages of UBER, LSTM and isolated forest algorithms. The method comprises the steps of data acquisition, dynamic feature mining, model training and fusion, anomaly detection and response and the like, and whether the current access behavior is abnormal or not is judged. And efficient and accurate monitoring of abnormal data access behaviors can be realized by utilizing the machine learning model of the collaborative architecture.

Description

Technical Field

[0001] The present invention relates to the technical field of power data security monitoring, and in particular to a method for monitoring abnormal access data behavior based on collaborative framework dynamic feature mining. Background Art

[0002] The maturity of power big data technology will drive companies to shift from qualitative to quantitative data analysis, promoting more refined, coordinated, and efficient business management. Big data will undoubtedly play an increasingly important role in the construction of a global energy internet. However, new technological applications and models inevitably carry potential risks. If big data is tampered with or leaked, it will have a significant impact on power grid production, operations, management, and customer services, and even worse, it could affect social stability and national security.

[0003] Currently, existing methods for monitoring abnormal data access behaviors have many shortcomings.

[0004] (1) Traditional rule-based monitoring methods rely on pre-set rule bases and are difficult to cope with ever-changing attack patterns and complex and diverse abnormal behaviors. Once new abnormal access methods appear, failure to update rules in a timely manner will lead to missed reports.

[0005] (2) Although machine learning-based monitoring methods can automatically learn data features to a certain extent, single algorithms often have limitations. For example, when faced with complex time series data, it is difficult to accurately capture long-term dependencies; when dealing with a small number of abnormal points in a large amount of normal data, misjudgment is prone to occur.

[0006] Therefore, it is necessary to propose a method that can avoid the impact of untimely updates of the abnormal access data rule library and the difficulty in accurately capturing long-term dependencies, fully utilize the advantages of UBER and fusion machine learning algorithms, and improve the accuracy and efficiency of abnormal access data behavior monitoring. Summary of the Invention

[0007] To address the problems of the existing technology, the present invention provides a method for monitoring abnormal access data behavior based on dynamic feature mining within a collaborative framework. This method establishes a collaborative machine learning model that can fully address the issues of delayed updates to the abnormal access data rule base and the difficulty in accurately capturing long-term dependencies. The model integrates the advantages of the UBER, LSTM, and Isolation Forest algorithms. It covers multiple steps, including data collection, dynamic feature mining, model training and fusion, and anomaly detection and response, to determine whether the current access behavior is abnormal. The collaborative machine learning model enables efficient and accurate monitoring of abnormal access data behavior.

[0008] The method for monitoring abnormal access data behavior based on collaborative framework dynamic feature mining described in the present invention is implemented by the following steps: 1. A method for monitoring abnormal access data behavior based on collaborative framework dynamic feature mining, comprising the following steps: S1, building a collaborative framework; S101, Design the Data Collection Layer: Based on UBER technology, deploy data collection modules at key nodes in the network, such as routers, servers, and databases. These modules are responsible for collecting access data generated by these devices, including network traffic data, system logs, and database operation records. The collection modules transmit the data in real time to the data storage center according to a unified data format and transmission protocol. S102, build the feature mining and analysis layer: This layer integrates LSTM and the Isolation Forest algorithm; LSTM is used to analyze time series data and mine dynamic features in the time dimension; the Isolation Forest algorithm is used to analyze the overall distribution of data and identify outlier features; a data preprocessing unit is set up to clean, denoise, and normalize the collected data to provide high-quality data for subsequent feature mining and model training; S103, establish the model training and monitoring layer: deploy the LSTM model and the isolation forest model, and design a model fusion module; divide the data processed by the feature mining and analysis layer into a training set and a test set, and train and optimize the LSTM model and the isolation forest model respectively; during the training process, adjust model parameters, such as the number of hidden layer units in the LSTM and the number of trees in the isolation forest, to improve model performance. The trained model monitors the input access data features in real time, and uses the model fusion module to determine whether there is any abnormal access behavior; S2: dynamic feature mining; S201, Time Series Feature Mining (LSTM-based): Read access data arranged in time series, such as access time and access frequency, from the data storage center; pre-process this data and input it into the LSTM model; the LSTM model captures the periodicity and trend changes in access behavior by learning the time series features of the data; S202, outlier feature mining based on the Isolation Forest Algorithm: The original access data, which has not been processed into a time series, is input into the Isolation Forest Algorithm. The Isolation Forest Algorithm constructs a tree structure based on the data characteristics and calculates the path length from each data point to the root node. Data points with longer path lengths are considered outliers, and the access behavior characteristics corresponding to these outliers are the mined outlier features. S3: Model training and fusion; S301, LSTM model training: Use the training data processed through time series feature mining to train the LSTM model. During the training process, use an appropriate loss function and continuously adjust the model parameters through the backpropagation algorithm to enable the LSTM model to accurately learn the time series characteristics of normal access behavior and establish a normal behavior model. S302, Isolation Forest Model Training: Input the training data after outlier feature mining into the Isolation Forest Model for training; During the training process, adjust the parameters of the Isolation Forest Model, such as the number of trees and the depth of the trees, so that the model can accurately learn the distribution characteristics of normal data and improve the ability to identify outliers; S303, model fusion: Fuse the trained LSTM model and the isolation forest model. A weighted fusion approach can be used to assign different weights to the LSTM model and the isolation forest model based on their performance on the test set. S4: Abnormal access behavior monitoring and response; S401, real-time monitoring: After data preprocessing and dynamic feature mining, the access data collected in real time is input into the fused model. The model determines whether the current access behavior is abnormal based on the learned feature patterns; S402, Alert and Record: Once abnormal access behavior is detected, the system will immediately issue an alert and notify relevant security management personnel via email, SMS, or system pop-up window. At the same time, detailed information about the abnormal access, including access time, source IP address, target resource, abnormality type, etc., will be recorded for subsequent in-depth analysis and tracing. S403, Feedback Optimization: After investigating and analyzing abnormal access events, security managers provide feedback to the collaborative framework. Based on this feedback, they adjust the parameters of the feature mining algorithm, optimize the model structure, or update the training data to further improve the accuracy and adaptability of the monitoring system.

[0009] The beneficial effect of this invention lies in the establishment of a collaborative machine learning model that effectively addresses the issues of delayed updates to the rule base for abnormal access data and the difficulty in accurately capturing long-term dependencies. This model integrates the strengths of the UBER, LSTM, and Isolation Forest algorithms. It encompasses multiple steps, including data collection, dynamic feature mining, model training and fusion, and anomaly detection and response, to determine whether current access behavior is abnormal. This collaborative machine learning model enables efficient and accurate monitoring of abnormal data access behavior.

[0010] This invention can mine data features from multiple perspectives, more accurately identifying abnormal data access behavior and reducing false positives and false negatives. It can also promptly discover new abnormal behavior patterns and adapt to changing network environments and attack methods, enabling rapid detection and response to abnormal access behavior. DETAILED DESCRIPTION

[0011] The present invention is further described by way of examples below, which do not limit the present invention in any way. Without departing from the technical solution of the present invention, any modification or alteration of the present invention that can be easily implemented by a person skilled in the art will fall within the scope of the claims of the present invention. Example 1

[0012] S1, building a collaborative framework S101, Design the Data Collection Layer: Based on UBER technology, deploy data collection modules at key nodes in the network, such as routers, servers, and databases. These modules are responsible for collecting access data generated by these devices, including network traffic data, system logs, and database operation records. The collection modules transmit the data in real time to the data storage center using a unified data format and transmission protocol. S102, build the feature mining and analysis layer: This layer integrates LSTM and the Isolation Forest algorithm. LSTM is used to analyze time series data and mine dynamic features in the time dimension. The Isolation Forest algorithm is used to analyze the overall distribution of data and identify outlier features. Simultaneously, a data preprocessing unit is set up to clean, denoise, and normalize the collected data, providing high-quality data for subsequent feature mining and model training. S103, establish a model training and monitoring layer: deploy the LSTM model and the isolation forest model, and design a model fusion module; divide the data processed by the feature mining and analysis layer into a training set and a test set, and train and optimize the LSTM model and the isolation forest model respectively; during the training process, adjust model parameters, such as the number of hidden layer units in the LSTM and the number of trees in the isolation forest, to improve model performance; the trained model monitors the input access data features in real time, and uses the model fusion module to determine whether there is any abnormal access behavior; S2: Dynamic Feature Mining S201, Time Series Feature Mining (LSTM-based): Read access data arranged in time series, such as access time and access frequency, from the data storage center; preprocess this data and input it into the LSTM model. The LSTM model captures the periodic and trend changes in access behavior by learning the time series characteristics of the data. For example, the LSTM model can detect that a user's access frequency on Monday mornings is significantly higher than at other times, or that the access time interval of a certain IP address is abnormally shortened. S202, Outlier Feature Mining (Based on the Isolation Forest Algorithm): Raw access data, not processed as a time series, is fed into the Isolation Forest Algorithm. The Isolation Forest Algorithm constructs a tree structure based on the data's characteristics and calculates the path length from each data point to the root node. Data points with longer paths are considered outliers. The access behavior characteristics corresponding to these outliers, such as rare source IP addresses and unusually high access frequencies, are then mined as outlier features. S3: Model training and integration S301, LSTM model training: Use the training data processed by time series feature mining to train the LSTM model. During the training process, use an appropriate loss function, such as the mean squared error (MSE) loss function, and continuously adjust the model parameters through the backpropagation algorithm to enable the LSTM model to accurately learn the time series characteristics of normal access behavior and establish a normal behavior model. S302, Isolation Forest Model Training: Input the training data after outlier feature mining into the Isolation Forest Model for training; During the training process, adjust the parameters of the Isolation Forest Model, such as the number of trees and the depth of the trees, so that the model can accurately learn the distribution characteristics of normal data and improve the ability to identify outliers; S303, Model Fusion: Fuse the trained LSTM model and the Isolation Forest model. A weighted fusion approach can be used, assigning different weights to the LSTM and Isolation Forest models based on their performance on the test set. For example, if the LSTM model performs better in detecting time series anomalies, while the Isolation Forest model is more accurate in identifying outliers, then a higher weight can be assigned to the LSTM model for time series anomaly detection and a higher weight to the Isolation Forest model for outlier detection. The fused model combines the advantages of both models to improve the accuracy of anomaly detection. S4: Abnormal access behavior monitoring and response S401, real-time monitoring: After data preprocessing and dynamic feature mining, the access data collected in real time is input into the fused model. The model determines whether the current access behavior is abnormal based on the learned feature patterns. For example, if the model detects that the time series characteristics of a certain access behavior are significantly different from the normal pattern learned by the LSTM model, and the isolation forest model also determines that the behavior is an anomaly, the access behavior is determined to be abnormal. S402, Alert and Record: Once abnormal access behavior is detected, the system immediately issues an alert and notifies relevant security management personnel via email, SMS, or system pop-up window. At the same time, detailed information about the abnormal access, including access time, source IP address, target resource, and abnormality type, is recorded for subsequent in-depth analysis and tracing. S403, Feedback Optimization: After investigating and analyzing abnormal access events, security managers feed back the results to the collaborative framework. Based on the feedback information, they adjust the parameters of the feature mining algorithm, optimize the model structure, or update the training data to further improve the accuracy and adaptability of the monitoring system.

Claims

1. A method for monitoring abnormal access data behavior based on collaborative framework dynamic feature mining, comprising the following steps: S1, building a collaborative framework; S101, design the data collection layer: Based on UBER technology, deploy data collection modules on key nodes in the network, such as routers, servers, databases, and other devices; These modules are responsible for collecting access data generated by devices, including network traffic data, system log records, and database operation records. The acquisition module transmits the data to the data storage center in real time according to a unified data format and transmission protocol. S102, build the feature mining and analysis layer: This layer integrates LSTM and Isolation Forest algorithm; LSTM is used to analyze time series data and mine dynamic features of the time dimension; The isolation forest algorithm is used to analyze the overall distribution of data and identify abnormal point features; Set up a data preprocessing unit to clean, denoise, and normalize the collected data to provide high-quality data for subsequent feature mining and model training; S103, establish the model training and monitoring layer: deploy the LSTM model and the isolation forest model, and design a model fusion module; divide the data processed by the feature mining and analysis layer into a training set and a test set, and train and optimize the LSTM model and the isolation forest model respectively; During the training process, model parameters are adjusted, such as the number of hidden layer units in LSTM and the number of trees in Isolation Forest, to improve model performance. The trained model monitors the input access data features in real time and uses the model fusion module to determine whether there is abnormal access behavior. S2: dynamic feature mining; S201, Time Series Feature Mining (LSTM-based): Read access data arranged in time series, such as access time and access frequency, from the data storage center; pre-process this data and input it into the LSTM model; the LSTM model captures the periodicity and trend changes in access behavior by learning the time series features of the data; S202, outlier feature mining based on the Isolation Forest Algorithm: The original access data, which has not been processed into a time series, is input into the Isolation Forest Algorithm. The Isolation Forest Algorithm constructs a tree structure based on the data characteristics and calculates the path length from each data point to the root node. Data points with longer path lengths are considered outliers, and the access behavior characteristics corresponding to these outliers are the mined outlier features. S3: Model training and fusion; S301, LSTM model training: Use the training data processed through time series feature mining to train the LSTM model. During the training process, use an appropriate loss function and continuously adjust the model parameters through the backpropagation algorithm to enable the LSTM model to accurately learn the time series characteristics of normal access behavior and establish a normal behavior model. S302, Isolation Forest Model Training: Inputting the training data processed by outlier feature mining into the Isolation Forest Model for training; During the training process, the parameters of the isolation forest model, such as the number of trees and the depth of the trees, are adjusted so that the model can accurately learn the distribution characteristics of normal data and improve its ability to identify outliers. S303, model fusion: Fuse the trained LSTM model and the isolation forest model. A weighted fusion approach can be used to assign different weights to the LSTM model and the isolation forest model based on their performance on the test set. S4: Abnormal access behavior monitoring and response; S401, real-time monitoring: After data preprocessing and dynamic feature mining, the access data collected in real time is input into the fused model. The model determines whether the current access behavior is abnormal based on the learned feature patterns; S402, Alert and Record: Once abnormal access behavior is detected, the system will immediately issue an alert and notify relevant security management personnel via email, SMS, or system pop-up window. At the same time, detailed information about the abnormal access, including access time, source IP address, target resource, abnormality type, etc., will be recorded for subsequent in-depth analysis and tracing. S403, Feedback Optimization: After investigating and analyzing abnormal access events, security managers feed back the results to the collaborative framework. Based on the feedback information, the parameters of the feature mining algorithm are adjusted, the model structure is optimized, or the training data is updated to further improve the accuracy and adaptability of the monitoring system.