A computer network security analysis method and system based on big data

Abstract

The present application discloses a computer network security analysis method based on big data, which relates to the field of network security, including: collecting user behavior data and interaction data; preprocessing the collected user data set; using a federated learning architecture, taking the preprocessed user data set as a training sample, training a machine learning model locally on the user side for user status analysis; uploading the parameters of the trained machine learning model to a cloud server, performing global aggregation on the cloud service, and obtaining a global machine learning model; based on the global machine learning model and interaction data, using an unsupervised learning method based on anomaly detection to perform security analysis on the user's computer network behavior. In view of the low network security efficiency existing in the prior art, the present application performs real-time analysis of user status through a local BiLSTM neural network model on the user side, and at the same time, uses a federated learning architecture to perform global aggregation on the cloud, etc., thereby improving the efficiency of analysis.

Description

Technical Field

[0001] The present application relates to the field of network security, and in particular to a computer network security analysis method and system based on big data. Background Art

[0002] With the rapid development and widespread application of the Internet, network security issues are becoming increasingly serious. According to statistics, the number and frequency of global cyber attacks show a clear upward trend, posing a huge threat to the security of information assets of individuals, enterprises and countries. Although traditional network security protection methods, such as firewalls and intrusion detection systems, can resist known attack patterns to a certain extent, their effectiveness and adaptability are obviously insufficient in the face of evolving new attacks and complex and changing network environments. Therefore, there is an urgent need for an intelligent and adaptive network security analysis method that can identify potential security risks in real time and accurately, and provide effective protection and disposal measures. The rise of big data and artificial intelligence technologies has provided new ideas and means to solve the above problems. Big data platforms can realize the efficient collection, storage and processing of massive network data, providing rich data support for security analysis. Machine learning algorithms, especially deep learning models, can automatically learn complex patterns of network behavior and discover hidden security threats. Combining the two and applying them to the field of network security is expected to significantly improve the level of intelligence in security analysis and protection.

[0003] However, existing network security analysis methods still have some limitations. On the one hand, centralized machine learning models need to upload user data to the cloud for training, which poses a risk of privacy leakage and is difficult to adapt to the personalized needs of different users. On the other hand, a single security analysis model often has difficulty coping with complex and changing network environments and lacks adaptability and explainability. In addition, the real-time processing and analysis of massive network data also poses challenges to the efficiency and scalability of the system. Therefore, there is an urgent need for a new network security analysis framework that can fully utilize the value of distributed data while protecting user privacy, and improve the efficiency, accuracy, and adaptability of security analysis.

[0004] In the related art, for example, Chinese patent document CN116027903B provides a computer network security analysis system and method based on big data, which belongs to the field of network security analysis. The network security analysis system includes a data acquisition module, a database, a data analysis module and a mouse driver module; the data acquisition module is used to collect basic data information, user computer usage information and web page sensitive word information, the database is used to encrypt and store the collected information and the analysis results, the data analysis module is used to analyze and process the collected data information, and the mouse driver module is used to automatically control the mouse cursor to operate according to the analysis results. The present invention enters the basic sensitive word information of the database, analyzes the user's computer usage status, the sensitive word information of the page and the click position information, establishes an operation learning model, and automatically drives the mouse cursor to operate when the user no longer uses the computer. However, the scheme adopts a centralized data processing mode, which has privacy leakage and performance bottlenecks, so the network security efficiency of the scheme needs to be further improved. Summary of the invention

[0005] In response to the problem of low network security efficiency in the prior art, the present application provides a computer network security analysis method and system based on big data, which performs real-time analysis of user status through a local BiLSTM neural network model on the user side. At the same time, it utilizes a federated learning architecture to perform global aggregation in the cloud, thereby improving the efficiency of the analysis.

[0006] Technical solution, the purpose of this application is achieved through the following technical solution.

[0007] One aspect of the present specification provides a computer network security analysis method based on big data, including: collecting user behavior data and interaction data to form a user data set; wherein the behavior data includes the user's head posture, keyboard input and mouse movement trajectory; the interaction data includes the location where the user clicks on the web page; preprocessing the collected user data set, and storing the preprocessed user data set in a distributed database HDFS; adopting a federated learning architecture, using the preprocessed user data set as a training sample, and training a machine learning model locally on the user side for user status analysis; uploading the parameters of the trained machine learning model to a cloud server, and performing global aggregation in the cloud service to obtain a global machine learning model; wherein, using an integrated learning method, according to the judgment results of multiple local machine learning models, a global machine learning model after global aggregation is obtained; according to the global machine learning model and the interaction data, an unsupervised learning method based on anomaly detection is adopted to perform security analysis on the user's computer network behavior.

[0008] Federated learning is a distributed machine learning paradigm that allows multiple parties to collaboratively train models without sharing original data. In federated learning, each party (such as the user end) trains the machine learning model locally using its own data, and uploads the trained model parameters to a central server (such as a cloud server) for aggregation to obtain a global model. This architecture can protect user privacy while leveraging the richness of distributed data to improve model performance.

[0009] User status refers to the behavior patterns and characteristics of users when using computer networks. By analyzing behavioral data such as the user's head posture, keyboard input, mouse movement trajectory, and interactive data such as the location of the user's click on the web page, the user status can be modeled and analyzed. The purpose of user status analysis is to identify the user's normal behavior pattern and detect abnormal or suspicious behavior to provide a basis for network security.

[0010] Global aggregation is a key step in federated learning, which refers to summarizing and fusing the parameters of local machine learning models trained by multiple participants to obtain a global model. Common global aggregation methods include the FedAvg (federal average) algorithm, which is a weighted average of local model parameters. In this application, an integrated learning method is used for global aggregation, that is, based on the judgment results of multiple local models, the final global model is obtained by voting, weighted averaging, etc., to improve the robustness and generalization ability of the model.

[0011] Anomaly detection is a commonly used unsupervised learning task that aims to identify abnormal samples in a data set that are significantly different from most samples. Unsupervised learning methods based on anomaly detection do not require prior labeling of data, but automatically discover anomalies by learning the normal patterns of data. Common anomaly detection algorithms include statistical-based methods (such as Gaussian distribution), distance-based methods (such as k-nearest neighbors), density-based methods (such as LOF), etc. In this application, the normal pattern of user behavior is learned through a global machine learning model, and combined with interaction data, abnormal network behavior that deviates from the normal pattern is detected.

[0012] Security analysis refers to a comprehensive assessment of computer network systems and user behaviors to identify potential security threats and risks. This application uses a combination of federated learning and anomaly detection to perform security analysis on users' network behaviors. First, a machine learning model is trained locally on the user side to learn the user's normal behavior patterns; then, through global aggregation in the cloud, a global model that integrates the behavioral characteristics of multiple users is obtained; finally, using the global model and interaction data, an unsupervised anomaly detection method is used to identify users' abnormal or suspicious network behaviors, providing real-time protection and early warning for network security.

[0013] Further, preprocessing includes: using a statistical analysis method to obtain statistical features of the user data set; based on the extracted statistical features, calculating the information gain of each statistical feature for the user abnormal behavior detection task; selecting features with information gain greater than a first threshold to form a key feature subset; randomly selecting a sample subset from the user data set, and randomly selecting features from the key feature subset, recursively dividing the sample subset according to the selected features, and constructing an isolated tree; constructing multiple isolated trees to generate an isolated forest; for each data point in the user data set, calculating the average path length of the data point in all isolated trees, and marking the data points with an average path length less than a second threshold as outliers; wherein calculating the information gain includes: calculating the empirical entropy of the user data set; calculating the conditional entropy of each statistical feature for the user data set; and taking the difference between the empirical entropy and the conditional entropy as the information gain of the corresponding statistical feature.

[0014] Among them, information gain is a commonly used feature selection method to measure the influence of a certain feature on the target variable. It calculates the reduction of the empirical entropy of the feature on the data set, that is, the degree to which the feature eliminates the uncertainty of the target variable. The larger the information gain, the stronger the decision-making ability of the feature on the target variable, and the greater the contribution to the classification or prediction task. In this application, by calculating the information gain of each statistical feature for the user abnormal behavior detection task, the features with information gain greater than the threshold are selected to form a key feature subset for subsequent anomaly detection model construction.

[0015] An isolation tree is a tree structure used for anomaly detection. It isolates outliers by recursively partitioning the data space. The construction process of an isolation tree is as follows: randomly select a sample subset from the data set and randomly select a feature from the feature set; based on the selected feature, randomly select a split point within its value range to divide the sample subset into two subsets; recursively repeat the above process for each subset until the number of samples in the subset is less than a certain threshold or the maximum tree depth is reached. The core idea of ​​the isolation tree is that outliers are usually located in sparse areas of the data space, so they are easier to be isolated during the partitioning process, and their path length in the tree is shorter. The average path length is an indicator to measure the degree of abnormality of a data point in the isolation forest. For each data point, calculate its path length in all isolation trees and take the average to get the average path length. Because outliers are relatively isolated in the data space, they are usually quickly isolated in isolation trees, so their average path length is short. By setting a threshold, data points with an average path length less than the threshold are marked as outliers. The shorter the average path length, the more likely the data point is an outlier.

[0016] Empirical entropy is an indicator of the purity of a data set, reflecting the degree of confusion of samples of different categories in the data set. For a binary classification problem, the formula for calculating empirical entropy is: where p iIt represents the proportion of samples of the i-th category in the data set D. The larger the empirical entropy, the more evenly the samples of different categories in the data set are distributed, and the lower the purity; the smaller the empirical entropy, the more dominant one of the samples in the data set is, and the higher the purity is. In feature selection, the empirical entropy is often used as a benchmark to calculate the degree to which each feature improves the purity of the data set. Conditional entropy represents the uncertainty of a data set under the condition that a certain feature is known. For feature A and data set D, the calculation formula of conditional entropy H(D|A) is: Where n is the number of values ​​of feature A, D i It is the sample subset of feature A in D taking the i-th value. The conditional entropy reflects the classification or prediction ability of feature A for data set D. The smaller the conditional entropy, the stronger the decision-making ability of feature A for the target variable. When calculating information gain, conditional entropy is used to measure the degree to which the feature improves the purity of the data set. Information gain is the difference between empirical entropy and conditional entropy.

[0017] Specifically, the first threshold is used to screen features with large information gain to form a key feature subset. The effect of different thresholds on anomaly detection performance is evaluated through cross-validation, and the optimal threshold is selected. The specific steps are as follows: Divide the data set into a training set and a validation set. For different first threshold values, construct key feature subsets and isolation forest models respectively, and evaluate the anomaly detection performance (such as F1 score, AUC, etc.) on the validation set. Select the threshold with the best performance on the validation set as the first threshold. The second threshold is used to determine the degree of abnormality of the data point, and mark the data points with an average path length less than the threshold as outliers. The second threshold is used as a hyperparameter, and the optimal threshold is searched through an optimization algorithm (such as grid search, Bayesian optimization, etc.). In each iteration, the outliers are marked according to the current second threshold, and the performance indicators of anomaly detection (such as precision, recall, F1 score, etc.) are evaluated, and finally the threshold with the best performance is selected as the second threshold.

[0018] Furthermore, the preprocessing also includes: modeling the user data set based on the obtained key feature subset, fitting the user data distribution with a Gaussian mixture distribution, and estimating the parameters of the Gaussian mixture distribution by an expectation-maximization algorithm; calculating the posterior probability of data points marked as outliers in the Gaussian mixture distribution; setting a third threshold based on the sorted quantile of the posterior probability of the outliers; filtering outliers whose posterior probability is less than the third threshold to obtain a filtered user data set; and using the filtered user data set as the preprocessed user data set.

[0019] Among them, Gaussian mixture distribution is a commonly used probability density function used to fit complex data distribution. It is composed of multiple Gaussian distributions (normal distributions), each of which is called a component. Gaussian mixture distribution can fit data distributions with multiple peaks or different shapes. In anomaly detection, it can be used to model normal patterns of user behavior. The sorting quantile is a statistic that describes the location of data distribution, indicating the proportion of data points in the data set that are less than or equal to a certain value. In this application, the third threshold is set according to the sorting quantile of the posterior probability of the outlier, which means selecting a suitable quantile (such as 5% or 10%), filtering out the outliers whose posterior probability is less than the corresponding value of the quantile, and obtaining a filtered user data set. This method can adaptively adjust the filtering criteria for outliers, retain most of the normal data points, and remove extreme outliers. The expectation-maximization algorithm (EM algorithm) is an iterative optimization algorithm that is commonly used for parameter estimation of probability models containing hidden variables. In the Gaussian mixture model, the EM algorithm is used to estimate the parameters of the mixture distribution, including the weights, means, and covariance matrices of each component.

[0020] (Expectation Step): According to the current model parameters, calculate the posterior probability (Posterior Probability) of each data point belonging to each component. M Step (Maximization Step): According to the posterior probability calculated in the E step, update the model parameters so that the likelihood function reaches the maximum. The EM algorithm converges the model parameters to a local optimal solution by continuously iterating the E step and the M step, thereby realizing parameter estimation of the Gaussian mixture distribution. In this application, the EM algorithm is used to estimate the parameters of the Gaussian mixture model of the user data distribution, obtain the fitted probability density function, and calculate the posterior probability of the outlier under the distribution, providing a basis for outlier filtering.

[0021] Furthermore, a local machine learning model is obtained, including: initializing the parameters of the global machine learning model on the cloud server side, and distributing the initialized model parameters to each user side; on each user side, dividing the preprocessed user data set into a training set, a validation set, and a test set; using the training set, training the neural network BiLSTM as a local machine learning model; calculating the gradient of the loss function relative to each parameter in the BiLSTM model, and accumulating the sum of the squares of the gradients; adjusting the learning rate of each parameter according to the sum of the squares of the accumulated gradients; using the adjusted learning rate, updating the parameters of the BiLSTM model until the model converges or reaches a preset number of iterations; using the validation set, evaluating the performance indicators of the BiLSTM model, the performance indicators including accuracy and F1 score; uploading the parameters and performance indicators of the trained BiLSTM model to the cloud server for global aggregation.

[0022] Among them, BiLSTM (Bidirectional Long Short-Term Memory) is a bidirectional long short-term memory neural network, which is often used for modeling and prediction of sequence data. It consists of two LSTM (Long Short-Term Memory) networks, one propagating along the forward direction of the sequence and the other propagating along the reverse direction of the sequence. In this application, BiLSTM is used as a local machine learning model to learn the temporal characteristics of user behavior by training the preprocessed user data set for user status analysis and anomaly detection.

[0023] Parameters are variables that need to be learned and optimized in machine learning models, which determine the performance and generalization ability of the model. In neural networks, parameters usually refer to the weights and biases of the network. The weights represent the strength of the connection between neurons, and the biases represent the activation thresholds of neurons. These parameters are continuously adjusted through training data so that the prediction results of the model are as close as possible to the true labels, thereby achieving fitting and generalization of the data. In this application, the parameters of the BiLSTM model include the weight matrices and bias vectors of the input layer, hidden layer, and output layer. These parameters are optimized by back propagation algorithm and gradient descent method, and finally a local machine learning model with excellent performance is obtained.

[0024] Learning rate is an important hyperparameter in machine learning, which controls the adjustment range of model parameters at each iterative update. In this application, an adaptive learning rate adjustment strategy is adopted: the learning rate is dynamically adjusted according to the cumulative square sum of the gradients of each parameter. Specifically, for parameters with a larger cumulative square sum of the gradients, the learning rate will be reduced accordingly to avoid excessive update steps; for parameters with a smaller cumulative square sum of the gradients, the learning rate will be increased accordingly to accelerate convergence. This adaptive learning rate strategy can take into account both convergence speed and stability, and improve the efficiency and robustness of model training. Common adaptive learning rate algorithms include AdaGrad, RMSprop, Adam, etc.

[0025] Furthermore, the training set is used to train the neural network BiLSTM as a local machine learning model, including: converting the behavior data and interaction data in the training set into time series data as the input sequence of the BiLSTM network; wherein the BiLSTM network is composed of a forward LSTM and a backward LSTM; the forward LSTM adjusts the forward hidden state of the current time step according to the input of the current time step and the hidden state of the previous time step; the backward LSTM adjusts the backward hidden state of the current time step according to the input of the current time step and the hidden state of the next time step; the forward and backward hidden states are concatenated to obtain the BiLSTM output of the current time step; The hidden state of the last time step of iLSTM is used as the query vector of user status; user status includes the user's behavior pattern and degree of abnormality; multi-layer perceptron MLP is used as the attention function, and the attention weight of each time step is calculated according to the output of BiLSTM and the query vector of user status; the output sequence of BiLSTM is summed up with attention weights to obtain an aggregated feature vector; the aggregated feature vector is input into the fully connected layer to calculate the probability distribution of user status: according to the probability distribution of user status, the cross entropy loss function is calculated, and the AdaGrad optimization algorithm is sampled to adjust the parameters of the BiLSTM model to obtain the trained local machine learning model.

[0026] Among them, user status refers to the comprehensive behavioral characteristics and risk level of the user at a certain point in time or time period. It comprehensively considers factors such as the user's behavior pattern and degree of abnormality, and is the target variable for user status analysis and anomaly detection. In this application, the user status is modeled as a multi-classification problem, and each category corresponds to a user status (such as normal, abnormal, high risk, etc.). The BiLSTM model predicts and analyzes the user status by learning the mapping relationship between the user behavior sequence and the user status.

[0027] Multi layer Perceptron (MLP) is a feedforward neural network consisting of an input layer, one or more hidden layers, and an output layer. MLP can learn the nonlinear mapping relationship between input and output and is widely used in tasks such as classification and regression. In this application, MLP is used as an attention function to calculate the attention weight of each time step based on the output of BiLSTM and the query vector of the user state. This MLP-based attention mechanism can adaptively adjust the importance of different time steps and improve the accuracy of user state prediction.

[0028] The query vector of user status is a compact feature representation used to retrieve and compare different user statuses. In this application, the hidden state of the last time step of BiLSTM is used as the query vector of user status. This vector contains the contextual information and semantic features of the entire user behavior sequence and can characterize the comprehensive status of the user. In the attention mechanism, the query vector interacts with the output sequence of BiLSTM to calculate the relevance and importance of each time step, thereby achieving adaptive feature aggregation.

[0029] The probability distribution of user states describes the probability of occurrence of different user states. In this application, the output of the BiLSTM model is converted through the attention weighted sum and the fully connected layer to obtain the probability distribution of user states. This distribution is a discrete probability vector, and each element represents the estimated probability of the corresponding user state. Through the probability distribution, the user state can be soft-classified and risk assessed. During the model training process, the parameters of the BiLSTM model are optimized by minimizing the cross entropy loss between the predicted probability distribution and the true distribution.

[0030] AdaGrad (Adaptive Gradient) is an adaptive learning rate optimization algorithm, commonly used in neural network training. AdaGrad maintains a learning rate for each parameter and dynamically adjusts the learning rate based on the historical gradient information of the parameter. For parameters with larger gradients, AdaGrad will reduce their learning rate to avoid excessive update steps; for parameters with smaller gradients, AdaGrad will increase their learning rate to accelerate convergence. In this application, the AdaGrad algorithm is used to optimize the parameters of the BiLSTM model to improve training efficiency and convergence speed. Compared with the traditional SGD algorithm, AdaGrad can adaptively adjust the learning rate and is more suitable for sparse data and non-stationary problems.

[0031] Furthermore, the probability distribution of user status is calculated using the following formula:

[0032] Among them, p i is the normalized probability value of the i-th user state, xi is the i-th output value of the aggregated feature vector after transformation by the fully connected layer, μ is the mean of the fully connected layer output after Batch Norm normalization, σ is the standard deviation after BatchNorm normalization, ε is a very small positive number, and N is the total number of user status categories;

[0033] The definitions of various parameters and operations are as follows: i Represents the normalized probability value of the i-th user state, indicating the probability that the user is currently in the i-th state, satisfying x i It represents the i-th output value obtained after the aggregated feature vector is transformed by the fully connected layer, reflecting the original score of the user status in the i-th category.

[0034] μ represents the mean value obtained after Batch Norm normalization of the output of the fully connected layer, which is used to unify the outputs of different categories to the same scale. Its calculation formula is: σ represents the standard deviation obtained after Batch Norm normalization of the output of the fully connected layer, which is used to measure the dispersion of outputs of different categories. Its calculation formula is: ε represents a very small positive number, which is used to prevent the denominator from being zero and improve numerical stability. It is usually taken as e -8 To e -6 N represents the total number of user status categories, that is, the number of neurons in the final output layer.

[0035] Furthermore, a global machine learning model is obtained, including: on the cloud server side, the weights of the local machine learning models are calculated according to the accuracy and F1 score of each local machine learning model; according to the weights, the parameters of each local machine learning model are weighted averaged to obtain the parameters of the global machine learning model; the aggregated global machine learning model is evaluated using a test set; the parameters of the aggregated global machine learning model are distributed to each user terminal as initial parameters for the next round of local training; the trained global machine learning model is deployed to the cloud server side and the user side for security analysis of the user's computer network behavior.

[0036] Among them, accuracy is one of the commonly used indicators for evaluating the performance of classification models, which indicates the proportion of samples predicted correctly by the model to the total number of samples. The F1 score is the harmonic mean of precision and recall, which comprehensively considers the accuracy and completeness of the model. In the global machine learning model aggregation process of this application, accuracy and F1 scores are used as indicators to evaluate the performance of local models, and the weights of local models are calculated accordingly. This weighted average strategy can make full use of the differences and complementarities of different user data to obtain a robust global model with better generalization performance. At the same time, through the combination of accuracy and F1 scores, the performance of local models and global models in user state analysis and anomaly detection tasks can be evaluated more comprehensively and objectively.

[0037] Furthermore, the weight is calculated by the following formula:

[0038]

[0039] Among them, acc i 、f1 i 、auc i are the accuracy, F1 score and AUC value of the i-th local model respectively; a, b, c are weight coefficients; ε is the smoothing term; N is the total number of local models; Represents dot product; specifically, the definitions of various parameters and operations are as follows: i represents the normalized weight value of the i-th local model, indicating the importance of the model in the global aggregation, satisfying acc i It represents the accuracy of the i-th local model on the validation set, and measures the overall judgment ability of the model on the user status. Its calculation formula is: f1 i It represents the F1 score of the i-th local model on the validation set, which measures the balance and detection ability of the model for different categories of user states. Its calculation formula is: auc i Represents the AUC value (Area Under Curve) of the i-th local model on the validation set, which measures the model's ability to judge the user status under different thresholds. It is calculated by integrating the area under the ROC curve (True Positive Rate vs. False Positive Rate). a, b, c: three hyperparameters used to balance the impact of accuracy, F1 score and AUC value on model weight. Usually adjusted according to specific tasks and data characteristics to satisfy a+b+c=1. ε represents a very small positive number, which is used to prevent the denominator from being zero and improve numerical stability. Usually the value is e -8 To e -6 N: The total number of local models participating in the global aggregation.

[0040] Furthermore, an unsupervised learning method based on anomaly detection is used to perform security analysis on the user's computer network behavior, including: extracting the location information of the user clicking on the web page in the interactive data, encoding the location information using a pre-trained CNN neural network, and obtaining a location feature vector for anomaly detection; concatenating the extracted location feature vector and the obtained statistical features as a risk feature vector; classifying the obtained risk feature vector using a pre-trained SVM classifier to obtain a predicted risk level; adjusting the weight coefficients a, b, and c of the global machine learning model according to the predicted risk level; sending the adjusted model parameters to the local machine learning model of each user terminal; at the user terminal, updating the local machine learning model using the sent parameters, and using the updated local machine learning model to perform security analysis on the user's computer network behavior.

[0041] Specifically, first, the location information of the user's click on the web page is extracted from the interaction data, such as the coordinates and area of ​​the click. Then, the pre-trained CNN (Convolutional Neural Network) neural network is used to encode the location information. CNN can effectively extract local features and spatial relationships in data such as images and texts, and has a wide range of applications in feature representation learning. Through CNN encoding, the original location information can be converted into a low-dimensional, compact location feature vector, which contains the spatial pattern and semantic information of the user's click behavior. Statistical feature acquisition: In addition to location features, some statistical features need to be obtained, such as the user's click frequency, visit duration, and distribution of visited web page types. These statistical features reflect the overall pattern and habits of user behavior and also play an important role in anomaly detection. Statistical features can be obtained by aggregating and statistically analyzing the interaction data, such as calculating the mean, variance, quantile, etc. After obtaining the location feature vector and statistical features, they need to be spliced ​​to form a complete risk feature vector. The splicing method can be a simple vector connection, that is, connecting two feature vectors end to end to form a longer vector. Some more complex splicing methods can also be used, such as weighted splicing and feature interaction. The spliced ​​risk feature vector contains information on multiple aspects of user behavior, which can more comprehensively and meticulously characterize the risk status of the user. The spliced ​​risk feature vector can be used for risk prediction and anomaly detection. Specifically, the risk feature vector can be classified using a pre-trained SVM (Support Vector Machine) classifier to obtain the predicted risk level. SVM is a commonly used binary classifier that classifies samples by finding a maximum margin hyperplane in the feature space. The pre-trained SVM classifier can be trained on large-scale historical data to learn the mapping relationship between risk features and risk levels, thereby realizing risk prediction for new samples. According to the predicted risk level, the weight coefficients a, b, and c of the global machine learning model can be adjusted, and the adjusted model parameters can be sent to the local machine learning models of each user end. This feedback mechanism enables the global model to be adaptively updated according to the real-time risk prediction results, and continuously improves its anomaly detection performance. On the user end, the local machine learning model is updated with the sent parameters to keep it consistent with the global model, and is used to perform real-time security analysis and anomaly detection on the user's computer network behavior.

[0042] Another aspect of the present specification also provides a computer network security analysis system based on big data, which is used to execute a computer network security analysis method based on big data of the present application.

[0043] Beneficial effects: Compared with the prior art, the advantages of this application are:

[0044] By using the BiLSTM neural network to perform real-time modeling and anomaly detection on the user's local behavioral data and interaction data, it is possible to promptly detect abnormal changes in the user's network behavior, achieve early warning and rapid response, greatly improve the efficiency of security threat discovery and handling, and enhance the user's network security protection capabilities.

[0045] By adopting a federated learning architecture, cloud aggregation and knowledge fusion of local model training results are achieved while protecting user data privacy. This not only leverages the powerful computing and storage capabilities of the cloud, but also takes advantage of the timeliness and diversity of local data, enabling the global model to continuously learn and optimize, and continuously improve the generalization performance and robustness of detection.

[0046] In the cloud, we use aggregated global models and interaction data, and adopt methods such as unsupervised anomaly detection to conduct global evaluation and analysis of user network behavior, explore group behavior patterns and security trends, and combine dynamic adjustment and updating of model parameters to achieve collaborative analysis, interactive optimization and dynamic evolution between the cloud and local ends, building an adaptive and evolvable network security analysis system.

[0047] By collecting and preprocessing user behavior data and interaction data, combined with statistical analysis and anomaly detection methods, efficient acquisition and intelligent screening of user data are achieved, providing high-quality data support for subsequent network behavior security analysis.

[0048] Deploying lightweight behavioral data collection and anomaly analysis modules on the user's local end not only reduces the data transmission and computing pressure on the cloud, but also enables real-time analysis and early warning of personalized behaviors of different users. At the same time, global aggregation optimization is performed on the cloud, achieving continuous improvement of model performance and shared application of knowledge, while taking into account both user experience and system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is an exemplary flow chart of a computer network security analysis method based on big data according to some embodiments of this specification;

[0050] Figure 2 is an exemplary flow chart of calculating anomaly values ​​according to some embodiments of the present specification;

[0051] Figure 3 is an exemplary flow chart of obtaining a filtered user data set according to some embodiments of this specification;

[0052] Figure 4 This is an exemplary flowchart for obtaining a global machine learning model according to some embodiments of this specification. DETAILED DESCRIPTION

[0053] The method and system provided in the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0054] Figure 1 This is an exemplary flow chart of a computer network security analysis method based on big data shown in some embodiments of the present specification, including: collecting user behavior data and interaction data to form a user data set; wherein the behavior data includes the user's head posture, keyboard input and mouse movement trajectory; the interaction data includes the location where the user clicks on the web page; preprocessing the collected user data set, and storing the preprocessed user data set in the distributed database HDFS; adopting a federated learning architecture, using the preprocessed user data set as a training sample, and training a machine learning model locally on the user side for user status analysis; uploading the parameters of the trained machine learning model to the cloud server, and performing global aggregation in the cloud service to obtain a global machine learning model; wherein, an integrated learning method is adopted to obtain a global machine learning model after global aggregation based on the judgment results of multiple local machine learning models; based on the global machine learning model and the interaction data, an unsupervised learning method based on anomaly detection is adopted to perform security analysis on the user's computer network behavior.

[0055] First, the user's behavior data (including the user's head posture, keyboard input and mouse movement trajectory) and interaction data (including the location where the user clicks on the web page) are collected on the user side to form the original user data set. Specifically, a head posture acquisition module is installed on the user's computer device. The module captures the head image sequence of the user in real time when using the computer by calling the front camera of the device. The head posture acquisition module uses computer vision technology, such as face detection and key point positioning algorithm, to process each frame of the head image, extract the three-dimensional posture information of the user's head (including pitch angle, yaw angle and roll angle) and the position coordinates of the head in the image, and form a head posture data stream. The head posture data stream is sampled at a certain time interval (such as 0.1 seconds), recorded synchronously with the timestamp, and stored in the local cache. The keyboard input acquisition module is installed on the user's computer device. The module monitors and records all keyboard input events of the user by registering a global keyboard hook. Each keyboard event includes the key encoding, press / release status and the timestamp of the event. The keyboard input acquisition module serializes continuous keyboard events into a keyboard input data stream and stores it in the local cache. In order to protect user privacy, keyboard input data is processed by hash encryption, and only key codes and key behaviors are recorded, but not specific input content. A mouse movement acquisition module is installed on the user's computer device. This module monitors and records all mouse movement and click events of the user by registering a global mouse hook. Each mouse event includes the screen coordinates, movement distance, movement direction and timestamp of the mouse pointer. The mouse movement acquisition module serializes continuous mouse events into a mouse movement trajectory data stream and stores it in the local cache. At the same time, the mouse movement acquisition module also records the mouse's residence time in different applications and web pages to reflect the user's focus and behavior habits. A web page interaction acquisition module is installed on the user's computer device. This module runs in the form of a browser plug-in to monitor the user's web page interaction behavior in the browser. When a user clicks on a link, button, form or other element on a web page, the web page interaction acquisition module records the target element of the click event, the page coordinates of the click location and the timestamp of the event, forming an interactive data stream of the user's click on the web page location. In addition, the web page interaction acquisition module also records other interactive data such as the user's stay time on each web page and scrolling behavior to fully characterize the user's web page access pattern. The data collection module on the user side locally fuses multi-source heterogeneous data such as head posture, keyboard input, mouse movement trajectory, and web page interaction to form a structured user behavior data record. Each record contains metadata such as the corresponding data type, timestamp, and device ID to support subsequent data preprocessing and feature extraction. The locally fused user behavior data record will be further compressed and encrypted, and regularly uploaded to the cloud server to form the original user data set for subsequent network security analysis and modeling tasks.

[0056] Figure 2 It is an exemplary flow chart for calculating abnormal values ​​according to some embodiments of this specification. The user end uploads the collected original user data set to the cloud server through a secure network channel. The cloud server performs integrity verification and decryption processing on the received data, and stores it in a distributed file system (such as HDFS) to support subsequent large-scale data processing and analysis tasks. At the same time, the cloud server will perform necessary backup and disaster recovery on the user data to ensure the reliability and recoverability of the data. The cloud server performs preliminary cleaning and conversion on the stored original user data, performs statistical analysis on the cleaned user data, and extracts a series of statistical features reflecting the user behavior pattern. Statistical features include the central tendency (such as mean, median), discrete degree (such as variance, standard deviation), distribution form (such as skewness, kurtosis), etc. of various types of data. For different types of data, corresponding statistical methods are used for feature extraction, such as for head posture data, the average time and gaze angle distribution of the user's gaze on the screen are calculated; for keyboard input data, the average input speed and key usage frequency distribution are calculated; for mouse movement data, the average movement speed and movement direction distribution are calculated. The extracted statistical features will be used as input for subsequent anomaly detection models. In order to screen out the key features that contribute most to the task of detecting abnormal user behavior, the cloud server adopts a feature selection method based on information gain. First, the empirical entropy of the user data set is calculated to represent the overall uncertainty of the data set; then, for each statistical feature, its conditional entropy on the data set is calculated to represent the uncertainty of the data set under the condition of the given feature value. The information gain of a feature is defined as the difference between the empirical entropy and the conditional entropy, which indicates the contribution of the feature to reducing the uncertainty of the data set and improving the performance of the anomaly detection task. The cloud server calculates the information gain of all statistical features, selects features whose gain is greater than the preset threshold (first threshold), and constitutes a subset of key features for subsequent anomaly detection model training.

[0057] Figure 3This is an exemplary flow chart for obtaining a filtered user data set according to some embodiments of this specification. The cloud server uses the isolation forest algorithm to detect and mark outliers on the user data. The isolation forest is based on the assumption that abnormal data points are more easily isolated in the feature space, that is, they are farther away from other data points and are more likely to be individually assigned to leaf nodes in the randomly divided data subsets. The cloud server randomly selects a certain proportion of sample subsets from the user data set, and randomly selects features from the key feature subset, recursively divides the sample subset according to the selected features, and generates an isolation tree. By repeating this process, multiple isolation trees are constructed to form an isolation forest. For each data point in the user data set, the average path length (the average depth from the root node to the leaf node where the data point is located) in all isolated trees is calculated. The smaller the average path length, the more likely the data point is an outlier. The cloud server sets a threshold (second threshold) for the average path length and marks data points below the threshold as outliers. The cloud server adds the outlier tag information to the cleaned user data set to form a preprocessed user data set. The preprocessed data set contains key feature subsets and outlier markers, providing high-quality training samples for subsequent user behavior modeling and anomaly detection analysis. The cloud server stores the preprocessed user data set in HDFS and sets corresponding data lifecycle management strategies based on the timeliness and frequency of use of the data to optimize storage costs and access efficiency.

[0058] The cloud uses a Gaussian mixture model to model the user data set and filter outliers, including: representing the preprocessed user data set as a feature matrix X, where each row corresponds to a user data point and each column corresponds to a key feature. The dimension of the feature matrix X is N×D, where N is the number of user data points and D is the number of key features. The cloud server loads the feature matrix X into memory for the subsequent GMM modeling process. According to the characteristics and complexity of the user data, the cloud server sets the number of components K of the GMM, that is, the number of Gaussian distributions. The value of K needs to balance the model's fitting ability and computational efficiency, and can be optimized through cross-validation or information criteria (such as AIC, BIC) and other methods. Initialize the parameters of the GMM, including the mean vector μ of each Gaussian distribution k , covariance matrix Σ k and the mixing coefficient π k The mean vector μ k The dimension is D, the covariance matrix Σ k The dimension is D×D, and the mixing coefficient is π k is a scalar quantity, satisfying Usually, the result of K-means clustering is used to initialize the mean vector, the empirical covariance matrix of the data is used to initialize the covariance matrix, and the uniform distribution is used to initialize the mixing coefficient. The expectation maximization (EM) algorithm is used to estimate the parameters of GMM. The EM algorithm alternately executes the E step and the M step until the model parameters converge or the preset number of iterations is reached.

[0059] Step E: Calculate the value of each data point x according to the current model parameters. n The posterior probability (responsibility) γ(z nk ), Among them, an additional weight parameter w is introduced k , represents the importance or prior probability of the kth Gaussian component. The weight parameter w k It can be set based on domain knowledge or historical data to reflect the importance of different Gaussian components. k =1, it degenerates into the original posterior probability calculation formula. nk ): data point x n The posterior probability (responsibility) of the kth Gaussian component is in the range [0, 1]. Specifically, z nk represents the latent variable, representing the data point x n Whether it comes from the kth Gaussian component, the value is 0 or 1. x_n: the nth data point, a D-dimensional vector. The current model parameters, including π k , μ k ,Σ k and w k π k represents the mixing coefficient of the k-th Gaussian component, which represents the weight of the k-th component in the entire mixing model, and its value range is [0, 1], and N(x n |μ k ,Σ k ) represents the probability density function of the kth Gaussian component, representing the data point x n The probability under the k-th Gaussian distribution. μ k Represents the mean vector of the kth Gaussian component, which is a D-dimensional vector. Σ k represents the covariance matrix of the kth Gaussian component, which is a D×D positive definite matrix. k represents the weight parameter of the kth Gaussian component, reflecting the importance or prior probability of the kth component, and its value range is [0, 1], and

[0060] Step M: Update the model parameters according to the posterior probability calculated in step E so that the likelihood function reaches the local maximum. The parameter update formula is: Mixing coefficient: Mean vector: Covariance matrix: Weight parameters: In the parameter update formula, the weight parameter w is introduced k , which is used to adjust the contribution of different Gaussian components to parameter estimation. k = 1, it degenerates into the original parameter update formula. By introducing the weight parameter w k The improved implementation of the present application can consider the importance or prior probability of different Gaussian components in the GMM modeling process, making the model more in line with the needs of actual scenarios. k It can be flexibly set based on domain knowledge or historical data to reflect the importance of different Gaussian components in user behavior patterns. For example, a higher weight can be set for a frequently occurring normal behavior pattern and a lower weight can be set for a rare abnormal behavior pattern to improve the model's expressiveness and the accuracy of anomaly detection. In step E, the improved posterior probability calculation formula introduces the weight parameter w k , the distribution of the responsibility of different Gaussian components to data points is adjusted, so that important Gaussian components obtain higher posterior probabilities, thus playing a greater role in the model learning process. In the M step, the improved parameter update formula introduces the weight parameter w k , the contribution of different Gaussian components to parameter estimation is adjusted, so that important Gaussian components occupy a larger weight in parameter update, so as to better adapt to the distribution characteristics of the data. Repeat the E step and the M step until the change in the model parameters is less than the preset threshold or the preset maximum number of iterations is reached. The converged model parameters are the maximum likelihood estimates of the GMM. Use the preprocessed user data set to evaluate the fitting performance of the learned GMM. Common evaluation indicators include log-likelihood function value, AIC, BIC, etc. The larger the log-likelihood function value, the smaller the AIC and BIC values, which means that the GMM has a better fitting performance for the user data. At the same time, visualization methods can be used, such as drawing contour lines or three-dimensional surfaces of different Gaussian components, to intuitively show the fitting effect of GMM on the distribution of user data. The learned GMM is used as a probabilistic model of user behavior patterns for subsequent network security analysis tasks such as anomaly detection, behavior clustering, and user profiling. GMM can be used to calculate the posterior probability of new user data points to determine whether they belong to a known behavior pattern; GMM can be used to soft-cluster user data points to identify different user groups; GMM can be used to generate typical samples of user behavior to characterize the behavioral characteristics of different user groups.

[0061] The cloud server calculates the outlier posterior probability based on the estimated GMM parameters and filters outliers based on the posterior probability, including: for each data point x in the user data set nThe cloud server uses the learned GMM parameters to calculate the posterior probability γ(z nk ), that is, γ(z nk )=P(z nk =1|x n ,θ). The posterior probability γ(z nk ) means that at a given data point x n , the probability that the data point comes from the kth Gaussian component. nk ) is calculated as: For data points marked as outliers in the preprocessing stage, the cloud server focuses on their posterior probability distribution in GMM. The posterior probability distribution of outliers can reflect the degree of difference between outliers and learned user behavior patterns. If the posterior probability of an outlier on all Gaussian components is very low (such as less than a certain threshold ε), it indicates that the outlier is very different from all learned user behavior patterns and is likely to be a true abnormal behavior. If the posterior probability of an outlier on some Gaussian components is high (such as greater than a certain threshold δ), it indicates that the outlier is close to some user behavior patterns and may be a variant of normal behavior, which requires further identification.

[0062] The cloud server sets the third threshold τ according to the posterior probability distribution of outliers, and a ={x1,x2,......,x M}, the cloud server uses the learned GMM parameters to calculate each outlier x m The posterior probability γ(z mk ), where m = 1, 2, ..., M, k = 1, 2, ..., K. The posterior probability γ(z mk ) indicates that given an outlier value x m In the case of , the probability that the outlier comes from the kth Gaussian component. mk ) is calculated as: For each outlier x m , the cloud server calculates the maximum posterior probability γ on the K Gaussian components max (x m ), that is, γ max (x m )=max k γ(z mk ), where k = 1, 2, ..., K. Maximum a posteriori probability γ max (x m) indicates the Gaussian component where the outlier x_m is most likely to come from, reflecting the maximum similarity between the outlier and the learned user behavior pattern. The cloud server has the maximum posterior probability of all outliers {γ max (x1),γ max (x2),......,γ max (x M )} are sorted to obtain an ascending sequence of maximum posterior probability. The cloud server selects the maximum posterior probability value corresponding to a specific quantile (such as 5% or 10%) as the filtering threshold τ. Specifically, assuming that the 5% quantile is selected, the filtering threshold τ is equal to the maximum posterior probability value at the 5% position in the ascending sequence. The selection of the filtering threshold τ needs to balance the sensitivity and specificity of anomaly detection. A lower filtering threshold will increase the sensitivity of anomaly detection, but may lead to a higher false alarm rate; a higher filtering threshold will reduce the sensitivity of anomaly detection, but can reduce the false alarm rate. The selection of the filtering threshold can be adjusted according to actual needs, such as the severity of the abnormal behavior, the cost of false alarms and other factors.

[0063] For each outlier x_m, the cloud server compares its maximum posterior probability γ max (x m ) and the size of the filtering threshold τ. If γ max (x m ) is less than the filtering threshold τ, the outlier x_m is considered as a real abnormal behavior and needs to be paid special attention and analyzed. This means that the outlier x m If the behavior patterns of all learned users are significantly different, it is likely to be a real anomaly. max (x m ) is greater than or equal to the filtering threshold τ, then the outlier x m is considered a variant of normal behavior and is retained. This means that the outlier x m It is close to some learned user behavior patterns and may be a variation of normal behavior. The cloud server will filter the true outlier set X a 'As potential abnormal behaviors, they are analyzed and processed. These abnormal values ​​are quite different from the learned user behavior patterns and may pose a threat to network security. The cloud server will filter the normal behavior variants set X n 'Merge with the normal user data set as an expanded normal user behavior data set. Although these outliers are initially marked as abnormal, they are similar to some normal behavior patterns and can enrich the diversity of normal behavior. The filtered outlier set X a ' and the normal behavior variant set X n'It can be used for subsequent network security analysis, such as abnormal behavior clustering, threat assessment, security policy optimization, etc. The cloud server takes the user data set that has undergone GMM modeling and posterior probability filtering as the final preprocessing result. Compared with preliminary statistical feature extraction and isolation forest anomaly detection, GMM modeling can better characterize the diversity and correlation of user behaviors and provide more accurate and fine-grained feature representation. The cloud server stores the preprocessed user data set in HDFS (Hadoop Distributed File System) to support distributed storage and processing of large-scale data. At the same time, the cloud server updates the metadata information of the preprocessed user data set, such as the version number of the data set, processing timestamp, data source, feature list, etc., in order to track the source and evolution of the data. The management of metadata information can be achieved using HDFS extended attributes or a dedicated metadata management system (such as Apache Atlas).

[0064] Figure 4 This is an exemplary flow chart for obtaining a global machine learning model according to some embodiments of this specification. Then, the cloud server and the user end collaborate to complete federated learning. The cloud server randomly divides the preprocessed user data set into a training set, a validation set, and a test set. The cloud server distributes the divided data subsets (training set, validation set, and test set) and the initialized global model parameters to each user end participating in federated learning. Each user end uses the received training set data to train the local neural network model BiLSTM (Bidirectional LongShort-Term Memory). The user behavior data and interaction data in the training set are converted into time series data as the input sequence of the BiLSTM network. The BiLSTM network consists of a forward LSTM and a backward LSTM: the forward LSTM adjusts the forward hidden state of the current time step according to the input of the current time step and the hidden state of the previous time step. The backward LSTM adjusts the backward hidden state of the current time step according to the input of the current time step and the hidden state of the next time step. The forward and backward hidden states are spliced ​​to obtain the BiLSTM output of the current time step. The hidden state of the last time step of BiLSTM is used as the query vector of the user state, which includes the user's behavior pattern and abnormality. Multi-Layer Perceptron (MLP) is used as the attention function. The attention weight of each time step is calculated based on the output of BiLSTM and the query vector of the user state. The output sequence of BiLSTM is summed with attention weights to obtain an aggregated feature vector. The aggregated feature vector is input into the fully connected layer to calculate the probability distribution of the user state. The formula is as follows: Among them, p iis the normalized probability value of the i-th user state, x i is the i-th output value obtained after the aggregate feature vector is transformed by the fully connected layer, μ is the mean of the fully connected layer output after Batch Norm normalization, σ is the standard deviation after Batch Norm normalization, ε is a very small positive number, and N is the total number of user status categories. According to the probability distribution of user status, the cross entropy loss function is calculated, and the parameters of the BiLSTM model are adjusted using the AdaGrad optimization algorithm to obtain the trained local machine learning model.

[0065] The AdaGrad optimization algorithm is used to optimize the parameters of the BiLSTM model. Specifically, the initial learning rate η is set and applied to all parameters of the BiLSTM model. The gradient accumulation variable G of each parameter is initialized i is zero, where i represents the index of the parameter. In each training iteration, the loss function L is calculated with respect to each parameter θ according to the current BiLSTM model parameters and training data. i The gradient g i For each parameter θ in the BiLSTM model i , calculate its gradient g i : For each parameter θ i , its gradient g i The square of is accumulated to the corresponding gradient accumulation variable G i middle: Gradient accumulation variable G i The parameter θ is recorded i The cumulative sum of the squared gradients in all previous iterations. For each parameter θ i , according to its gradient accumulation variable G i , adjust its learning rate η i : Where ε is a small constant (such as 1e-8) to prevent the denominator from being zero. The adjusted learning rate η i and the gradient accumulation variable G i Inversely proportional. Gradient accumulation variable G i The larger the value, the greater the gradient change of the parameter in the previous iteration, and the smaller the learning rate; conversely, the gradient accumulation variable G i The smaller it is, the larger the learning rate is. Using the adjusted learning rate η i , update each parameter θ i : The updated parameter θ i Move in the direction of decreasing loss function L, with a step size of the adjusted learning rate η i Multiply by the gradient g of the current iteration i. The parameters of the BiLSTM model are optimized for multiple rounds of iterations until one of the following conditions is met: the preset maximum number of iterations is reached. The performance indicators of the model on the validation set (such as accuracy, F1 score) reach a satisfactory level or converge. At the end of each iteration, the performance indicators of the current model on the validation set are evaluated to monitor the optimization process of the model. By applying the AdaGrad optimization algorithm, the present application can adaptively adjust the learning rate of each parameter in the BiLSTM model, accelerate the convergence speed of the model and improve the performance of the model. The AdaGrad algorithm independently adjusts the learning rate of each parameter based on the gradient history information of each parameter, so that the learning rate of the parameter with a larger gradient change is smaller, and the learning rate of the parameter with a smaller gradient change is larger. This adaptive learning rate adjustment strategy can better adapt to the characteristics of different parameters and avoid the problems that may be caused by using a unified learning rate, such as slow convergence and local optimality.

[0066] Global model aggregation: The cloud server receives the local BiLSTM model parameters and performance indicators uploaded by each user. The weight of the local machine learning model is calculated based on the accuracy, F1 score, and AUC value of each local machine learning model. The weight calculation formula is as follows:

[0067]

[0068] auc i are the accuracy, F1 score and AUC value of the i-th local model respectively; a, b, c are weight coefficients; ε is the smoothing term; N is the total number of local models; Represents dot product. According to the calculated local model weights, the BiLSTM model parameters uploaded by each user end are weighted averaged to obtain the aggregated global BiLSTM model parameters. By weighted averaging, the local model with better performance is given a higher weight, so that it occupies a more important position in the global aggregation, thereby obtaining better global model parameters. The aggregated global BiLSTM model parameters are distributed to each user end for the next round of local model training and update. The global model parameters are transmitted to each user end through a secure and reliable communication protocol to ensure the integrity and confidentiality of the parameters. After receiving the global model parameters, the user end uses them as the initial parameters for the next round of local model training and continues to train and optimize local data. Multiple rounds of global model aggregation and parameter distribution are performed to realize the iterative optimization process of federated learning. In each round of iteration, the user end uses the updated global model parameters as the initial value, combines local data for training and optimization, and uploads the updated local model parameters and performance indicators to the cloud. After each round of iteration, the cloud server recalculates the weights and performs global aggregation based on the latest local model parameters and performance indicators to obtain updated global model parameters and distributes them to the user end. Through multiple rounds of iterations, the global model is continuously updated and optimized to better adapt to the data distribution and characteristics of different user terminals, and to improve the generalization ability and performance of the model. After the federated learning process is completed, the cloud server uses the final global BiLSTM model obtained by aggregation to evaluate the test set data and calculate the performance indicators of the model, such as accuracy, F1 score, AUC value, etc. The trained global BiLSTM model is deployed to the production environment for real-time prediction and anomaly detection of new user behavior data. According to the prediction results of the global BiLSTM model, the user's behavior pattern and degree of anomaly are analyzed and evaluated to provide support for subsequent network security decisions.

[0069] While training the global model, the cloud server also uses the preprocessed interaction data to extract location features and conduct risk assessment, and dynamically adjusts the global model parameters according to the assessment results to achieve adaptive optimization of the model. The cloud server uses the preprocessed interaction data to extract the location information of the user clicking on the web page. The extracted location information is encoded using a pretrained CNN (convolutional neural network) and converted into a fixed-length location feature vector. Through multi-layer convolution and pooling operations, CNN automatically learns and extracts high-level semantic features in the location information to obtain a compact and effective representation. The extracted location feature vector is concatenated with the previously obtained statistical features to form a comprehensive risk feature vector. The statistical features can include information such as the user's behavior frequency, time distribution, and access pattern, reflecting the overall characteristics of the user's behavior. By combining location features and statistical features, the user's behavior pattern and risk level can be more comprehensively characterized. The generated risk feature vector is classified using a pretrained SVM (support vector machine) classifier to obtain a predicted risk level. The SVM classifier maps the risk feature vector to different risk levels, such as low risk, medium risk, and high risk, by finding the optimal decision boundary. The pre-trained SVM classifier is trained using historical data and expert knowledge, and can effectively identify characteristic patterns of different risk levels. According to the predicted risk level, the weight coefficients a, b, and c of the global machine learning model are dynamically adjusted. The adjustment strategy of the weight coefficients can be based on preset rules or adaptive algorithms. For example, when the predicted risk level is high, the weight coefficient a is increased to increase the importance of accuracy in the global model; when the predicted risk level is low, the weight coefficients b and c are increased to increase the importance of F1 score and AUC value in the global model. By dynamically adjusting the weight coefficients, the global model can adaptively adjust the focus on different performance indicators according to the current risk situation, improving the pertinence and real-time performance of the model. The adjusted global model parameters are sent to each user end to update the local machine learning model. The updated model parameters are transmitted to each user end through a secure and reliable communication protocol to ensure the integrity and confidentiality of the parameters. After receiving the new model parameters sent by the cloud, the user end fine-tunes and updates the local BiLSTM model. By using the updated parameters, the user end can obtain an upgraded personalized behavior analysis model that integrates global knowledge and local characteristics. The newly generated network behavior data of users is input into the updated local model in real time for network security analysis and abnormal behavior identification. The updated local model can timely capture changes in user behavior patterns and accurately identify potential abnormal behaviors based on the knowledge of the global model. Once suspicious abnormal behavior is found, the local model will trigger an early warning and report relevant information to the cloud to complete the real-time response and subsequent disposal of network security incidents.

Claims

1. A computer network security analysis method based on big data, comprising: Collect user behavior data and interaction data to form a user data set; behavior data includes the user's head posture, keyboard input and mouse movement trajectory; interaction data includes the location where the user clicks on the web page; Preprocess the collected user data set and store the preprocessed user data set in the distributed database HDFS; Adopting the federated learning architecture, using the pre-processed user data set as the training sample, the machine learning model is trained locally on the user side for user status analysis; The parameters of the trained machine learning model are uploaded to the cloud server, and globally aggregated in the cloud service to obtain a global machine learning model; wherein an ensemble learning method is used to obtain a globally aggregated global machine learning model based on the judgment results of multiple local machine learning models; Based on the global machine learning model and interaction data, an unsupervised learning method based on anomaly detection is used to conduct security analysis on the user's computer network behavior; Pre-processing, including: Using statistical analysis methods to obtain statistical characteristics of user data sets; According to the extracted statistical features, the information gain of each statistical feature for the user abnormal behavior detection task is calculated; Select features with information gain greater than a first threshold to form a key feature subset; Randomly select a sample subset from the user data set, and randomly select features from the key feature subset, recursively divide the sample subset according to the selected features, and construct an isolation tree; Build multiple isolated trees to generate an isolation forest; For each data point in the user data set, the average path length of the data point in all isolated trees is calculated, and the data point whose average path length is less than the second threshold is marked as an outlier; Among them, calculating information gain includes: Calculate the empirical entropy of the user dataset; Calculate the conditional entropy of each statistical feature on the user data set; The difference between the empirical entropy and the conditional entropy is taken as the information gain of the corresponding statistical feature; Preprocessing also includes: Based on the obtained key feature subset, the user data set is modeled, the Gaussian mixture distribution is used to fit the user data distribution, and the parameters of the Gaussian mixture distribution are estimated by the expectation maximization algorithm; Calculate the posterior probability of data points marked as outliers in the Gaussian mixture distribution; The third threshold is set according to the ranked quantile of the posterior probability of the outlier; Filter outliers whose posterior probabilities are less than a third threshold to obtain a filtered user data set; The filtered user data set is used as the preprocessed user data set; Get local machine learning models, including: Initialize the parameters of the global machine learning model on the cloud server and distribute the initialized model parameters to each user end; On each user side, the preprocessed user data set is divided into a training set, a validation set, and a test set; Using the training set, train the neural network BiLSTM as a local machine learning model; Calculate the gradient of the loss function relative to each parameter in the BiLSTM model and accumulate the sum of the squares of the gradients; Adjust the learning rate of each parameter based on the sum of squares of the accumulated gradients; Use the adjusted learning rate to update the parameters of the BiLSTM model until the model converges or reaches the preset number of iterations; Use the validation set to evaluate the performance indicators of the BiLSTM model, including accuracy and F1 score; Upload the parameters and performance indicators of the trained BiLSTM model to the cloud server for global aggregation; Using the training set, train the neural network BiLSTM as a local machine learning model, including: The behavior data and interaction data in the training set are converted into time series data as the input sequence of the BiLSTM network; the BiLSTM network consists of a forward LSTM and a backward LSTM; The forward LSTM adjusts the forward hidden state of the current time step according to the input of the current time step and the hidden state of the previous time step; The backward LSTM adjusts the backward hidden state of the current time step according to the input of the current time step and the hidden state of the next time step; Concatenate the forward and backward hidden states to get the BiLSTM output of the current time step; The hidden state of the last time step of BiLSTM is used as the query vector of the user state; the user state includes the user's behavior pattern and abnormality level; A multi-layer perceptron (MLP) is used as the attention function to calculate the attention weight of each time step based on the output of BiLSTM and the query vector of the user state. Perform attention-weighted summation on the output sequence of BiLSTM to obtain the aggregated feature vector; The aggregated feature vector is input into the fully connected layer to calculate the probability distribution of the user state: According to the probability distribution of user status, the cross entropy loss function is calculated, and the parameters of the BiLSTM model are adjusted using the AdaGrad optimization algorithm to obtain the trained local machine learning model; Calculate the probability distribution of user status using the following formula: Among them, p i is the normalized probability value of the i-th user state, x i is the i-th output value of the aggregated feature vector after transformation by the fully connected layer, μ is the mean of the fully connected layer output after Batch Norm normalization, σ is the standard deviation after Batch Norm normalization, ε is a very small positive number, and N is the total number of user status categories.

2. The computer network security analysis method based on big data according to claim 1 is characterized in that: Get the global machine learning model, including: On the cloud server side, the weight of the local machine learning model is calculated based on the accuracy and F1 score of each local machine learning model; According to the weights, the parameters of each local machine learning model are weighted averaged to obtain the parameters of the global machine learning model; Use the test set to evaluate the aggregated global machine learning model; Distribute the aggregated parameters of the global machine learning model to each user terminal as the initial parameters for the next round of local training; The trained global machine learning model is deployed to the cloud service end and the user end for security analysis of the user's computer network behavior.

3. The computer network security analysis method based on big data according to claim 2 is characterized in that: The weight is calculated by the following formula: Among them, acc i 、f1 i 、auc i are the accuracy, F1 score and AUC value of the i-th local model respectively; a, b, c are weight coefficients; ε is the smoothing term; N is the total number of local models; Represents dot product.

4. The computer network security analysis method based on big data according to claim 3 is characterized by: Use unsupervised learning methods based on anomaly detection to conduct security analysis of users' computer network behaviors, including: Extract the location information of the user clicking on the web page in the interactive data, encode the location information using the pre-trained CNN neural network, and obtain the location feature vector for anomaly detection; The extracted location feature vector and the obtained statistical features are concatenated as the risk feature vector; The obtained risk feature vector is classified using the pre-trained SVM classifier to obtain the predicted risk level; Adjust the weight coefficients a, b, and c of the global machine learning model according to the predicted risk level; Send the adjusted model parameters to the local machine learning model of each user end; On the user side, the local machine learning model is updated using the sent parameters, and the updated local machine learning model is used to perform security analysis on the user's computer network behavior.

Images