A network information abnormal behavior detection method based on bidirectional LLM

Abstract

The application relates to the technical field of information security and belongs to a network information abnormal behavior detection method based on a bidirectional LLM, which comprises the following steps: S1: constructing an information list of nodes, checking whether there are repeated transaction records in the list, correcting or deleting abnormal data, ensuring that all data formats are consistent, then integrating data, integrating the information of the nodes into time series data, and facilitating analysis of the change of the node behavior over time; S2: constructing a BPERT framework, inputting the integrated time series data into the BPERT framework, decomposing the node transaction information into multiple node features through the BPERT framework, and the multiple node features comprising a transaction amount, a transaction type, a transaction sender and a transaction receiver; the method has the following beneficial effects: compared with other detection methods based on a Transformer and an RNN, the fundamental limitation of a traditional unidirectional model is effectively broken through, historical transactions and future transactions are simultaneously learned and fused, the accuracy is maximally improved by 35%, and the recall rate is maximally improved by 41%.

Description

Technical Field

[0001] This invention relates to the field of information security technology, and pertains to a method for detecting abnormal network information behavior based on bidirectional LLM. Background Technology

[0002] In recent years, large language models (LLMs) based on deep learning technology have made groundbreaking progress. These models have a huge number of parameters and powerful natural language understanding and generation capabilities. They can be widely used in various language-related tasks such as text understanding, text generation, and question answering systems. my country attaches great importance to the development of artificial intelligence technologies represented by large language models, and relevant policies clearly support their research and application. In the field of network information security, with the popularization of information technology, various anomalies and attacks are becoming increasingly complex. Accurate and efficient anomaly detection of network information systems, especially critical transaction behaviors, has become an urgent need to ensure the safe and stable operation of systems. Large language models, with their superior feature extraction and context sequence understanding capabilities, provide a new technical path for detecting abnormal network information behavior. Currently, some research attempts to use pre-trained models based on the Transformer architecture as feature extractors, combined with downstream regression models, to analyze transaction features and identify potential malicious behaviors. However, existing methods typically only utilize the model to extract unidirectional feature sequences from historical transaction information. Since the impact of attacks may span the entire transaction cycle, relying solely on historical information without considering future transaction information will prevent the model from capturing the complete pattern of attack behavior, thus limiting further improvements in detection accuracy. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a method for detecting abnormal network information behavior based on bidirectional LLM.

[0004] To achieve the above objectives, the present invention provides the following technical solution: S1: Construct a list of node information: Check and process duplicate and abnormal transaction records, and integrate node information into time series data; S2: Constructing the BPERT framework: By inputting the integrated time series data into the BPERT framework, the BPERT framework decomposes the node transaction information into multiple node features, including transaction amount, transaction type, transaction sender, and receiver. S3: Constructing the embedding matrix and encoding representation: Construct an embedding layer, assign a vector representation to each node feature to form an embedding matrix, map the node feature sequence to a vector sequence, and add position encoding to obtain the initial sequence embedding vector representation of the node; S4: Pre-training: The initial sequence is embedded into the BPERT framework that represents the input vector. Before input, a certain proportion of the node features are randomly selected and replaced with a special mask label MASK to generate mask features. The sequence containing the mask label is then input into the model to complete the pre-training task. S5: Anomaly Detection: In the BPERT framework, the accuracy of the model in extracting node features is improved through pre-training. Then, the extracted node features are used as input to the random forest ensemble algorithm to complete the detection of abnormal network information behavior.

[0005] Furthermore, the information list described in S1 includes node balance, node historical behavior records, node transaction information, and node activity level.

[0006] Furthermore, the process of constructing the node feature sequence from the node features described in S2 is as follows: In the time series data epocht, the node balance Bt is represented as: (1) Where Bt−1 is the balance at the previous time point t-1, and Tt is the list of transactions that occurred at time t; The node's historical behavior record Hn is represented as: (2) Where T1:n is the sequence of transactions from the first to the nth transaction of a node, and each transaction Ti is further decomposed into multiple node features including transaction type Tt, transaction object Pt, transaction amount Dt, and transaction timestamp St; the number of transactions per node is represented as... The node activity A, defined based on transaction frequency and time interval, is expressed as: (3) Where log(N) represents the natural logarithm of the number of transactions N at each node. This represents the time span from the first transaction t1 to the last transaction tN; Using the max function, the feature sequence of a node in time series data at epoch t is represented as follows: ,in The feature sequence is used as the input to the embedding matrix.

[0007] Furthermore, in S3, the embedding matrix is ​​described by the set... It is a feature sequence, where Let be the i-th feature in the sequence, and n be the length of the sequence. The process for constructing the embedding matrix is ​​as follows: s31: Assign a random vector representation E to each feature to form an embedding matrix. E is an N×N matrix, where V is the number of different types of features and d is the embedding dimension. Each row of matrix E corresponds to the embedding vector of a feature in the sequence. s32: Through Mapping the feature f onto the embedding matrix E yields the embedding vector of feature f. , ,in The row in pointer E corresponding to feature f; It is to take each feature in sequence X The embedded vector sequence obtained by the embedding function emb, i.e. ; s33: Construct sequence embedding vectors Through the The weighted average of the embedding vectors in the sequence yields the sequence embedding vector, i.e. ; s34: The positional encoding W is a vector with the same embedding dimension, calculated based on the position: in It is the position of feature f in the sequence. It is the position encoding vector corresponding to that position. The sequence embedding representation of the position embedding information is as follows: The final sequence embedding vector yes The aggregation, that is This completes the representation of the embedding matrix.

[0008] Furthermore, the Transformer encoding layer includes a multi-head attention layer and a feedforward neural network layer. It captures the dependencies between elements in the sequence through a self-attention mechanism. Features are extracted by the multi-head attention layer and the feedforward neural network layer. The multi-head attention layer includes three matrix inputs: query Q, key K, and value V. The multi-head self-attention mechanism consists of multiple single-head attention. Each matrix is ​​obtained by adding the previously calculated feature embedding matrix and position embedding matrix and then multiplying them by the transpose of different weight matrices.

[0009] Furthermore, the specific calculation process for each matrix is ​​as follows: a preliminary attention score is obtained by calculating the dot product of Q and K, i.e. By using scaling operations, we ensure that the value of the dot product is neither too large nor too small. The scaled dot product is... (4) Using the softmax activation function to Normalization is performed to ensure that the attention scores of all outputs sum to 1 for each element in a vector Z. Define the activation function as (5) i and j are indices of elements in vector Z, then weighted and calculated. The normalized attention score is combined with the value vector V to generate the final representation of each element in the sequence, i.e. .

[0010] Furthermore, the single-head attention calculation process is as follows: n single-head attention points are set, the output vectors of all heads are concatenated, and then the concatenated vector is transformed through another linear layer. The integrated multi-head information Output is: (5) (6) The output of multi-head attention is staggered with the original input and then normalized by layers to maintain the stability of deep network training. The final output of multi-head attention is defined as FOutput, which is then linked with the original embedded input. The residuals are joined together to form the result. (7) Then, the results of the residual connection are normalized to form: (8) This ensures that the mean of each feature dimension is 0 and the standard deviation is 1. In the normalization layer, μ and σ are the mean and standard deviation of x in a specific dimension, respectively, and γ and β are learnable parameters.

[0011] Furthermore, the prediction process for the masked node features in S4 is as follows: In the node feature sequence, a certain proportion of features are randomly selected and replaced with a special mask label (MASK). The sequence containing the mask label is then input into the model. The input sequence is converted into a set of vector representations using an embedding matrix and positional encoding. The model processes the sequence through a multi-head attention mechanism. The input sequence passes through multiple Transformer encoder layers. Each Transformer encoder layer includes a self-attention mechanism, a feedforward neural network, and residual connections. For each mask position, the model predicts the original node features. During the pre-training phase, it is hoped that the model can predict the behavior in both of these periods to complete the training.

[0012] Furthermore, in the model training phase described in S5, half of the feature set output by the BPERT framework is used as the training set, and the other half is used as the test set. The model will construct multiple decision trees based on the samples and target variables in the training set, and perform feature selection and partitioning on each tree. A subset is randomly selected to form a subsample set. For each node of each decision tree, a subset of features is randomly selected. A decision tree is constructed using the CART algorithm on each subsample set. During the growth of the decision tree, the best splitting feature is recursively selected to divide the dataset into subsets with the minimum impurity. Averaging is used to integrate multiple decision trees, assuming the training dataset is... ,in For feature vectors, If the target value is given, then the probability of node i launching a 51% attack in one epoch is... for: (9) Where T is the number of decision trees, This is the prediction result of the t-th decision tree for the input x: Using the R-squared coefficient of determination To evaluate the accuracy of the predicted values, The value is between 0 and 1, which is relevant to the random forest ensemble algorithm. The calculation formula is: (10) ,in It is the i-th true value. It is the predicted value for the i-th true value. It is the average of the true values.

[0013] Compared with existing technologies, this invention provides a method for detecting abnormal network information behavior based on bidirectional LLM, which has the following beneficial effects: 1. Compared with other detection methods based on Transformer and RNN, it effectively overcomes the fundamental limitations of traditional one-way models. The model can learn and integrate historical transactions and future transactions simultaneously, resulting in an improvement of up to 35% in accuracy and up to 41% in recall. 2. By establishing the BPERT framework, high-level, abstract deep feature representations can be automatically learned and decomposed from raw, structured transaction data. End-to-end deep feature extraction and effective fusion improve the efficiency of feature engineering. 3. The masked node feature prediction task mimics the idea of ​​masked language model in natural language processing, which can achieve faster and more stable convergence and obtain better generalization performance, especially when dealing with new and variant attacks. The features and advantages of the present invention will be described in detail through embodiments and in conjunction with the accompanying drawings. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a network information abnormal behavior detection method based on bidirectional LLM according to the present invention.

[0015] Figure 2 This is a schematic diagram of a novel bidirectional feature encoding representation method based on Transformer according to the present invention.

[0016] Figure 3 This is a schematic diagram of a single-head attention mechanism.

[0017] Figure 4 This is a schematic diagram of the Random Forest ensemble learning algorithm. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0019] Specific implementation method one: Combining Figures 1 to 4 This embodiment describes a method for detecting abnormal network information behavior based on bidirectional LLM, which includes the following steps: S1: Data collection and preprocessing are the foundation for building a large language model. Therefore, before building BPERT, it is necessary to build a list of node information, including node balance, node historical behavior records, node transaction information, and node activity. Check the list for duplicate transaction records and delete them. Check the list for abnormal transactions or balance changes and delete or correct abnormal data. At the same time, ensure that all data formats are consistent. Then, integrate the data to integrate the node information into time series data to facilitate the analysis of changes in node behavior over time. S3: Construct an embedding layer, assign a vector representation to each node feature to form an embedding matrix, map the node feature sequence to a vector sequence, and add position encoding to obtain the initial sequence embedding vector representation of the node; S4: Pre-training: The initial sequence is embedded into the BPERT framework of the vector representation input. Before input, a certain proportion of the node features are randomly selected and replaced with a special mask label MASK to generate mask features. The sequence containing the mask label contains both the actual node features and the mask positions to be predicted. The model uses an embedding layer and positional encoding to convert the input sequence into a set of vector representations. This set of vector representations captures the semantic and positional information of the node features. The model processes the sequence through a multi-head attention mechanism, which allows the relationship between node features to be captured in multiple representation subspaces at the same time. S5: Anomaly Detection: In the BPERT framework, the accuracy of the model in extracting node features is improved through pre-training. Then, the extracted node features are used as input to the random forest ensemble algorithm. Multiple decision trees are constructed to perform classification or regression prediction. That is, the overall prediction accuracy and robustness are improved by combining the prediction results of multiple models. In the random forest, each decision tree is trained independently on randomly selected subsamples, which can effectively reduce the risk of overfitting. In summary, the Transformer-based bidirectional feature encoding representation method (BPERT framework) enables the model to simultaneously learn and fuse feature representations of historical and future transactions by designing a bidirectional pre-training task. This means that the feature vector extracted by the model for each transaction node contains behavioral information about its behavior throughout the past, present, and predictable future trends, resulting in an improvement of up to 35% in accuracy and up to 41% in recall. Through a hybrid architecture composed of the BPERT framework and random forest, a high-precision and robust detection system is established, effectively preventing or curbing security incidents such as data breaches, financial fraud, and service interruptions, and effectively ensuring the confidentiality, integrity, and availability of network information systems.

[0020] Specific Implementation Method Two: Combining Figures 1 to 4 This embodiment describes an information list in S1 that includes node balance, node historical behavior records, node transaction information, and node activity. The node transaction information includes transaction amount, transaction type, transaction sender, and receiver.

[0021] Specific implementation method three: Combining Figures 1 to 4 This embodiment describes how the node feature representation in S2 essentially transforms node information into a format that a mathematical model can process. For preprocessed data, at time series data epoch t, ​​the node balance Bt is represented as: (1) Where Bt−1 is the balance at the previous time point t-1, and Tt is the list of transactions that occurred at time t; The node's historical behavior record Hn is represented as: (2) Where T1:n is the sequence of transactions from the first to the nth transaction of a node, and each transaction Ti is further decomposed into multiple node features including transaction type Tt, transaction object Pt, transaction amount Dt, and transaction timestamp St; the number of transactions per node is represented as... The node activity A, defined based on transaction frequency and time interval, is expressed as: (3) Here, log(N) represents the natural logarithm of the number of transactions N of a node. This reduces the direct impact of the number of transactions, making the measurement of activity smoother and less sensitive to nodes with a large number of transactions. This represents the time span from the first transaction t1 to the last transaction tN; Using the max function ensures that calculations can be performed even when the time span is zero. The feature sequence of a node in time series data at epoch t is represented as follows: ,in The feature sequence can be directly used as input to the embedding layer for subsequent operations.

[0022] Specific implementation method four: Combination Figures 1 to 4 This embodiment describes how the embedding matrix in step S3 primarily functions to convert input data into a numerical vector form that the model can process, serving as input to the Transformer encoding layer. The process for constructing the embedding matrix is ​​as follows: s31: Through the set It is a feature sequence, where Let V be the i-th feature in the sequence, n be the length of the sequence, and assign a random vector representation E to each feature to form an embedding matrix. E is an n x n matrix, where V is the number of different types of features, d is the embedding dimension, and each row of matrix E corresponds to the embedding vector of a feature in the sequence. s32: Construct the embedded function, through Mapping the feature f onto the embedding matrix E yields the embedding vector of feature f. , ,in The row in pointer E corresponding to feature f; It is to take each feature in sequence X The embedded vector sequence obtained by the embedding function emb, i.e. ; s33: Construct sequence embedding vectors Through the The weighted average of the embedding vectors in the sequence yields the sequence embedding vector, i.e. ; s34: Add positional encoding and output the final sequence embedding vector representation. In the sequence model, positional encoding is added to store the positional information of elements in the sequence. The positional encoding W is a vector with the same dimension as the embedding, calculated based on the position. in It is the position of feature f in the sequence. It is the position encoding vector corresponding to that position; The sequence embedding representation of position embedding information is as follows: The final sequence embedding vector yes The aggregation, that is This completes the representation of the embedding matrix.

[0023] Specific Implementation Method Five: Combining Figures 1 to 4 This embodiment describes the Transformer encoding layer, which includes a multi-head attention layer and a feed-forward neural network layer. It captures dependencies between elements in a sequence through a self-attention mechanism, regardless of whether these dependencies are local or long-range. Features are extracted by the multi-head self-attention mechanism and the feed-forward neural network layer. The core idea of ​​the multi-head attention mechanism is to allow the model to consider information from other elements in the sequence when processing each element. The multi-head attention layer includes three matrix inputs: query Q, key K, and value V. The multi-head self-attention mechanism consists of multiple single-head attention layers. Each matrix is ​​obtained by adding the previously calculated feature embedding matrix and position embedding matrix, and then multiplying it by the transpose of different weight matrices. The weights are learned using a linear neural network layer.

[0024] Specific Implementation Method Six: Combination Figures 1 to 4 This embodiment describes the specific calculation process for each matrix as follows: A preliminary attention score is obtained by calculating the dot product of Q and K, i.e. To ensure numerical stability, a scaling operation is used to ensure that the dot product value is neither too large nor too small. The scaled dot product is... (4) Using the softmax activation function to Normalization is performed to ensure that the attention scores of all outputs sum to 1 for each element in a vector Z. Define the activation function as (5) i and j are indices of elements in vector Z, then weighted and calculated. The normalized attention score is combined with the value vector V to generate the final representation of each element in the sequence, i.e. .

[0025] Specific implementation method seven: Combination Figures 1 to 4 This embodiment describes the single-head attention calculation process as follows: There are n single-head attention points. The output vectors of all single-head attention points are concatenated. Then, the concatenated vector is transformed through another linear layer. The integrated multi-head information output is: (5) (6) The output of multi-head attention is staggered with the original input and then normalized by layers to maintain the stability of deep network training. The final output of multi-head attention is defined as FOutput, which is then linked with the original embedded input. Perform residual join and summation. (7) Then, the results of the residual connection are normalized to form: (8) This ensures that the mean of each feature dimension is 0 and the standard deviation is 1. In the normalization layer, μ and σ represent the mean and standard deviation of x in a specific dimension, respectively, while γ and β are learnable parameters. Through residual connections and normalization, deep Transformer networks can be trained effectively while maintaining performance and generalization ability.

[0026] Specific implementation method eight: Combination Figures 1 to 4 This embodiment describes the masked node feature prediction in S4. The prediction process is as follows: In the node feature sequence, a feature of proportion one is randomly selected and replaced with a special mask marker MASK. A specific example of a sequence containing a selected mask marker MASK is shown below: Suppose node A has the following behavior sequence in the first 8 epochs, where H represents honesty and D represents dishonesty. The behavior of node A in epochs 5 and 8 is masked using a mask. The resulting original sequence and the mask are applied as follows: Original sequence: [H, H, D, H, D, H, D, H]; Mask application: [H, H, D, H, MASK, H, D, MASK]; The sequence containing the mask labels is then input into the model. This sequence contains both the actual node features and the mask positions to be predicted. The model uses an embedding matrix and positional encoding to convert the input sequence into a set of vector representations that capture the semantic and positional information of the node features. The model processes the sequence through a multi-head attention mechanism, allowing the relationship between node features to be captured simultaneously in multiple representation subspaces. The input sequence passes through multiple Transformer encoder layers, each of which includes a self-attention mechanism, a feedforward neural network, and residual connections. For each mask position, the model predicts the original node features. During the pre-training phase, the model is expected to predict the behavior at both times to complete the training.

[0027] Specific Implementation Method Nine: Combining Figures 1 to 4 This embodiment describes the model training phase described in S5. Pre-training improves the accuracy of feature extraction. The extracted features are used as input to the Random Forest Ensemble Algorithm to determine the probability (51%) of each node launching a proportional-to-two attack in the next epoch t. The Random Forest Ensemble Algorithm is the second key component of the 51% attack detection model. It constructs multiple decision trees for classification or regression prediction, and combines the prediction results of multiple models to improve overall prediction accuracy and robustness. In the Random Forest, each decision tree is trained independently on randomly selected subsamples, effectively reducing the risk of overfitting. The specific model training process is as follows: First, half of the feature set output by the BPERT framework is used as the training set, and the other half is used as the test set. The model will build multiple decision trees based on the samples and target variables in the training set, and perform feature selection and partitioning on each tree. A decision tree is built using the CART algorithm on each sub-sample set. During the growth of the decision tree, the best splitting feature is recursively selected to divide the dataset into subsets with the minimum impurity. For each node of each decision tree, when selecting the best splitting feature, only a subset of features is randomly selected. This prevents any single feature from having an excessive impact on the entire model. For the training set, a subset is randomly selected to form a subsample set. This allows each decision tree to be trained on a different sample set, thereby increasing the diversity of the model. The specific testing and calculation process is as follows: Averaging is used to integrate multiple decision trees, assuming the training dataset is... ,in For feature vectors, If the target value is given, then the probability of node i launching a 51% attack in one epoch is... for: (9) Where T is the number of decision trees, This is the prediction result of the t-th decision tree for the input x: Using the R-squared coefficient of determination To evaluate the accuracy of the predicted values, A value between 0 and 1 indicates how much of the variance of the target variable the model explains. A value of 0% indicates that there is no data variability, and the prediction result is the same as the mean of the target variable. A value of 100% indicates that the model perfectly fits all data points. This is theoretically possible, but in practice, it may lead to overfitting. By continuously adjusting the model parameters based on the value, the optimal value can be found, achieving the best model fit. For the random forest ensemble algorithm The calculation formula is: (10) ,in It is the i-th true value. It is the predicted value for the i-th true value. It is the average of the true values.

[0028] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting abnormal network information behavior based on bidirectional LLM, characterized in that, Includes the following steps: S1: Construct a list of node information: Check and process duplicate and abnormal transaction records, and integrate node information into time series data; S2: Constructing the BPERT framework: By inputting the integrated time series data into the BPERT framework, the BPERT framework decomposes the node transaction information into multiple node features, including transaction amount, transaction type, transaction sender, and receiver. S3: Constructing the embedding matrix and encoding representation: Construct an embedding layer, assign a vector representation to each node feature to form an embedding matrix, map the node feature sequence to a vector sequence, and add position encoding to obtain the initial sequence embedding vector representation of the node; S4: Pre-training: The initial sequence is embedded into the BPERT framework that represents the input vector. Before input, a certain proportion of the node features are randomly selected and replaced with a special mask label MASK to generate mask features. The sequence containing the mask label is then input into the model to complete the pre-training task. S5: Anomaly Detection: In the BPERT framework, the accuracy of the model in extracting node features is improved through pre-training. Then, the extracted node features are used as input to the random forest ensemble algorithm to complete the detection of abnormal network information behavior.

2. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 1, characterized in that: The information list of nodes in S1 includes: node balance, node historical behavior records, node transaction information, and node activity level.

3. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 1, characterized in that: The construction of multiple node feature sequences in S2 includes: for a node in the time series data epoch t, ​​calculating the balance B within the node. t Historical behavior records H n The number of transactions N and the activity level A form a characteristic sequence. Where log(N) represents the natural logarithm of the number of transactions N at each node. This represents the time from the first transaction t1 to the last transaction t. N The time span between; Using the max function, the feature sequence of a node in time series data at epoch t is represented as follows: ,in The feature sequence is used as the input to the embedding matrix.

4. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 1, characterized in that: The process for constructing the embedding matrix described in S3 is as follows: s31: By embedding functions feature sequence Each feature in the vector is mapped to a corresponding embedding vector, resulting in a sequence of embedding vectors, i.e. ; s32: for Different types of feature assignment embedding dimensions d The random vector representation, forming a dimension of The embedding matrix E; s33: Construct sequence embedding vectors Through the The weighted average of the embedding vectors in the sequence yields the sequence embedding vector, i.e. ; s34: Based on sequence position Calculated position encoding vector The two vectors are added together to generate the final sequence embedding vector representation, thus completing the representation of the embedding matrix.

5. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 1, characterized in that: The Transformer encoding layer includes a multi-head attention layer and a feedforward neural network layer, which encode the initial sequence embedding vector and extract the dependencies between features.

6. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 5, characterized in that: The multi-head self-attention mechanism is implemented by concatenating and linearly transforming the outputs of multiple single-head attention mechanisms. The calculation process of each single-head attention mechanism includes: generating a query matrix Q, a key matrix K, and a value matrix V based on the input vector; calculating the dot product of the query matrix and the key matrix and scaling it; then normalizing it using the Softmax function to obtain the attention weights; and finally performing a weighted summation of the value matrix.

7. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 1, characterized in that: The mask node feature prediction task described in step S4 involves the following prediction process: inputting the sequence containing the mask label into the BPERT model, processing the sequence through a multi-head attention mechanism, and passing the input sequence through multiple Transformer encoder layers. Each Transformer encoder layer includes a self-attention mechanism, a feedforward neural network, and residual connections. For each mask position, the model predicts the original node features. During the pre-training phase, it is hoped that the model can predict the behavior in both of these periods to complete the training.

8. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 1, characterized in that: The random forest ensemble algorithm process described in S5 is as follows: s51: The feature set output by the BPERT framework is divided into a training set and a test set. Multiple decision trees are constructed based on the samples and target variables in the training set, and feature selection and partitioning are performed on each tree. s52: Randomly select a portion of them to form a subset. For each node of each decision tree, randomly select a subset of features. Use the CART algorithm to build a decision tree on each subset. During the growth of the decision tree, divide the dataset into subsets with the minimum impurity. s53: The final probability of abnormal behavior is obtained by integrating the outputs of all decision trees through averaging. .

9. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 1, characterized in that: The prediction accuracy of the random forest model was evaluated using the coefficient of determination R-squared, and the model parameters were adjusted based on the evaluation results.

10. The network information abnormal behavior detection method based on bidirectional LLM as described in claim 1, characterized in that: The abnormal behavior detection specifically refers to detecting the possibility of a node launching a ratio-2 attack, that is, predicting the possibility that a node will accumulate more than two times the total network computing power in the next time period.

Images