Transform-VAE-based user behavior anomaly detection method and device

Abstract

The invention discloses a user behavior anomaly detection method and device based on Transform-VAE. The method comprises the following steps: extracting long-distance features in user behavior historical data by using a Transform encoder; splicing the long-distance features and the original data on the feature dimension to obtain a fused feature data set; inputting the fusion feature data set to a VAE encoder to obtain a local feature corresponding to a potential space variable; a cross attention mechanism is introduced, so that deep interaction is carried out on the long-distance features and the local features; according to the method, deep interaction results are classified by introducing an MLP classifier with a residual structure, the obtained parameters are adjusted and optimized by using a Bayesian optimization algorithm, and the optimal parameters are determined, so that an accurate user behavior anomaly detection result is obtained. The device comprises a processor and a memory. According to the method, the accuracy and reliability of user behavior anomaly detection are effectively improved, so that a network platform can be helped to identify security threats such as malicious login attempts, illegal access and data leakage.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of user behavior anomaly detection, and in particular to a user behavior anomaly detection method and device based on a Transformer-VAE (Variational Autoencoder). BACKGROUND

[0002] User behavior is an important factor affecting network platform security, and user behavior anomaly detection plays a crucial role in network platform security. By monitoring network traffic and user behavior in real time, anomaly detection technology can quickly detect abnormal patterns, identify malicious login attempts, illegal access and data breaches, and other security threats, thereby effectively preventing network attack incidents and ensuring system security.

[0003] With the rapid development of information technology and Internet applications, the scale of user behavior data has grown dramatically. These data not only record complex interactions between users and systems, but also hide potential security threats and abnormal behaviors. However, user behavior data has complex characteristics such as high dimensionality, time series, nonlinearity and dynamic changes, which pose higher requirements on the accuracy and generalization ability of models. Therefore, how to efficiently and accurately identify abnormal patterns in user behavior has become a key and difficult point of current research.

[0004] User behavior anomaly detection methods can be divided into two categories: non-deep learning-based methods and deep learning-based methods. Common non-deep learning-based user behavior anomaly detection algorithms include support vector machines (SVM), K-means, density clustering (DBSCAN), principal component analysis (PCA), decision trees (DT), K-nearest neighbors (KNN), isolation forests (IF), and statistical-based algorithms. Although non-deep learning methods perform well in certain specific scenarios, they often show limitations when dealing with high-dimensional, nonlinear, and time-series data. For example, PCA-based methods can reduce dimensionality, but cannot effectively capture time-series features. Clustering analysis methods and SVM, DT, KNN, IF, etc. are susceptible to the "curse of dimensionality" in high-dimensional data. Statistical-based methods are heavily dependent on assumptions, and only a small number of hypothesis testing statistics are available for anomaly detection, making it not simple to select the best statistics.

[0005] Deep learning-based methods have become the mainstream method for user behavior anomaly detection due to their strong feature extraction capabilities, modeling capabilities for complex nonlinear relationships, and processing capabilities for high-dimensional and time-series data. Common deep learning algorithms include autoencoders, generative adversarial networks, Transformers, graph neural networks, recurrent neural networks and their variants, convolutional neural networks, and capsule networks. However, single models still have problems such as weak ability to capture complex patterns, poor interpretability, and low accuracy. SUMMARY

[0006] The application provides a user behavior anomaly detection method and device based on a Transformer-VAE, which can effectively extract the time sequence, nonlinearity and dynamic change complex characteristics in user behavior data, effectively improve the accuracy and reliability of user behavior anomaly detection, and help the network platform to identify security threats such as malicious login attempts, illegal access and data leakage, effectively prevent network attack events, and ensure system security. Details are described below:

[0007] In a first aspect, a user behavior anomaly detection method based on a Transformer-VAE, the method comprising:

[0008] extracting long-distance features in user behavior history data using a Transformer encoder; and splicing the long-distance features with original data in a feature dimension to obtain a fusion feature dataset;

[0009] inputting the fusion feature dataset into a VAE encoder to obtain latent space variables corresponding to local features;

[0010] introducing a cross-attention mechanism to enable deep interaction between the long-distance features and the local features;

[0011] introducing an MLP classifier with a residual structure to classify the results of the deep interaction, using a Bayesian optimization algorithm to optimize the parameters obtained above to determine the optimal parameters, and obtaining accurate user behavior anomaly detection results.

[0012] The use of the Transformer encoder to extract long-distance features in the user behavior history data is as follows:

[0013] The input data is first converted into an embedding vector through input embedding, the attention degree of each feature is calculated through a multi-head attention mechanism, nonlinear transformation is performed through a feedforward neural network, and then residual connection and layer normalization are performed to obtain the long-distance features in the data.

[0014] The inputting of the fusion feature dataset into the VAE encoder to obtain latent space variables corresponding to local features is as follows:

[0015] Based on the VAE encoder framework, three fully connected layers are used to obtain the mean and variance, thereby obtaining the local features; the objective function of the VAE encoder is considered as a KL divergence term, which measures the difference between the posterior distribution obtained by the encoder and the prior distribution, and is used to ensure that the distribution of the latent space is close to the preset prior distribution.

[0016] Wherein, the long-distance features and local features are deeply interacted by introducing cross-attention mechanism:

[0017] In the cross-attention mechanism, the query vector comes from the latent space variable z of the VAE encoder, and the key vector and the value vector come from the long-distance features X extracted by the Transformer encoder n By calculating the dot product of the query vector and the key vector, the attention weight is obtained, and after the weight is normalized by the softmax function, the value vector is weighted and summed to obtain the fused feature representation;

[0018] The calculation method of the cross-attention mechanism includes:

[0019] Q i =zW Q

[0020] K j =X n W K , V j =X n W V

[0021] In the formula, W Q is the linear transformation matrix of Query, Q i is the projected Query matrix, W K is the linear transformation matrix of Key, K j is the projected Key vector, W V is the linear transformation matrix of Value, V j is the projected Value vector.

[0022] Wherein, the results of the deep interaction are classified by introducing an MLP classifier with residual structure:

[0023] h′1=W1h0+b1

[0024] h″1=BatchNorm(h′1)

[0025] h1=ReLU(h″1)

[0026] h′2=W2h1+b2

[0027] h2=ReLU(h′2)

[0028] h3=W3h2+b3

[0029] h residual =W residual h0+b residual

[0030] hresidual_add = h residual + h3

[0031] h norm = LayerNorm(h residual_add )

[0032]

[0033] In the formula, h'1 represents the output of the first layer full connection layer linear transformation, W1 is the weight matrix of the first layer, h o is the output vector of the cross attention mechanism, b1 is the bias vector of the first layer, h"1 represents the output after batch normalization, BatchNorm refers to batch normalization, h1 refers to the output of the first layer full connection layer, ReLU refers to the activation function, h2' represents the output of the second layer full connection layer linear transformation, W2 is the weight matrix of the second layer, b2 is the bias vector of the second layer, h2 is the output of the second layer full connection layer, h3 is the output of the third layer full connection layer, W3 is the weight matrix of the third layer, b3 is the bias vector of the third layer. residual represents the output of the residual connection layer, W residual refers to the weight matrix of the residual connection, b residual is the bias vector of the residual connection, h residual_add represents the result of the residual addition, LayerNorm represents the normalization layer processing, h norm represents the result of the normalization layer processing, h final represents the output result of the last layer classifier, represents the transpose of the last layer weight vector, T represents the transpose, and b4 represents the bias vector of the last layer classifier.

[0034] The second aspect is a user behavior anomaly detection device based on the Transformer-VAE, the device comprises a processor and a memory, the memory stores program instructions, and the processor calls the program instructions stored in the memory to make the device execute the method in any one of the first aspect.

[0035] The third aspect is a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program comprises program instructions, and the program instructions are executed by the processor to make the processor execute the method in any one of the first aspect.

[0036] The beneficial effects of the technical solutions provided by the application are:

[0037] 1、The application combines the sequence modeling capability of the Transformer and the latent space modeling capability of the VAE, splices the output of the Transformer encoder as new features with the original data in the feature dimension to form fusion features, then inputs the fusion features into the VAE encoder to obtain latent space variables, and then makes the output features of the Transformer encoder interact with the latent space variables through the cross-attention mechanism to strengthen feature fusion; finally, the features after interaction are input into the full connection classifier with residual connection to output the abnormal probability, realizing abnormal detection classification;

[0038] 2、The Transformer feature extraction module of the application is composed of multiple layers of Transformer encoders, and the dependence of feature data on a long distance is better captured by adjusting the parameters of the Transformer encoder;

[0039] 3、The VAE feature extraction module of the application is composed of a VAE encoder, and the VAE encoder is composed of three layers of full connection layers, the input of the VAE encoder is the data after splicing the features extracted by the Transformer and the original data, and the local features of the data are better captured by adjusting the parameters of the VAE encoder;

[0040] 4、The feature interaction module of the application uses the cross-attention mechanism, and the features output by the Transformer encoder and the latent space variables of the VAE are deeply interacted by introducing the cross-attention mechanism, further improving the capturing ability of the model to complex behavior patterns and the abnormal detection performance;

[0041] 5、The classifier of the application uses a full connection layer classifier with a residual structure, and the residual network structure is introduced in the multi-layer perception (MLP) classifier, effectively solving the common gradient disappearance problem in deep network training. In this way, not only the training efficiency and stability of the model are improved, but also the expression ability and feature learning effect of the model are significantly enhanced;

[0042] 6、The application uses accuracy, F1 value and ROC-AUC as three indexes to evaluate the effect of the neural network framework, and experiments show that the application has good generalization performance, high detection accuracy, and is effectively applicable to the user behavior abnormal detection task of the network platform;

[0043] 7、The application improves the accuracy and reliability of user behavior abnormal detection, helps the network platform to identify security threats such as malicious login attempts, illegal access and data leakage, effectively prevents network attack events, and safeguards system security. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1Fig. 1 is a schematic diagram of a data and user behavior anomaly detection method;

[0045] Figure 2 Fig. 2 is a schematic diagram of a user behavior anomaly detection process;

[0046] Figure 3 Fig. 3 is a schematic diagram of a structure of a Transformer-VAE;

[0047] Figure 4 Fig. 4 is a schematic diagram of a structure of a Transformer encoder;

[0048] Figure 5 Fig. 5 is a schematic diagram of a structure of a VAE encoder;

[0049] Figure 6 Fig. 6 is a schematic diagram of a structure of a cross-attention mechanism;

[0050] Figure 7 Fig. 7 is a schematic diagram of a structure of a classifier. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application are described in further detail below.

[0052] Embodiment 1

[0053] The embodiment 1 of the present application provides a user behavior anomaly detection method based on a Transformer-VAE, which comprises the following steps:

[0054] Step 101: Collect and analyze the user behavior history data of a network platform, establish a user behavior data set, and perform data preprocessing to obtain original data;

[0055] Step 102: Use a Transformer encoder to extract long-distance features in the user behavior history data;

[0056] Step 103: Concatenate the long-distance features extracted by the Transformer encoder with the original data in the feature dimension to obtain a fusion feature data set;

[0057] Step 104: Input the fusion feature data set into a VAE encoder to obtain latent space variables corresponding to local features;

[0058] Step 105: By introducing a cross-attention mechanism, the long-distance features extracted by the Transformer encoder and the local features extracted by the VAE encoder are deeply interacted, so as to further improve the capturing ability and detection performance of the model on complex behavior patterns;

[0059] Step 106: the result of the deep interaction is classified by introducing an MLP classifier with residual structure, so that the gradient vanishing problem common in deep network training can be solved while obtaining the classification result.

[0060] Step 107: the parameters in steps 102-106 are optimized by using a Bayesian optimization algorithm to determine the optimal parameters to obtain accurate user behavior anomaly detection results.

[0061] In summary, the embodiment of the application improves the accuracy and reliability of user behavior anomaly detection through steps 101-107, helps the network platform to identify security threats such as malicious login attempts, illegal access and data leakage, effectively prevents network attack events, and ensures system security.

[0062] Embodiment 2

[0063] The following specific examples are used to further introduce the scheme in embodiment 1, and the calculation formula is described below:

[0064] Step 201: collect and analyze the user behavior history data of the network platform, establish a network platform user behavior data set, and perform data preprocessing to obtain raw data;

[0065] The collected user behavior history data includes network-based features and user behavior-based features. The collected data is saved in csv file format, and Python is used to preprocess the abnormal values and missing values in the raw data to avoid the influence of missing values on the prediction result. In specific implementation, the embodiment of the application selects a certain data set on Kaggle as model input data for illustration.

[0066] The "communication protocol" feature in the data set has 1966 missing values, which represent the use of no encryption technology. Therefore, the application fills these missing values with 0.

[0067] Convert non-numeric data to numeric data, use label encoding to map each category to a unique integer, and then proceed to the next step.

[0068] According to 8:1:1, the training set, the validation set and the test set are divided, and the maximum and minimum normalization of the divided training set, the validation set and the test set is performed. The maximum and minimum normalization formula is:

[0069]

[0070] Y=X inorm *(mx-mi)+mi

[0071] In the formula, x imaxXi is the maximum value of the i-th feature of the input sample imin Xi is the minimum value of the i-th feature of the input sample inorm Xi is the normalized value of the i-th feature of the input sample, Y is the final result, mx is the maximum value of the specified interval, and mi is the minimum value of the specified interval.

[0072] Step 202: Use the Transformer encoder to extract long-distance features in user behavior data;

[0073] Referring to Figure 3 , the Transformer encoder is stacked by multiple identical layers, each layer sharing parameters. Each layer contains two main sub-layers: multi-head self-attention mechanism and feed-forward neural network. The multi-head self-attention mechanism allows the model to focus on information at different positions in the sequence simultaneously, thereby capturing dependencies between elements. The feed-forward neural network performs nonlinear transformation on the features of each position, further enhancing the expression ability of the model.

[0074] The input data is first converted into an embedding vector through input embedding, then passes through the multi-head attention mechanism to calculate the attention degree of each feature, and then passes through the feed-forward neural network for nonlinear transformation, and then passes through the residual connection and layer normalization, so as to obtain the long-distance features in the data.

[0075] The embodiment of the application uses the Transformer encoder to extract features. The Transformer encoder can effectively capture the long-distance dependencies between elements in the input sequence through the combination of multi-head self-attention mechanism and feed-forward neural network, thereby extracting feature representations rich in context information. The network model of the Transformer encoder is as shown in Figure 4 .

[0076] Step 203: Splice the long-distance features extracted by the Transformer encoder with the original data in the feature dimension to obtain a fusion feature dataset;

[0077] In the embodiment of the application, the long-distance features are spliced after the corresponding data features during splicing, thereby obtaining a fusion feature dataset that fuses the long-distance features and the original features.

[0078] Step 204: Input the fusion feature dataset into the VAE encoder to obtain local features;

[0079] The VAE encoder is a probability distribution that maps input data to a latent space, and outputs the mean and variance of the distribution. The network model of the VAE encoder is as shown in Figure 5The input is the fusion feature dataset, and the embodiment of the application uses three fully connected layers based on the VAE encoder framework to obtain the mean and variance, thereby obtaining the local features. The main goal of the VAE encoder is to learn the latent representation of the input data, so that these representations can conform to the prior distribution as much as possible. At this time, the objective function of the VAE can be regarded as a KL divergence term, which measures the difference between the posterior distribution obtained by the encoder and the prior distribution, and is used to ensure that the distribution of the latent space is close to the preset prior distribution, thereby improving the quality and diversity of the generated samples.

[0080] wherein the KL divergence is represented as follows:

[0081]

[0082] wherein, is the latent distribution output by the encoder, p(z) is the prior distribution, and d is the dimension of the latent variable z, is the variance, is the square of the mean.

[0083] Step 205: By introducing a cross-attention mechanism, the long-distance features extracted by the Transformer encoder and the local features extracted by the VAE are deeply interacted, further improving the model's ability to capture complex behavior patterns and detection performance.

[0084] Referring to Figure 6 , the cross-attention mechanism dynamically fuses the information in two feature spaces by calculating the similarity between the query vector in one feature space and the key vector and value vector in the other feature space. In the cross-attention mechanism used in the embodiment of the application, the query vector comes from the latent space variable z of the VAE encoder, and the key vector and value vector come from the long-distance features X n extracted by the Transformer encoder. By calculating the dot product of the query vector and the key vector, the attention weights are obtained. After normalization by the softmax function, these weights are used for weighted summation of the value vector, thereby obtaining the fused feature representation.

[0085] wherein the query vector comes from the latent space variable z of the VAE encoder, and the key vector and value vector come from the long-distance features X n extracted by the Transformer encoder. The calculation method of the cross-attention mechanism is as follows:

[0086] Q i = zW Q

[0087] K j = X n W K , V j = Xn W V

[0088]

[0089] α = Softmax(E)

[0090] h o =αV j

[0091] h0 = CrossAttention(z, X) n )

[0092] In the formula, W Q Q is the linear transformation matrix of the Query. i It is the projected Query matrix, W K K is the linear transformation matrix of Key. j W is the projected key vector. V V is the linear transformation matrix of Value. j is the projected Value vector, and E is the similarity score matrix, representing the similarity score between the Query and the Key. It is a scaling factor used to prevent the dot product value from becoming too large, α is the attention weight vector, and h o It is the output vector, representing the output representation after cross-attention adjustment, where z represents the latent space variables, and X... n represents the long-range features extracted by the Transformer encoder, CrossAttention represents the cross-attention mechanism, Softmax represents the activation function, and T represents the transpose.

[0093] Step 206: Introduce a residual network structure into the MLP classifier;

[0094] The specific method is as follows:

[0095] h'1=W1h0+b1

[0096] h″1 = BatchNorm(h'1)

[0097] h1 = ReLU(h″1)

[0098] h′2=W2h1+b2

[0099] h2 = ReLU(h′2)

[0100] h3=W3h2+b3

[0101] h residual =W residual h0+b residual

[0102] h residual_add =h residual +h3

[0103] h norm =LayerNorm(h residual_add )

[0104]

[0105] In the formula, h′1 represents the output of the linear transformation of the first fully connected layer, W1 is the weight matrix of the first layer, and h o This is the output vector of the cross-attention mechanism. b1 is the bias vector of the first layer, h″1 represents the output after batch normalization, and BatchNorm refers to batch normalization. h1 refers to the output of the first fully connected layer. ReLU refers to the activation function. h2' represents the output of the linear transformation of the second fully connected layer. W2 is the weight matrix of the second layer, b2 is the bias vector of the second layer, h2 is the output of the second fully connected layer, h3 is the output of the third fully connected layer, W3 is the weight matrix of the third layer, and b3 is the bias vector of the third layer. residual W represents the output of the residual connection layer. residual This refers to the weight matrix of the residual connection, b residual h is the bias vector of the residual link. residual_add This represents the result of summing the residuals, LayerNorm represents the normalization layer processing, and h norm h represents the result of the normalization layer processing. final This represents the output of the last classifier layer. b4 represents the transpose of the weight vector of the last layer, where T represents the transpose and b4 represents the bias vector of the last layer classifier.

[0106] Step 207: Use the Bayesian optimization algorithm to fine-tune the parameters of the entire model and determine the optimal parameters to obtain the user behavior anomaly detection results.

[0107] The Bayesian optimization algorithm is used for parameter tuning to determine the optimal parameters for the model. The biggest advantage of the Bayesian hyperparameter optimization algorithm lies in its efficient search for the optimal hyperparameter configuration. During the search, the algorithm searches for factors such as the feature dimension of the hidden layers, the number of heads in the multi-head attention mechanism of the Transformer encoder, the number of layers in the Transformer encoder, the dimension of the VAE latent variables, the number of heads in the cross-attention module, the learning rate, and the dropout rate to obtain the best hyperparameters.

[0108] When detecting anomalies, three metrics—accuracy, F1 score, and ROC-AUC—are used to evaluate the effectiveness of the neural network framework.

[0109] in:

[0110]

[0111] ROC-AUC is the area under the ROC curve with the false positive rate as the horizontal axis and the true positive rate as the vertical axis.

[0112] In summary, the embodiment of the application improves the accuracy and reliability of user behavior anomaly detection through the above steps, helps the network platform to identify security threats such as malicious login attempts, illegal access, and data leakage, effectively prevents network attack events from occurring, and safeguards system security.

[0113] Embodiment 3

[0114] A user behavior anomaly detection device based on Transformer-VAE, the device comprises: a processor and a memory, the memory stores program instructions, the processor calls the program instructions stored in the memory to make the device execute the following method steps in embodiment 1:

[0115] Use the Transformer encoder to extract long-distance features in user behavior history data; splice the long-distance features with the original data in the feature dimension to obtain a fusion feature dataset;

[0116] Input the fusion feature dataset into the VAE encoder to obtain latent space variables corresponding to local features;

[0117] By introducing a cross-attention mechanism, the long-distance features and the local features are deeply interacted;

[0118] By introducing an MLP classifier with a residual structure, the results of the deep interaction are classified, and the Bayesian optimization algorithm is used to optimize the parameters obtained above to determine the optimal parameters to obtain accurate user behavior anomaly detection results.

[0119] Wherein, the long-distance features in the user behavior history data extracted by the Transformer encoder are:

[0120] The input data is first converted into an embedding vector through input embedding, the attention degree of each feature is calculated through the multi-head attention mechanism, nonlinear transformation is performed through the feedforward neural network, and then the long-distance features in the data are obtained through residual connection and layer normalization.

[0121] Wherein, the long-distance features in the user behavior history data extracted by the Transformer encoder are:

[0122] The VAE encoder framework uses three layers of fully connected layers to obtain the mean and variance, thereby obtaining the local features; the objective function of the VAE encoder is regarded as a KL divergence term, which measures the difference between the posterior distribution obtained by the encoder and the prior distribution, and is used to ensure that the distribution of the latent space is close to the preset prior distribution.

[0123] Wherein, by introducing the cross attention mechanism, the long distance features and the local features are deeply interacted as:

[0124] In the cross attention mechanism, the query vector comes from the latent space variable z of the VAE encoder, and the key vector and the value vector come from the long distance features X extracted by the Transformer encoder n By calculating the dot product of the query vector and the key vector, the attention weight is obtained, and after the weights are normalized by the softmax function, the value vector is weighted and summed to obtain the fused feature representation;

[0125] The calculation method of the cross attention mechanism includes:

[0126] Q i =zW Q

[0127] K j =X n W K , V j =X n W V

[0128] In the formula, W Q is the linear transformation matrix of Query, Q i is the projected Query matrix, W K is the linear transformation matrix of Key, K j is the projected Key vector, W V is the linear transformation matrix of Value, V j is the projected Value vector.

[0129] Wherein, by introducing the MLP classifier with residual structure to classify the result of deep interaction as:

[0130] h′1=W1h0+b1

[0131] h″1=BatchNorm(h′1h1=ReLU(h″1)

[0132] h′2=W2h1+b2

[0133] h2=ReLU(h′2)

[0134] h3=W3h2+b3

[0135] h residual = W residual h0 + b residual

[0136] h residual_add = h residual + h3

[0137] h norm = LayerNorm(h residual_add )

[0138]

[0139] In the formula, h'1 represents the output of the first layer full connection layer linear transformation, W1 is the weight matrix of the first layer, h o is the output vector of the cross attention mechanism, b1 is the bias vector of the first layer, h"1 represents the output after batch normalization, BatchNorm refers to batch normalization, h1 refers to the output of the first layer full connection layer, ReLU refers to the activation function, h2' represents the output of the second layer full connection layer linear transformation, W2 is the weight matrix of the second layer, b2 is the bias vector of the second layer, h2 is the output of the second layer full connection layer, h3 is the output of the third layer full connection layer, W3 is the weight matrix of the third layer, b3 is the bias vector of the third layer. residual represents the output of the residual connection layer, W residual refers to the weight matrix of the residual connection, b residual is the bias vector of the residual connection, h residual_add represents the result of the residual addition, LayerNorm represents the normalization layer processing, h norm represents the result of the normalization layer processing, h final represents the output result of the last layer classifier, represents the transpose of the weight vector of the last layer, T represents the transpose, and b4 represents the bias vector of the last layer classifier.

[0140] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiments of the application will not be repeated here.

[0141] The execution subject of the processor and the memory described above can be a computer, a single-chip microcomputer, a microcontroller, etc. having a computing function, and in specific implementation, the embodiments of the application do not limit the execution subject, and selection is made according to the needs in actual application.

[0142] The memory and the processor transmit data signals through a bus, and the embodiments of the application will not be repeated here.

[0143] Based on the same inventive concept, the embodiment of the present application further provides a computer readable storage medium, which comprises a stored program, and the program controls a device where the storage medium is located to execute the method steps in the above embodiment when the program is running.

[0144] The computer readable storage medium includes, but is not limited to, a flash memory, a hard disk, a solid state disk, etc.

[0145] It should be noted that the readable storage medium in the above embodiment is described corresponding to the method description in the embodiment, and the embodiment of the present application will not be described here.

[0146] In the above embodiment, all or part of the embodiment can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the embodiment can be realized in the form of a computer program product. The computer program product comprises one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiment of the present application is generated.

[0147] The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in or transmitted by a computer readable storage medium. The computer readable storage medium can be any available medium accessible by the computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available medium can be a magnetic medium or a semiconductor medium, etc.

[0148] The model of each device in the embodiment of the present application is not limited unless otherwise specified, and any device that can complete the above functions can be used.

[0149] Those skilled in the art can understand that the above drawings are only schematic diagrams of a preferred embodiment, and the above embodiment serial numbers are only for description, not representing the advantages and disadvantages of the embodiment.

[0150] The above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting user behavior anomalies based on Transformer-VAE, characterized in that, The method includes: The Transformer encoder is used to extract long-range features from historical user behavior data; the long-range features are then concatenated with the original data along the feature dimension to obtain a fused feature dataset. The fused feature dataset is input into the VAE encoder to obtain the latent spatial variables corresponding to local features; By introducing a cross-attention mechanism, long-range features and local features can interact deeply. By introducing an MLP classifier with residual structure to classify the results of deep interactions, and using a Bayesian optimization algorithm to fine-tune the parameters obtained above, the optimal parameters are determined to obtain accurate user behavior anomaly detection results.

2. The user behavior anomaly detection method based on Transformer-VAE according to claim 1, characterized in that, The method of using the Transformer encoder to extract long-range features from historical user behavior data is as follows: The input data is first transformed into an embedding vector through input embedding. The attention of each feature is calculated through a multi-head attention mechanism. The data undergoes a nonlinear transformation through a feedforward neural network, followed by residual connections and layer normalization to obtain long-range features in the data.

3. The user behavior anomaly detection method based on Transformer-VAE according to claim 1, characterized in that, The process of inputting the fused feature dataset into the VAE encoder yields the latent spatial variables corresponding to the local features as follows: The VAE encoder framework uses three fully connected layers to obtain the mean and variance, thereby obtaining local features. The objective function of the VAE encoder is regarded as the KL divergence term. The KL divergence term measures the difference between the posterior distribution obtained by the encoder and the prior distribution, and is used to ensure that the distribution of the latent space is close to the preset prior distribution.

4. The user behavior anomaly detection method based on Transformer-VAE according to claim 1, characterized in that, The introduction of a cross-attention mechanism enables deep interaction between long-range features and local features as follows: In the cross-attention mechanism, the query vector comes from the latent space variable z of the VAE encoder, while the key vector and value vector come from the long-range features X extracted by the Transformer encoder. n Attention weights are obtained by calculating the dot product of the query vector and the key vector. These weights are normalized by the softmax function and then used to perform a weighted summation on the value vectors to obtain the fused feature representation. The computational methods for the cross-attention mechanism include: Q i =zW Q ; K j =X n W K ,V j =X n W V ; In the formula, W Q Q is the linear transformation matrix of the Query. i It is the projected Query matrix, W K K is the linear transformation matrix of Key. j W is the projected key vector. v V is the linear transformation matrix of Value. j It is the projected Value vector.

5. The user behavior anomaly detection method based on Transformer-VAE according to claim 1, characterized in that, The classification of deep interaction results by introducing an MLP classifier with residual structure is as follows: h'1 = w1h0 + b1; h”1 = batchNorm(h'1); h1 = ReLU(h″1); h′2=W2h1+b2; h2 = ReLU(h′2); h3 = W3h2 + b3; h residual =W residualh0 +b residual ; h residual_add =h residual +h3; h norm =LayerNorm(h residual_add ); In the formula, h′1 represents the output of the linear transformation of the first fully connected layer, W1 is the weight matrix of the first layer, and h o This is the output vector of the cross-attention mechanism. b1 is the bias vector of the first layer. h″1 represents the output after batch normalization. BathNorm refers to batch normalization. h1 refers to the output of the first fully connected layer. ReLU refers to the activation function. h2 represents the output of the linear transformation of the second fully connected layer. W2 is the weight matrix of the second layer. b2 is the bias vector of the second layer. h2 is the output of the second fully connected layer. h3 is the output of the third fully connected layer. W3 is the weight matrix of the third layer. b3 is the bias vector of the third layer. residual W represents the output of the residual connection layer. residual This refers to the weight matrix of the residual connection, b residual h is the bias vector of the residual link. residual_add This represents the result of summing the residuals, LayerNorm represents the normalization layer processing, and h norm h represents the result of the normalization layer processing. final This represents the output of the last classifier layer. b4 represents the transpose of the weight vector of the last layer, where T represents the transpose and b4 represents the bias vector of the last layer classifier.

6. A user behavior anomaly detection device based on Transformer-VAE, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-5.

Images