Method and device for improving text sequence single classification anomaly detection capability

Abstract

The invention discloses a method and a device for improving text sequence single classification anomaly detection capability, and belongs to the technical field of natural language processing anomaly detection. The method comprises the following steps: introducing a lightweight one-dimensional convolution module into a context vector data description (CVDD) framework, sequentially performing local feature enhancement on an embedded sequence of an input text, and sending the enhanced sequence into a multi-head self-attention network to obtain a context representation matrix; a learnable semantic center is initialized for each attention head, an abnormal score is obtained by calculating a cosine distance between context representation and the corresponding semantic center, distance loss and an orthogonal penalty term are taken as total loss, and center and network parameters are updated from the upper end to the end of a training set only containing positive samples; on the verification set, the mean value of abnormal scores of the training set is used as an initial threshold value, a multiple step multiplicative search strategy is adopted, and the F1 score is maximized to determine an optimal decision threshold value; and finally, completing anomaly detection on the test set and outputting a performance index.

Description

Technical Field

[0001] This invention belongs to the field of anomaly detection, and in particular relates to a method and apparatus for improving the ability of single-class anomaly detection in text sequences. Background Technology

[0002] With the explosive growth of text data, how to automatically identify abnormal content (such as fraudulent information, illegal text, and abnormal logs) from massive amounts of text has become an important research topic in the field of natural language processing. In scenarios where only normal samples can be obtained during the training phase, such tasks are usually modeled as a single-class anomaly detection problem: learning the boundary of the normal semantic space using only normal samples, and then judging samples that deviate from the boundary as abnormal.

[0003] Existing deep learning-based text anomaly detection techniques include methods based on pre-trained word embeddings, end-to-end deep single-classification models, and self-attention single-classification models. Methods based on pre-trained word embeddings use static word vectors from Word2Vec, GloVe, etc., as input, and obtain sentence-level vector representations through recurrent neural networks (RNNs), convolutional neural networks (CNNs), or Transformer encoders. These are then combined with algorithms such as single-class support vector machines (OC-SVM) or support vector data description (SVDD) to learn hypersphere boundaries in the vector space. End-to-end deep single-classification models, such as Deep Support Vector Data Description (DeepSVDD), directly couple neural network feature extraction with hypersphere optimization to achieve joint feature-boundary learning. Self-attention single-classification models, such as Context Vector Data Description (CVDD), use pre-trained word embeddings as input and employ a multi-head self-attention mechanism to learn multiple context centers. Anomaly scores are obtained by calculating the weighted cosine distance from the text to each center. This model can capture long-distance dependencies while maintaining high inference speed.

[0004] However, the aforementioned existing technologies still have the following shortcomings:

[0005] 1. Weak ability to capture local features: Although models based on self-attention mechanisms (such as CVDD) can effectively model long-distance dependencies, they are not good at capturing local short-distance features (such as semantic relationships between adjacent words), resulting in poor detection of local anomalies (such as sensitive word insertion and character-level tampering).

[0006] 2. Performance and efficiency are difficult to balance: Although convolutional neural networks (CNNs) can capture short-range features through local receptive fields, the large number of parameters and high computational complexity of multi-layer convolutional structures lead to a significant increase in inference latency, making it difficult to meet the requirements of industrial-grade high-throughput and low-latency applications.

[0007] 3. Insufficient application of attention and convolutional fusion architecture in single-classification tasks: The existing fusion structure of attention mechanism and convolutional neural network is mainly designed for supervised multi-classification tasks. Its training objectives, loss function and sample distribution are fundamentally different from those of single-classification anomaly detection tasks, and cannot be directly applied to unsupervised or single-classification anomaly detection scenarios. Summary of the Invention

[0008] Purpose of the invention: This invention provides a method and apparatus for improving the single-class anomaly detection capability of text sequences, aiming to solve the problem of missed fine-grained anomalies caused by insufficient local feature capture in existing single-class anomaly detection models.

[0009] Technical solution: This invention provides a method for improving the single-class anomaly detection capability of text sequences, including:

[0010] S1. Obtain text sequence data, and divide the text sequence data into a training set, a test set, and a validation set, wherein the training set contains only positive sample data, and the test set and validation set contain both positive and negative sample data; preprocess the text sequence data and convert it into an embedded tensor sequence;

[0011] S2. Input the embedded tensor sequence into the convolution module, extract local features along the sequence length direction to obtain the enhanced tensor sequence;

[0012] S3. Input the enhanced tensor sequence into the multi-head self-attention module, extract local weighted features from each attention head, generate an initial context representation matrix, and fuse the initial context representation matrices of all attention heads to generate a fused context representation matrix.

[0013] S4. Build a semantic model, initialize a semantic center for each attention head, calculate the cosine distance between each initial context representation matrix and the corresponding semantic center, and obtain the anomaly score based on the cosine distance; construct a total loss function based on the anomaly score and the orthogonal penalty term in the training set, take minimizing the total loss function as the training objective, iteratively update the parameters of the semantic center and the parameters of the multi-head self-attention module, and obtain the trained semantic model.

[0014] S5. Using the average of the abnormal scores in the training set as the initial decision threshold, calculate the F1 score of the trained semantic model in the validation set, and adjust the decision threshold to maximize the F1 score as the optimization objective to obtain the optimal decision threshold.

[0015] S6. Test the trained semantic model on the test set using the optimal decision threshold, evaluate the performance of the trained semantic model, and output the evaluation results.

[0016] Furthermore, the preprocessing of the text sequence data and its conversion into an embedded tensor sequence includes:

[0017] The text sequences in the training set, validation set, and test set are segmented into words to obtain corresponding word sequences, and a vocabulary is constructed based on the word sequences in the training set. Using the vocabulary, the word sequences in the training set, validation set, and test set are mapped into index sequences, and an embedding vector sequence is extracted from the index sequences using a pre-trained word embedding matrix. The embedding vector sequence is then padded with a preset length to obtain an embedding tensor sequence.

[0018] Furthermore, S2 includes:

[0019] Perform a one-dimensional convolution on the embedded tensor sequence along the sequence direction to generate a local feature map;

[0020] The local feature map is added element by element to the embedded tensor sequence to form a residual output;

[0021] The residual output is layer normalized to obtain an enhanced tensor sequence with enhanced local features.

[0022] Furthermore, S3 includes:

[0023] Within each attention head, the enhanced tensor sequence is linearly mapped to generate a query matrix, a key matrix, and a value matrix composed of a query vector, a key vector, and a value vector, respectively.

[0024] The query matrix and key matrix are multiplied by a dot product, scaled, and normalized to obtain the attention weight matrix. The value matrix is ​​then weighted and summed using the attention weight matrix to obtain the initial context representation matrix of each attention head. The initial context representation matrices of each attention head are then concatenated to obtain the fused context representation matrix.

[0025] Furthermore, S4 includes:

[0026] The semantic centers of each attention head are initialized using the K-means++ algorithm, and the parameters of the multi-head self-attention module are randomly initialized.

[0027] Calculate the cosine distance between each initial context representation matrix and the semantic center of the corresponding attention head;

[0028] The cosine distance is normalized using Softmax to obtain the context weight, and the cosine distance is weighted and summed using the context weight to obtain the anomaly score;

[0029] A total loss function is constructed in the training set, which is the sum of a distance penalty term and an orthogonality penalty term; the distance penalty term is the average of the anomaly scores of the training samples in the current round; the orthogonality penalty term is the average of the cosine similarity between each pair of semantic centers of each attention head in the current round.

[0030] With minimizing the total loss function as the training objective, the parameters of the semantic center and the multi-head self-attention module are updated through backpropagation. If the training convergence condition is not met, the parameters of the semantic center and the multi-head self-attention module are updated iteratively, and the Softmax temperature parameter is dynamically adjusted during the iteration process. If the training convergence condition is met, the trained semantic model is obtained.

[0031] Furthermore, S5 includes:

[0032] The validation set includes a validation set sample set and validation set sample results; using the average anomaly score in the training set as the initial decision threshold, the validation set sample set is input into the trained semantic model to obtain the prediction results of the validation set sample set; based on the prediction results and the validation set sample results, the initial F1 score is calculated.

[0033] The initial decision threshold is adjusted according to a preset adjustment multiple range to obtain multiple candidate decision thresholds, and the F1 score corresponding to each decision threshold is calculated.

[0034] Compare the F1 scores corresponding to all decision thresholds, and take the decision threshold with the highest F1 score as the optimal decision threshold.

[0035] Furthermore, S6 includes:

[0036] The test set includes a test set sample set and test set sample results; the test set sample set is input into the trained semantic model to obtain the anomaly score of each test set sample in the test set sample set; if the anomaly score is greater than the optimal decision threshold, the text sequence data is determined to be abnormal, otherwise it is normal;

[0037] Based on the judgment results of the trained semantic model and the test set sample results, the accuracy, recall, F1 score, and confusion matrix are calculated and the calculation results are output.

[0038] Furthermore, the convolution module described in S2 is a parallel multi-scale one-dimensional convolution structure: it uses three types of convolution kernels, namely 1×3, 1×5, and 1×7, to convolve the same embedding tensor in the sequence length direction. After batch normalization and activation function processing, each kernel is concatenated or weighted in the feature dimension to obtain a convolution output that fuses multi-scale local features. Then, the convolution output is added to the input embedding tensor by adding the residual to obtain the enhanced tensor sequence.

[0039] Furthermore, the convolutional module and the multi-head self-attention module are replaced with a local-global dual-branch attention module. The local-global dual-branch attention module performs local self-attention and global self-attention in parallel on the same embedding tensor. Local self-attention opens the attention window for each token only for the k tokens before and after it to obtain the local context representation matrix. Global self-attention allows each token to perform attention operations with all tokens in the entire sequence to obtain the global context representation matrix. Subsequently, the local context representation matrix and the global context representation matrix are concatenated or weighted and fused along the feature dimension to generate a fused context representation matrix that contains both short-range and long-range semantic information.

[0040] The present invention also provides an apparatus for improving the single-class anomaly detection capability of text sequences, comprising:

[0041] The data preprocessing module is used to acquire text sequence data, divide the text sequence data into training set, test set and validation set, wherein the training set contains only positive sample data, and the test set and validation set contain both positive sample data and negative sample data; and preprocess the text sequence data and convert it into embedded tensor sequences.

[0042] The convolution module is used to input the embedded tensor sequence into the convolution module, extract local features along the sequence length direction, and obtain the enhanced tensor sequence;

[0043] The self-attention module is used to input the enhanced tensor sequence into the multi-head self-attention module, extract local weighted features from each attention head, generate an initial context representation matrix, and fuse the initial context representation matrices of all attention heads to generate a fused context representation matrix.

[0044] The model training module is used to build a semantic model. It initializes a semantic center for each attention head, calculates the cosine distance between each initial context representation matrix and the corresponding semantic center, and obtains anomaly scores based on the cosine distance. In the training set, a total loss function is constructed based on the anomaly scores and orthogonal penalty terms. The training objective is to minimize the total loss function. The parameters of the semantic center and the parameters of the multi-head self-attention module are iteratively updated to obtain the trained semantic model.

[0045] The anomaly detection module is used to calculate the F1 score of the trained semantic model on the validation set, using the average anomaly score in the training set as the initial decision threshold. The decision threshold is adjusted to maximize the F1 score as the optimization objective, and the optimal decision threshold is obtained.

[0046] The results output module is used to test the trained semantic model on the test set using the optimal decision threshold, evaluate the performance of the trained semantic model, and output the evaluation results.

[0047] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0048] 1. Apply a convolution to the input tensor along the length of the sample sequence to capture short-distance feature relationships between neighboring words in the sample;

[0049] 2. Map the convolution output to the same dimension as the input tensor, add the residual with the original tensor and perform LayerNorm normalization to generate a new tensor as the input of the self-attention module, ensuring that local enhancement does not destroy the original feature expression and taking stability into account.

[0050] 3. The lightweight Conv1d→ReLU→1×1 projection module is selected, which has significantly lower parameter count and implementation complexity than typical multi-core parallel CNNs, thereby improving performance while controlling model overhead and adapting to industrial needs.

[0051] 4. The threshold selection is no longer based on experience, but instead uses a validation set to compare and analyze the classification results under different thresholds, and selects the threshold corresponding to the best F1 score. Attached Figure Description

[0052] Figure 1 This is a structural framework diagram of the present invention.

[0053] Figure 2 This is a flowchart of the data preprocessing process of the present invention.

[0054] Figure 3 This is a flowchart of the trainable parameter optimization process for the model of the present invention.

[0055] Figure 4 This is a flowchart of the anomaly detection module of the present invention.

[0056] Figure 5 This is a structural diagram of an optional convolution module in this invention.

[0057] Figure 6 This is a structural framework diagram of the present invention, which includes a local-global dual-branch attention module. Detailed Implementation

[0058] The method for improving the single-class anomaly detection capability of text sequences according to the present invention includes:

[0059] S1. Obtain text sequence data, and divide the text sequence data into a training set, a test set, and a validation set, wherein the training set contains only positive sample data, and the test set and validation set contain both positive and negative sample data; preprocess the text sequence data and convert it into an embedded tensor sequence;

[0060] S2. Input the embedded tensor sequence into the convolution module, extract local features along the sequence length direction to obtain the enhanced tensor sequence;

[0061] S3. Input the enhanced tensor sequence into the multi-head self-attention module, extract local weighted features from each attention head, generate an initial context representation matrix, and fuse the initial context representation matrices of all attention heads to generate a fused context representation matrix.

[0062] S4. Build a semantic model, initialize a semantic center for each attention head, calculate the cosine distance between each initial context representation matrix and the corresponding semantic center, and obtain the anomaly score based on the cosine distance; construct a total loss function based on the anomaly score and the orthogonal penalty term in the training set, take minimizing the total loss function as the training objective, iteratively update the parameters of the semantic center and the parameters of the multi-head self-attention module, and obtain the trained semantic model.

[0063] S5. Using the average of the abnormal scores in the training set as the initial decision threshold, calculate the F1 score of the trained semantic model in the validation set, and adjust the decision threshold to maximize the F1 score as the optimization objective to obtain the optimal decision threshold.

[0064] S6. Test the trained semantic model on the test set using the optimal decision threshold, evaluate the performance of the trained semantic model, and output the evaluation results.

[0065] In this embodiment,

[0066] like Figure 2 S1 includes:

[0067] Obtain text sequence data and divide the text sequence data into a training set, a test set, and a validation set. The training set contains only positive sample data, while the test set and validation set contain both positive and negative sample data. Perform word segmentation on the text sequences in the training set, validation set, and test set to obtain the corresponding word sequence.

[0068] A vocabulary is constructed based on the word sequence of the training set; continuous text sequences are decomposed into discrete tokens to form a lexicon. Tokens can be characters, words, subwords, etc. Additional elements should be added to the lexicon. <pad>,and <unk>Two tokens are used, one to unify the sample sequence length and the other to represent an unknown token encountered during testing;

[0069] Using the vocabulary, the word sequences in the training set, validation set, and test set are mapped to index sequences; the one-hot encoding method is applied to construct a token-index mapping table and generate an N-dimensional sparse vector representation, where N is the total number of tokens;

[0070] By using a pre-trained word embedding matrix, an embedding vector sequence is extracted from the index sequence; a pre-trained language model is then applied to map discrete lexical units into continuous D-dimensional word embeddings, resulting in an embedding matrix. This reduces the burden of training from scratch;

[0071] Using a preset length L as a reference, the embedded vector sequence is filled to obtain the embedded tensor sequence;

[0072] Divide each set of B samples into batches to obtain the tensors that the model can process. , used as input for the convolution module.

[0073] S2 includes:

[0074] Perform a one-dimensional convolution on the embedded tensor sequence X along the sequence direction to generate a local feature map;

[0075] The local feature map is added element by element to the embedded tensor sequence to form a residual output;

[0076] The residual output is layer-normalized to obtain an enhanced tensor sequence with enhanced local features. .

[0077] S3 includes:

[0078] Within each attention head h, a linear mapping is performed on the augmented tensor sequence Y. , , Generate query matrix Q, key matrix K, and value matrix V, which are composed of query vector, key vector, and value vector, respectively;

[0079] The attention weight matrix is ​​obtained by performing a dot product on the query matrix Q and the key matrix K, followed by scaling and normalization. The initial context representation matrix of each attention head is obtained by weighting and summing the value matrix using the attention weight matrix. The initial context representation matrices of all attention heads are concatenated to generate a fused context representation matrix.

[0080] like Figure 3 As shown, the trainable parameters of this invention mainly include n semantic centers. Attention grid parameters S4 includes:

[0081] The semantic centers of each attention head are initialized using the K-means++ algorithm, and the parameters of the multi-head self-attention module are randomly initialized.

[0082] Calculate the cosine distance between each initial context representation matrix and the semantic center of the corresponding attention head;

[0083] The cosine distance is normalized using Softmax to obtain the context weight, and the anomaly score is obtained by weighted summation of the cosine distance using the context weight, as shown in the formula:

[0084] ;

[0085] in, These are abnormal scores. Cosine distance The weight, The number of semantic centers. The initial context representation matrix and corresponding semantic center of the i-th attention head The cosine distance;

[0086] A total loss function is constructed in the training set, which is the sum of a distance penalty term and an orthogonality penalty term; the distance penalty term is the average of the anomaly scores of the training samples in the current round; the orthogonality penalty term is the average of the cosine similarity between each pair of semantic centers of each attention head in the current round.

[0087] With minimizing the total loss function as the training objective, the parameters of the semantic center and the multi-head self-attention module are updated through backpropagation. If the training convergence condition is not met, the parameters of the semantic center and the multi-head self-attention module are updated iteratively, and the Softmax temperature parameter is dynamically adjusted during the iteration process. If the training convergence condition is met, the trained semantic model is obtained.

[0088] like Figure 4 As shown, S5 includes:

[0089] The validation set includes a validation set sample set and validation set sample results; using the average anomaly score in the training set as the initial decision threshold, the validation set sample set is input into the trained semantic model to obtain the prediction results of the validation set sample set; based on the prediction results and the validation set sample results, the initial F1 score is calculated.

[0090] The initial decision threshold is adjusted according to a preset adjustment multiple range to obtain multiple candidate decision thresholds, and the F1 score corresponding to each decision threshold is calculated.

[0091] Compare the F1 scores corresponding to all decision thresholds, and take the decision threshold with the highest F1 score as the optimal decision threshold.

[0092] S6 includes:

[0093] The test set includes a test set sample set and test set sample results; the test set sample set is input into the trained semantic model to obtain the anomaly score of each test set sample in the test set sample set; if the anomaly score is greater than the optimal decision threshold, the text sequence data is determined to be abnormal, otherwise it is normal;

[0094] Based on the judgment results of the trained semantic model and the test set sample results, the evaluation indicators such as accuracy, recall, F1 score, confusion matrix, and inference speed are calculated and the calculation results are output.

[0095] like Figure 5 Optionally, the convolution module in S2 can also be a parallel multi-scale one-dimensional convolution structure: using three types of convolution kernels, namely 1×3, 1×5, and 1×7, to convolve the same embedding tensor in the sequence length direction. After batch normalization and activation function processing, each kernel is concatenated or weighted in the feature dimension to obtain a convolution output that fuses multi-scale local features. Then, the convolution output is added to the input embedding tensor by adding the residual to obtain the enhanced tensor sequence.

[0096] like Figure 6 As shown, optionally, the convolutional module and multi-head self-attention module can be replaced with a local-global dual-branch attention module. This local-global dual-branch attention module performs local and global self-attention in parallel on the same embedding tensor. In the local self-attention, the model is intentionally instructed to focus on the k tokens before and after each token while ignoring other tokens. In this length... By applying a self-attention mechanism to the sequence, we can obtain the local self-attention weight matrix of each attention head. Multiplying it by the original input yields the local context representation matrix. Global self-attention allows each token to perform attention operations with all tokens in the entire sequence to obtain a global context representation matrix. Subsequently, the local context representation matrix and the global context representation matrix are concatenated or weighted and fused along the feature dimension to generate a fused context representation matrix that contains both short-range and long-range semantic information, thus taking into account both short-range and long-range features.

[0097] Existing CVDD models map variable-length input sequences to fixed-length semantic representations using a multi-head self-attention mechanism and learn multiple semantic centers during training. This brings the high-dimensional representation of normal samples closer to the semantic centers, resulting in good detection performance. However, the attention mechanism of CVDD models is mainly used to capture global features, and its ability to capture local features is insufficient, potentially leading to inadequate representation of subtle local anomalies. This invention introduces a convolutional module into the CVDD model, cascaded with the self-attention module, so that the output representation simultaneously includes both global and local features. This enables more accurate differentiation between samples that are semantically similar overall but differ locally, improving anomaly detection capabilities. The use of 1D convolutions with smaller kernels efficiently extracts local information, improving model performance while controlling computational overhead, and facilitating engineering deployment and inference optimization.

[0098] like Figure 1 The device for improving the single-class anomaly detection capability of text sequences according to the present invention includes:

[0099] The data preprocessing module is used to acquire text sequence data, divide the text sequence data into training set, test set and validation set, wherein the training set contains only positive sample data, and the test set and validation set contain both positive sample data and negative sample data; and preprocess the text sequence data and convert it into embedded tensor sequences.

[0100] The convolution module is used to input the embedded tensor sequence into the convolution module, extract local features along the sequence length direction, and obtain the enhanced tensor sequence;

[0101] The self-attention module is used to input the enhanced tensor sequence into the multi-head self-attention module, extract local weighted features from each attention head, generate an initial context representation matrix, and fuse the initial context representation matrices of all attention heads to generate a fused context representation matrix.

[0102] The model training module is used to build a semantic model. It initializes a semantic center for each attention head, calculates the cosine distance between each initial context representation matrix and the corresponding semantic center, and obtains anomaly scores based on the cosine distance. In the training set, a total loss function is constructed based on the anomaly scores and orthogonal penalty terms. The training objective is to minimize the total loss function. The parameters of the semantic center and the parameters of the multi-head self-attention module are iteratively updated to obtain the trained semantic model.

[0103] The anomaly detection module is used to calculate the F1 score of the trained semantic model on the validation set, using the average anomaly score in the training set as the initial decision threshold. The decision threshold is adjusted to maximize the F1 score as the optimization objective, and the optimal decision threshold is obtained.

[0104] The results output module is used to test the trained semantic model on the test set using the optimal decision threshold, evaluate the performance of the trained semantic model, and output the evaluation results.< / unk> < / pad>

Claims

1. A method for improving the single-class anomaly detection capability of text sequences, characterized in that, include: S1. Obtain text sequence data, and divide the text sequence data into a training set, a test set, and a validation set, wherein the training set contains only positive sample data, and the test set and validation set contain both positive and negative sample data; preprocess the text sequence data and convert it into an embedded tensor sequence; S2. Input the embedded tensor sequence into the convolution module, extract local features along the sequence length direction to obtain the enhanced tensor sequence; S3. Input the enhanced tensor sequence into the multi-head self-attention module, extract local weighted features from each attention head, generate an initial context representation matrix, and fuse the initial context representation matrices of all attention heads to generate a fused context representation matrix. S4. Build a semantic model, initialize a semantic center for each attention head, calculate the cosine distance between each initial context representation matrix and the corresponding semantic center, and obtain the anomaly score based on the cosine distance; construct a total loss function based on the anomaly score and the orthogonal penalty term in the training set, take minimizing the total loss function as the training objective, iteratively update the parameters of the semantic center and the parameters of the multi-head self-attention module, and obtain the trained semantic model. S5. Using the average of the abnormal scores in the training set as the initial decision threshold, calculate the F1 score of the trained semantic model in the validation set, and adjust the decision threshold to maximize the F1 score as the optimization objective to obtain the optimal decision threshold. S6. Test the trained semantic model on the test set using the optimal decision threshold, evaluate the performance of the trained semantic model, and output the evaluation results.

2. The method for improving the single-class anomaly detection capability of text sequences according to claim 1, characterized in that, The preprocessing of text sequence data and its conversion into embedded tensor sequences includes: The text sequences in the training set, validation set, and test set are segmented into words to obtain corresponding word sequences, and a vocabulary is constructed based on the word sequences in the training set. Using the vocabulary, the word sequences in the training set, validation set, and test set are mapped into index sequences, and an embedding vector sequence is extracted from the index sequences using a pre-trained word embedding matrix. The embedding vector sequence is then padded with a preset length to obtain an embedding tensor sequence.

3. The method for improving the single-class anomaly detection capability of text sequences according to claim 1, characterized in that, S2 include: Perform a one-dimensional convolution on the embedded tensor sequence along the sequence direction to generate a local feature map; The local feature map is added element by element to the embedded tensor sequence to form a residual output; The residual output is layer normalized to obtain an enhanced tensor sequence with enhanced local features.

4. The method for improving the single-class anomaly detection capability of text sequences according to claim 1, characterized in that, S3 include: Within each attention head, the enhanced tensor sequence is linearly mapped to generate a query matrix, a key matrix, and a value matrix composed of a query vector, a key vector, and a value vector, respectively. The query matrix and key matrix are multiplied by a dot product, scaled, and normalized to obtain the attention weight matrix. The value matrix is ​​then weighted and summed using the attention weight matrix to obtain the initial context representation matrix of each attention head. The initial context representation matrices of each attention head are then concatenated to obtain the fused context representation matrix.

5. The method for improving the single-class anomaly detection capability of text sequences according to claim 1, characterized in that, S4 include: The semantic centers of each attention head are initialized using the K-means++ algorithm, and the parameters of the multi-head self-attention module are randomly initialized. Calculate the cosine distance between each initial context representation matrix and the semantic center of the corresponding attention head; The cosine distance is normalized using Softmax to obtain the context weight, and the cosine distance is weighted and summed using the context weight to obtain the anomaly score; A total loss function is constructed in the training set, which is the sum of a distance penalty term and an orthogonality penalty term; the distance penalty term is the average of the anomaly scores of the training samples in the current round; the orthogonality penalty term is the average of the cosine similarity between each pair of semantic centers of each attention head in the current round. With minimizing the total loss function as the training objective, the parameters of the semantic center and the multi-head self-attention module are updated through backpropagation. If the training convergence condition is not met, the parameters of the semantic center and the multi-head self-attention module are updated iteratively, and the Softmax temperature parameter is dynamically adjusted during the iteration process. If the training convergence condition is met, the trained semantic model is obtained.

6. The method for improving the single-class anomaly detection capability of text sequences according to claim 1, characterized in that, S5 include: The validation set includes a validation set sample set and validation set sample results; using the average anomaly score in the training set as the initial decision threshold, the validation set sample set is input into the trained semantic model to obtain the prediction results of the validation set sample set; based on the prediction results and the validation set sample results, the initial F1 score is calculated. The initial decision threshold is adjusted according to a preset adjustment multiple range to obtain multiple candidate decision thresholds, and the F1 score corresponding to each decision threshold is calculated. Compare the F1 scores corresponding to all decision thresholds, and take the decision threshold with the highest F1 score as the optimal decision threshold.

7. The method for improving the single-class anomaly detection capability of text sequences according to claim 1, characterized in that, S6 include: The test set includes a test set sample set and test set sample results; the test set sample set is input into the trained semantic model to obtain the anomaly score of each test set sample in the test set sample set; If the anomaly score is greater than the optimal decision threshold, the text sequence data is determined to be abnormal; otherwise, it is normal. Based on the judgment results of the trained semantic model and the test set sample results, the accuracy, recall, F1 score, and confusion matrix are calculated and the calculation results are output.

8. The method for improving the single-class anomaly detection capability of text sequences according to claim 1, characterized in that, The convolution module described in S2 is a parallel multi-scale one-dimensional convolution structure: it uses three types of convolution kernels, namely 1×3, 1×5, and 1×7, to convolve the same embedding tensor in the sequence length direction. After batch normalization and activation function processing, each kernel is concatenated or weighted in the feature dimension to obtain a convolution output that integrates multi-scale local features. Then, the convolution output is added to the input embedding tensor by adding the residual to obtain the enhanced tensor sequence.

9. The method for improving the single-class anomaly detection capability of text sequences according to claim 1, characterized in that, The convolutional module and the multi-head self-attention module are replaced by a local-global dual-branch attention module. The local-global dual-branch attention module performs local self-attention and global self-attention in parallel on the same embedding tensor. Local self-attention opens the attention window for each token to the k tokens before and after it to obtain the local context representation matrix. Global self-attention allows each token to perform attention operations with all tokens in the entire sequence to obtain the global context representation matrix. Then, the local context representation matrix and the global context representation matrix are concatenated or weighted and fused along the feature dimension to generate a fused context representation matrix that contains both short-range and long-range semantic information.

10. An apparatus for improving the single-class anomaly detection capability of text sequences, characterized in that, include: The data preprocessing module is used to acquire text sequence data, divide the text sequence data into training set, test set and validation set, wherein the training set contains only positive sample data, and the test set and validation set contain both positive sample data and negative sample data; and preprocess the text sequence data and convert it into embedded tensor sequences. The convolution module is used to input the embedded tensor sequence into the convolution module, extract local features along the sequence length direction, and obtain an enhanced tensor sequence; The self-attention module is used to input the enhanced tensor sequence into the multi-head self-attention module, extract local weighted features from each attention head, generate an initial context representation matrix, and fuse the initial context representation matrices of all attention heads to generate a fused context representation matrix. The model training module is used to build a semantic model. It initializes a semantic center for each attention head, calculates the cosine distance between each initial context representation matrix and the corresponding semantic center, and obtains anomaly scores based on the cosine distance. In the training set, a total loss function is constructed based on the anomaly scores and orthogonal penalty terms. The training objective is to minimize the total loss function. The parameters of the semantic center and the parameters of the multi-head self-attention module are iteratively updated to obtain the trained semantic model. The anomaly detection module is used to calculate the F1 score of the trained semantic model on the validation set, using the average anomaly score in the training set as the initial decision threshold. The decision threshold is adjusted to maximize the F1 score as the optimization objective, and the optimal decision threshold is obtained. The results output module is used to test the trained semantic model on the test set using the optimal decision threshold, evaluate the performance of the trained semantic model, and output the evaluation results.

Images