Telecommunication fraud text classification detection method based on RoBERTa intermediate layer neuron screening
By using a RoBERTa-based intermediate layer neuron selection method, combined with multi-head attention and residual connections, the robustness and classification performance of existing telecom fraud detection models are addressed. A diverse and representative telecom fraud dataset is constructed, achieving higher accuracy and stronger robustness in fraud text detection.
Patent Information
- Application Number
- CN202511495963.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-20
- Publication Date
- 2026-01-16
AI Technical Summary
Existing methods for detecting text related to telecom fraud suffer from insufficient robustness and unsatisfactory classification performance, especially on datasets with imbalanced class distributions. Furthermore, the coverage of existing datasets is insufficient to meet the practical needs of rapidly changing fraud methods.
We adopted a RoBERTa-based method for selecting intermediate layer neurons, combined with multi-head attention mechanism and residual connection, and used cross-entropy loss and inconsistency loss to construct a dataset containing 28,357 telecom fraud texts, covering eight types of telecom fraud texts. Through data preprocessing and key neuron feature selection, we improved the model's ability to recognize complex semantics and its anti-interference ability.
It significantly improves the accuracy and robustness of text classification for telecom fraud, better adapts to the needs of fraud detection in complex scenarios, and enhances the model's generalization ability and ability to identify new fraud patterns.
Smart Images
Figure CN121350827A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, specifically to a method for classifying and detecting telecommunications fraud text based on RoBERTa intermediate layer neurons. Background Technology
[0002] With the rapid development of internet and communication technologies, telecommunications fraud cases are frequent, seriously threatening the property safety of the public and social stability. Fraudsters often use multiple channels such as telephone, text message, instant messaging tools, and social media platforms to commit fraud, and their methods are constantly evolving, making it increasingly difficult for traditional detection methods to meet the needs of real-time detection.
[0003] In recent years, text classification methods based on pre-trained language models have gradually become a research hotspot in telecommunications fraud detection. These methods can automatically learn rich semantic and contextual features from large-scale corpora, which improves the detection performance of fraudulent texts to some extent. However, existing methods generally rely on the final layer representation of the pre-trained model for classification, which is susceptible to high-dimensional redundant information and noise, resulting in slow model convergence, unstable training, and poor performance on datasets with imbalanced class distributions.
[0004] Meanwhile, although existing fake base station datasets and five types of fraud datasets have promoted research on telecom fraud detection to some extent, these datasets still suffer from insufficient coverage and outdated updates, failing to meet the practical needs of rapidly changing fraud methods. As telecom fraud continues to evolve, new fraud patterns emerge constantly, making it difficult for existing detection models to effectively capture and identify these new patterns. Therefore, there is an urgent need for a more comprehensive and timely updated telecom fraud text dataset to support the development of more efficient and accurate fraud detection models. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of existing methods for detecting telecom fraud text, namely insufficient robustness and unsatisfactory classification performance. This invention proposes a telecom fraud text classification and detection method based on RoBERTa intermediate layer neuron selection. This method effectively improves the model's ability to recognize complex semantics by selecting key neuron features from the intermediate layers of the RoBERTa model and combining multi-head attention mechanisms with residual connections. Simultaneously, cross-entropy loss and inconsistency loss are introduced during training to alleviate class imbalance and enhance the model's anti-interference ability, thereby achieving higher accuracy and robustness in fraud text detection.
[0006] To enhance the model's generalization ability and adaptability to telecom fraud texts, this invention constructs a telecom fraud dataset containing 28,357 texts, covering eight types of telecom fraud texts. This dataset not only possesses diversity and representativeness but also features detailed annotation and classification of Chinese telecom fraud texts, aiming to help the model learn key features and subtle differences within these texts. Using this dataset for training, the proposed model significantly improves the accuracy of telecom fraud text classification, meeting the fraud detection needs in various complex scenarios.
[0007] This invention provides a method for classifying and detecting telecommunications fraud text based on RoBERTa intermediate layer neurons, comprising the following steps:
[0008] The data preprocessing of the telecom fraud text dataset includes cleaning irrelevant characters, word segmentation, and standardization. Sensitive fields such as names, bank card numbers, software, URLs, and contact information in the data are de-identified and cleaned. The text is also labeled to distinguish between normal text and fraudulent text. Finally, the processed dataset is divided into training and testing sets.
[0009] The input text is encoded using a pre-trained RoBERTa model to obtain the contextual semantic representation of the text. The hidden states or activation values of the intermediate layer are extracted in one forward propagation to obtain a high-dimensional intermediate layer representation.
[0010] The formulas for obtaining the hidden state and activation value of the intermediate layer are as follows:
[0011]
[0012] x l Let L represent the internal representation of the l-th layer, where L is the sentence length and D is the dimension of the internal representation.
[0013] The importance of each neuron is scored using an L1 regularized linear probe, and the neuron with the highest contribution is retained according to the weight threshold. The selected neurons are then sorted by importance and the dimensions are unified to ensure that the input of subsequent modules is consistent.
[0014] The formula for an L1 canonical linear probe is as follows:
[0015]
[0016] Where σ represents the sigmoid function, λ represents the regularization coefficient, N is the number of training samples, and W... l and b l These are the weights and biases of the l-th layer linear model.
[0017] The formula for determining the sparsity threshold η is as follows:
[0018]
[0019]
[0020] in, l,i Indicates the normalized weights. l * It is a subset of the selected key neurons.
[0021] The formula for a unified dimension is as follows:
[0022]
[0023] Where x represents the original neuron, w i Let x' be the importance weight of the i-th dimension, and let x' be the extracted neuron representation with a dimension of k×768, where k refers to the number of dimensions selected from the intermediate layer neurons.
[0024] After screening, key neurons are fed into a multi-head attention module to capture information from different semantic subspaces in the text in parallel and identify complex semantic features. The multi-head attention module employs twelve heads of attention, which can focus on semantic information at different locations, helping to improve the model's sensitivity to keywords or phrases in fraudulent texts.
[0025] The formula for the multi-head attention layer is as follows:
[0026]
[0027]
[0028] Where Q, K, and V represent query, keyword, and value, respectively, and W i Q W i K and W i V W represents the corresponding weight matrix. O Represents the output weight matrix, head i This represents the output of the i-th head.
[0029] After the attention output, the ReLU activation function is used to further extract nonlinear features, and the Dropout layer is used to prevent overfitting and promote gradient flow.
[0030] Further, residual connections are introduced to add the input and output, followed by Layer Normalization, which accelerates model convergence and improves training stability.
[0031] The formula for residual connection is as follows:
[0032]
[0033] Where, x l It is the input of the l-th layer, x l+1 It is the input of the (l+1)th layer. F(x) l W l ) is the residual function of the l-th layer.
[0034] The features, after residual connection and normalization, are input into the fully connected layer. Finally, the probability of each category is calculated through the Softmax layer, and the Adam optimizer is used to improve the classification effect, thereby completing the classification of fraudulent text and normal text.
[0035] Furthermore, the total loss function used during model training includes cross-entropy loss and inconsistency loss. Cross-entropy loss is used to improve classification accuracy, while inconsistency loss is used to enhance the model's resistance to interference and alleviate class imbalance, thereby further improving the model's accuracy and robustness.
[0036] The formula for the cross-entropy loss function is as follows:
[0037]
[0038] Where N is the number of samples, M is the number of categories, and y ij The true label of sample i is y'. ij It is the probability that sample i belongs to category j as predicted by the model.
[0039] The formula for the inconsistency loss function is as follows:
[0040]
[0041]
[0042]
[0043]
[0044] The inconsistency loss function consists of three parts: subspace inconsistency, location inconsistency, and representation inconsistency. Where L... subspace Represented as subspace inconsistency loss, L position Represented as location inconsistency loss, L representation This is represented as the inconsistency loss. V represents the attention value, H is the attention head, and A... h For the attention distribution of the h-th attention head, O h For the output of the h-th head, ⊙ represents element-wise multiplication, ||V i || represents the Vth digit iThe L2 norm of each is calculated. The inconsistency loss is calculated in the multi-head attention layer and then combined with the cross-entropy loss.
[0045] The overall loss function of the entire model is shown below:
[0046]
[0047]
[0048] Where L CE For cross-entropy loss, L subspace For subspace inconsistency loss, L position For location inconsistency loss, L representation To represent the inconsistency loss, α, β, and γ are the corresponding weighting coefficients, and their values were determined to be 0.1, 0.1, and 0.8 through experiments.
[0049] To further enhance the model's generalization ability and adaptability to telecom fraud texts, this invention has constructed a telecom fraud dataset containing 28,357 texts, covering eight different types of telecom fraud texts. The construction steps are as follows: Seven typical fraud texts were collected and extracted from the CCL2023 dataset, including texts on impersonating public security, procuratorate, and court officials, loan fraud, impersonating customer service, impersonating acquaintances, online shopping fraud with rebates, dating fraud, and fraudulent online game transactions; simultaneously, non-fraudulent texts were collected and constructed for comparison with the fraudulent texts.
[0050] The original text is anonymized and cleaned to remove or obfuscate sensitive fields such as names, contact information, software names, and bank card numbers, ensuring user privacy and data compliance.
[0051] The cleaned data is input into a pre-trained large language model, and fraudulent text that closely resembles real-world scenarios is generated using specific prompt words.
[0052] The generated text undergoes manual annotation and quality review to ensure accurate classification; non-fraudulent text is generated by manually collecting real communication content and combining it with text generated by a large model to ensure the diversity and naturalness of the data.
[0053] The final result is a dataset of 28,357 Chinese fraud texts in eight categories. The category distribution closely resembles real-world scenarios, and the samples are abundant and have broad coverage. Attached Figure Description
[0054] The following figures provide a detailed description of the invention:
[0055] Figure 1 This is a framework diagram of the fraud text classification model of the present invention;
[0056] Figure 2 This is a diagram illustrating the loss function framework of the present invention;
[0057] Figure 3 A dataset framework diagram is constructed for this invention; Detailed Implementation
[0058] In order to enable those skilled in the art to better understand the present invention and its objectives, the present invention will be further described in detail below with reference to the accompanying drawings of the embodiments.
[0059] With the continued high incidence of telecommunications fraud, accurately identifying fraudulent texts has become a crucial step in safeguarding public property and information security. The widespread use of mobile internet and social media allows fraudsters to instantly send out customized deceptive messages via SMS, WeChat, email, and other diverse channels. Their methods are constantly evolving and their tactics are varied, making it difficult for traditional rules and keyword databases to keep up, resulting in a high rate of false positives and false negatives. To address this challenge, this invention combines deep learning and natural language processing technologies, proposing a fraud text classification model centered on RoBERTa intermediate layer neurons. RoBERTa's pre-training mechanism can fully capture contextual semantics and latent contextual features. Compared to traditional methods, it not only improves the accuracy of identifying complex fraudulent messages but also significantly enhances the generalization ability against new types of scams, providing reliable technical support for real-time and accurate fraud detection.
[0060] This invention provides a method for classifying and detecting telecommunications fraud text based on RoBERTa intermediate layer neurons, comprising the following steps:
[0061] The data preprocessing of the telecom fraud text dataset includes cleaning irrelevant characters, word segmentation, and standardization. Sensitive fields such as names, bank card numbers, software, URLs, and contact information in the data are de-identified and cleaned. The text is also labeled to distinguish between normal text and fraudulent text. Finally, the processed dataset is divided into training and testing sets.
[0062] The input text is encoded using a pre-trained RoBERTa model to obtain the contextual semantic representation of the text. The hidden states or activation values of the intermediate layer are extracted in one forward propagation to obtain a high-dimensional intermediate layer representation.
[0063] The formulas for obtaining the hidden state and activation value of the intermediate layer are as follows:
[0064]
[0065] x l Let L represent the internal representation of the l-th layer, where L is the sentence length and D is the dimension of the internal representation.
[0066] The importance of each neuron is scored using an L1 regularized linear probe, and the neuron with the highest contribution is retained according to the weight threshold. The selected neurons are then sorted by importance and the dimensions are unified to ensure that the input of subsequent modules is consistent.
[0067] The formula for an L1 canonical linear probe is as follows:
[0068]
[0069] Where σ represents the sigmoid function, λ represents the regularization coefficient, N is the number of training samples, and W... l and b l These are the weights and biases of the l-th layer linear model.
[0070] The formula for determining the sparsity threshold η is as follows:
[0071]
[0072]
[0073] in, l,i Indicates the normalized weights. l * It is a subset of the selected key neurons.
[0074] The formula for a unified dimension is as follows:
[0075]
[0076] Where x represents the original neuron, w i Let x be the importance weight of the i-th dimension. ’ The extracted neuron representation has a dimension of k×768, where k refers to the number of dimensions selected from the intermediate layer neurons.
[0077] After screening, key neurons are fed into a multi-head attention module to capture information from different semantic subspaces in the text in parallel and identify complex semantic features. The multi-head attention module employs twelve heads of attention, which can focus on semantic information at different locations, helping to improve the model's sensitivity to keywords or phrases in fraudulent texts.
[0078] The formula for the multi-head attention layer is as follows:
[0079]
[0080]
[0081] Where Q, K, and V represent query, keyword, and value, respectively, and W i QW i K and W i V W represents the corresponding weight matrix. O Represents the output weight matrix, head i This represents the output of the i-th head.
[0082] After the attention output, the ReLU activation function is used to further extract nonlinear features, and the Dropout layer is used to prevent overfitting and promote gradient flow.
[0083] Further, residual connections are introduced to add the input and output, followed by Layer Normalization, which accelerates model convergence and improves training stability.
[0084] The formula for residual connection is as follows:
[0085]
[0086] Where, x l It is the input of the l-th layer, x l+1 It is the input of the (l+1)th layer. F(x) l, W l ) is the residual function of the l-th layer.
[0087] The features, after residual connection and normalization, are input into the fully connected layer. Finally, the probability of each category is calculated through the Softmax layer, and the Adam optimizer is used to improve the classification effect, thereby completing the classification of fraudulent text and normal text.
[0088] Furthermore, the total loss function used during model training includes cross-entropy loss and inconsistency loss. Cross-entropy loss is used to improve classification accuracy, while inconsistency loss is used to enhance the model's resistance to interference and alleviate class imbalance, thereby further improving the model's accuracy and robustness.
[0089] The formula for the cross-entropy loss function is as follows:
[0090]
[0091] Where N is the number of samples, M is the number of categories, and y ij The true label of sample i is y'. ij It is the probability that sample i belongs to category j as predicted by the model.
[0092] The formula for the inconsistency loss function is as follows:
[0093]
[0094]
[0095]
[0096]
[0097] The inconsistency loss function consists of three parts: subspace inconsistency, location inconsistency, and representation inconsistency. Where L... subspace Represented as subspace inconsistency loss, L position Represented as location inconsistency loss, L representation This is represented as the inconsistency loss. V represents the attention value, H is the attention head, and A... h For the attention distribution of the h-th attention head, O h For the output of the h-th head, ⊙ represents element-wise multiplication, ||V i || represents the Vth digit i The L2 norm of each is calculated. The inconsistency loss is calculated in the multi-head attention layer and then combined with the cross-entropy loss.
[0098] The overall loss function of the entire model is shown below:
[0099]
[0100]
[0101] Where L CE For cross-entropy loss, L subspace For subspace inconsistency loss, L position For location inconsistency loss, L representation To represent the inconsistency loss, α, β, and γ are the corresponding weighting coefficients, and their values were determined to be 0.1, 0.1, and 0.8 through experiments.
[0102] To further enhance the model's generalization ability and adaptability to telecom fraud texts, this invention has constructed a telecom fraud dataset of approximately 28,357 texts, covering eight different types of telecom fraud texts. The construction steps are as follows:
[0103] Seven types of typical fraud texts were collected and extracted from the CCL2023 dataset, including texts on impersonating public security, procuratorate and court officials, loan fraud, impersonating customer service, impersonating acquaintances, online shopping fraud, dating fraud, and fraudulent online game transactions. At the same time, non-fraud texts were collected and constructed for comparison with the fraud texts.
[0104] The original text is anonymized and cleaned to remove or obfuscate sensitive fields such as names, contact information, software names, and bank card numbers, ensuring user privacy and data compliance.
[0105] The cleaned data is input into a pre-trained large language model, and fraudulent text that closely resembles real-world scenarios is generated using specific prompt words.
[0106] The generated text undergoes manual annotation and quality review to ensure accurate classification; non-fraudulent text is generated by manually collecting real communication content and combining it with text generated by a large model to ensure the diversity and naturalness of the data.
[0107] The final result is a dataset of 28,357 Chinese fraud texts categorized into eight classes. The class distribution closely resembles real-world scenarios, and the sample is rich and comprehensive. See Table 1 for details.
[0108] Table 1 is a self-built dataset.
[0109] To further evaluate the impact of the extracted intermediate neuron parameters on model performance and explore the role of each module, experiments were conducted on our self-built dataset, where the training set and test set accounted for 80% and 20% of the total, respectively. The specific experimental setup is as follows:
[0110] 1. Baseline Model Group: Using pre-trained RoBERTa models without adding any additional modules, as experimental benchmarks.
[0111] 2. Internal Neuron Group: This group only incorporates internal neuron mechanisms to assess their independent contribution to model performance.
[0112] 3. Combination of internal neurons and k-values: Based on the internal neuron mechanism, modules such as multi-head attention were added, and the k-value was changed to analyze its specific impact on model performance. 3.1. k=5, 3.2. k=10, 3.3. k=2.5.
[0113] 4. Combinations of Inconsistency Parameters: Since k=5 yields the best results, the impact of different inconsistency parameters on model performance was evaluated based on this configuration. 4.1 Inconsistency Loss (α=0.1, β=0.1, γ=0.8), 4.2 Inconsistency Loss (α=0, β=0, γ=1), 4.3 Inconsistency Loss (α=0.3, β=0.1, γ=0.6)
[0114] The experimental results are shown in Table 2:
[0115] Table 2 shows the performance results of various models on the eight-class fraud dataset.
[0116] To further evaluate the model's performance comprehensively, we also conducted experiments on the public dataset THUCNews, and the results are shown in Table 3:
[0117] Table 3 shows the performance results of various models on the THUCNews dataset.
[0118] In summary, the technical advantages of this invention are reflected in the following aspects:
[0119] First, we perform text classification based on information from RoBERTa intermediate layer neurons. We use linear detection and sparse ensemble strategies to identify neurons crucial to the classification task, while reducing noise from irrelevant neurons to improve accuracy and efficiency.
[0120] Second, during training, the inconsistency loss function is combined with the cross-entropy loss function. The cross-entropy loss is used to improve classification accuracy, while the inconsistency loss is used to enhance the model's ability to resist interference and alleviate the class imbalance problem, thereby further improving the model's accuracy and robustness.
[0121] Third, this invention constructs an eight-category telecommunications fraud dataset containing 28,357 data points. The category distribution closely resembles real-world scenarios, with abundant and wide-ranging samples. The dataset undergoes manual annotation and quality review to ensure classification accuracy. This dataset can be used to support model training and evaluation, further improving the model's generalization ability and recognition performance.
Claims
1. A telecommunications fraud text classification detection method based on RoBERTa intermediate layer neuron screening, characterized in that, Comprise the following steps: S101: data preprocessing is performed on the telecom fraud text dataset, and the processed dataset is divided into a training set and a test set. S102: The pre-trained RoBERTa model is used to encode the input text, capture the context semantic representation of the text, and obtain the hidden state and activation value of each intermediate layer. S103: Use linear probing combined with L1 regularization and sparsification strategy to retain key neurons and unify dimensions, ensure consistent input for subsequent modules. S104: The screened neurons are sent to the multi-head attention module, which captures information in different semantic subspaces in the text in parallel, and identifies complex semantic features. S105: After attention output, add Dropout layer, ReLU activation layer and residual connection layer in turn to prevent overfitting and promote gradient flow, and finally input the normalized features into the fully connected layer to calculate the probability of each class through the Softmax layer. S106: Cross-entropy loss function and inconsistency loss function are used in the training process, and Adam optimizer is used to improve the classification effect.
2. The method of claim 1, wherein the method is a method of detecting telecom fraud text classification based on screening of intermediate layers of RoBERTa neurons. In step S101, the telecom fraud text dataset is preprocessed, including cleaning irrelevant characters, word segmentation and standardization operations, and labeling the text with labels to distinguish between normal text and fraud text. The encoding method of the RoBERTa model in step S102 is as follows:
3. The method of claim 1, wherein the method is characterized by: In step S103, the screening method of neurons is shown in the following formula: x l represents the internal representation of the l-th layer, L is the sentence length, and D is the dimension of the internal representation.
4. The method of claim 1, wherein the method is characterized by: In step S104, the model uses a multi-head attention mechanism, as shown in the following formula: where σ denotes sigmoid function, λ denotes regularization coefficient, N is the number of training samples, W l and b l are the weights and bias of the lth linear model. wherein, l,i denotes the normalized weight, l * is a subset of selected key neurons. where x is the original neuron representation, w i is the importance weight of the i-th dimension, x' is the extracted neuron representation with dimension k x 768, and k refers to the number of dimensions selected from the intermediate layer neurons.
5. The method of claim 1, wherein the method is characterized by: The residual connection in step S105 is defined as follows: where Q, K and V represent query, key and value respectively, W i Q , W i K and W i V represent the corresponding weight matrix respectively, W O represents the output weight matrix, head i represents the output of the i-th head.
6. The method of claim 1, wherein the method is a method of detecting telecom fraud text classification based on screening of intermediate layers of RoBERTa, characterized in that, In step S106, the total loss function of the model consists of cross-entropy loss and inconsistency loss, where cross-entropy loss is used to improve classification accuracy, and inconsistency loss is used to enhance the anti-interference ability of the model and alleviate the class imbalance problem, thereby further improving the accuracy and robustness of the model; The weighting method of the total loss is as follows: where x l is the input of the lth layer, x l+1 is the input of the l+1th layer. F(x l , W l ) is the residual function of the lth layer.
7. The method of claim 1, wherein the method is characterized by: The calculation method of the cross-entropy loss function is shown in the following formula: where L CE is the cross-entropy loss, L subspace is the subspace inconsistency loss, L position is the position inconsistency loss, L representation is the representation inconsistency loss, and a, b, g are the corresponding weight coefficients with values of 0.1, 0.1, 0.
8.
8. The method of claim 7, wherein the method is based on a RoBERTa intermediate layer neuron screening of telecom fraud text classification detection. The inconsistency loss function is shown in the following formula: N is the number of samples, M is the number of classes, y ij is the true label of sample i, y' ij is the probability that the model predicts that sample i belongs to class j.
9. The method of claim 7, wherein the method is characterized by, The following construction steps are included: The inconsistency loss function is composed of three parts: subspace inconsistency, position inconsistency and representation inconsistency. Among them, L subspace represents the subspace inconsistency loss, L position represents the position inconsistency loss, L representation represents the representation inconsistency loss, V represents the value of attention, H is an attention head, A h is the attention distribution of the hth attention head, O h is the output of the hth head, ⊙ represents element-wise multiplication, ||V i || represents the L2 norm of the V i th.
10. A classification dataset based on telecom fraud text, characterized in that, S201: Collect and extract seven types of typical fraud text from CCL2023 dataset, including impersonation of public security organs, loan fraud, impersonation of customer service, impersonation of acquaintances, single return fraud, marriage and dating fraud, and network game product false transaction fraud text; At the same time, collect and build non-fraud text for classification comparison with fraud text. S202: Desensitize and clean the original text, remove or blur sensitive fields such as names, contact information, software names, and bank card numbers to ensure user privacy and data compliance. S203: Input the cleaned data into the pre-trained large language model, and generate fraud text that is highly similar to the real scene through specific prompt words. S204: Manually label and quality audit the generated text to ensure classification accuracy; non-fraud text is collected by manually collecting real communication content and combining with text generated by the large model to ensure the diversity and naturalness of the data. S205: Finally, a Chinese fraud text dataset containing 28357 eight categories is obtained, with a distribution close to the real scene, rich samples and wide coverage.