A multi-label sensitive text detection method integrating BERT and Naive Bayes

Abstract

This invention discloses a multi-label sensitive text detection method integrating BERT and Naive Bayes ensemble models, belonging to the field of data security detection technology. The method includes: extracting deep semantic features from input text using a pre-trained BERT model to understand subtle language patterns of sensitive content and calculating the prediction probability of the BERT model; constructing a multi-label Naive Bayes ensemble model and calculating its prediction probability based on lexical statistical rules; introducing consistency regularization loss during the training phase to force the prediction probabilities of the BERT model and the Naive Bayes ensemble model to align for each label; and jointly deploying the two models during the inference phase and performing probabilistic fusion prediction to obtain the multi-label sensitive text detection result. This invention enhances robustness against adversarial attacks such as code switching and effectively improves the accuracy of sensitive text detection in multilingual scenarios.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and data security detection technology, and in particular to a multi-label sensitive text detection method that integrates BERT and Naive Bayes. Background Technology

[0002] In the wave of digitalization, data has become a core element driving innovation and development across industries. Data security threats, however, are ever-present, exhibiting an increasingly diverse and complex nature. Natural Language Processing (NLP), a key branch of artificial intelligence, is deeply integrated into all aspects of data security detection, leveraging its powerful capabilities in text understanding, semantic analysis, and information extraction. Particularly in the niche area of ​​sensitive text detection, accurately and efficiently identifying multi-labeled text containing specific sensitive content has become a critical technical challenge for ensuring data security and purifying cyberspace.

[0003] Numerous studies have attempted to apply deep learning models to sensitive text detection tasks. For example, classification models are constructed using typical recurrent neural network architectures such as Long Short-Term Memory (LSTM). These models typically consist of an input layer, word embedding layers, LSTM layers, and a dense output layer, identifying sensitive information by capturing sequence dependencies in the text. While these methods have achieved some success in specific scenarios or single-language environments, they still suffer from several inherent limitations in large-scale, multilingual, and open dynamic application environments. These limitations include: significantly reduced generalization ability and detection performance in multilingual mixed scenarios; insufficient robustness against code-switching attacks; and particularly poor detection performance in resource-constrained, low-resource languages.

[0004] The existence of these problems greatly limits the practical application scope and effectiveness of sensitive text detection technology in a globalized, multilingual network environment, and there is an urgent need for a new monitoring method that can effectively improve the accuracy of sensitive text detection in multilingual scenarios. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a multi-label sensitive text detection method that integrates BERT and Naive Bayes, which can enhance robustness against adversarial attacks such as code switching and significantly improve detection performance in low-resource languages.

[0006] This invention adopts the following technical solution: a multi-label sensitive text detection method integrating BERT and Naive Bayes, comprising the following steps:

[0007] Step 1: Extract deep semantic features from the input text using a pre-trained BERT model, perform BERT-based sensitive text classification, understand the subtle language patterns of sensitive content, and calculate the prediction probability of the BERT model.

[0008] Step 2: Construct a Naive Bayes ensemble model for multi-label learning, and calculate the prediction probability of the Naive Bayes ensemble model based on lexical statistical rules.

[0009] Step 3: During the training phase, introduce consistency regularization loss, using the obtained Naive Bayes ensemble prediction vector as the external signal, to force the prediction probabilities of the BERT model and the Naive Bayes ensemble model to align with each label.

[0010] Step 4: In the inference stage, the BERT model and the Naive Bayes ensemble model are jointly deployed and fused for prediction in a probabilistic manner to obtain multi-label sensitive text detection results.

[0011] Preferably, in step 1, the prediction probability of the BERT model is calculated as follows:

[0012] Step 1.1: For a given input text sequence The maximum length is ,pass The function will assign each tag Mapping to context hidden vectors to generate hidden states ;

[0013] Step 1.2, Using special classification tokens The corresponding hidden state is used as a fixed-length representation of the entire sequence, and the hidden vector corresponding to the token is extracted. ;

[0014] Step 1.3: Project the aggregated vectors through a linear classification layer to a... dimensional output space middle;

[0015] Step 1.4: Perform multi-label classification and apply the Sigmoid activation function to each element. This yields the probability output vector of the BERT model. .

[0016] Preferably, the BERT model is trained using the standard binary cross-entropy loss averaged across all labels, and label imbalance is addressed by label-based loss weights.

[0017] Preferably, in step 2, the Naive Bayes ensemble model is derived from... A set of independent multinomial Naive Bayes classifiers, each classifier corresponding to One of the tags, the specific processing is as follows:

[0018] Step 2.1: Given a sequence of input text Extract its TF-IDF representation and calculate it as a word. exist TF-IDF score ;

[0019] Step 2.2, for each tag Posterior probabilities are estimated using a separate MNB classifier via Bayesian rules. ;

[0020] Step 2.3: Calculate the logarithmic posterior to maintain numerical stability;

[0021] Step 2.4, through Representing words In The number of times it appears in the labeled samples, using Smooth training data, estimate ;

[0022] Step 2.5: After calculating the original log probability for each class, normalize it using the Sigmoid function to convert it into a pseudo-probability. Then, use a Naive Bayes classifier to process the input text. After processing, the resulting text belongs to the first... Predicted probability of each label ;

[0023] Step 2.6: Calculate the final Naive Bayes ensemble prediction vector. The Naive Bayes ensemble prediction vector It is not updated during backpropagation and is used as an external signal to regularize the BERT model through consistency loss, providing robustness to poorly performing noise or short inputs in the BERT model.

[0024] Preferably, in step 3, the consistency regularization loss is defined using mean squared error. For samples where the inconsistency between the BERT model and the Naive Bayes ensemble model exceeds a preset threshold, the consistency regularization loss is calculated using a gating function. The overall training objective is calculated by combining the supervised classification loss and consistency loss of BERT, using non-negative hyperparameters. The relative importance of control consistency regularization.

[0025] Preferably, in the inference process described in step 4, the prediction vector is calculated through convex combination, the contribution of each model is balanced by fusion weight hyperparameters, a fixed threshold is applied to each label dimension, the soft probability is converted into a binary label decision for multi-label classification, and when the label distribution is highly unbalanced, the label-specific threshold is adjusted on the validation set.

[0026] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0027] 1. The multi-label sensitive text detection method of this invention constructs a hybrid modeling framework that combines converter-based contextual representation learning with probabilistic lexical modeling, and integrates the BERT encoder with the parallel Naive Bayes ensemble model. By jointly utilizing global semantic understanding and local statistical cues, it provides an effective solution for multi-label sensitive text classification.

[0028] 2. The multi-label sensitive text detection method of this invention develops a two-level integration mechanism, which forces the prediction-level consistency between deep and shallow classifiers during training, unifies their outputs through adaptive probability fusion during inference, improves generalization under label imbalance through cross-model alignment, and enhances prediction reliability under noisy or fuzzy inputs. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the overall process of the label-sensitive text detection method of the present invention.

[0030] Figure 2 This is a flowchart of the BERT model of the present invention;

[0031] Figure 3 This is a flowchart of the Naive Bayes ensemble model of the present invention;

[0032] Figure 4 This is a flowchart of the training process of the present invention;

[0033] Figure 5 This is a flowchart of the reasoning and fusion process of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0035] In one embodiment of the present invention, a hybrid modeling framework combining a deep model (BERT model) and a shallow model (Naive Bayes ensemble model) is constructed. A two-level mechanism of regularization during training and fusion during inference is adopted to allow the two heterogeneous models to complement each other's advantages, taking into account both the model's generalization ability and prediction reliability.

[0036] This embodiment integrates a multi-label sensitive text detection method that combines BERT and Naive Bayes, such as... Figure 1 As shown, it includes the following steps:

[0037] Step 1: Construct a sensitive text classification module based on BERT.

[0038] By capturing rich syntactic and semantic features in the input text through a pre-trained BERT (Bidirectional Encoder Representation) module, we can understand the subtle language patterns of sensitive content and solve the problem of deep semantic understanding.

[0039] In this embodiment, the BERT model processing flow is as follows: Figure 2 As shown, for a given input text sequence To truncate or supplement the text, the maximum length is... BERT will assign each tag Mapped to contextualized hidden vectors, generating:

[0040] ;

[0041] in, It is the hidden dimension of BERT.

[0042] Use with special classification tokens The corresponding hidden state serves as a fixed-length representation of the entire sequence. It is the hidden vector corresponding to the extracted token. It is a special classification token generated by the BERT model after encoding the input text. The corresponding hidden state vector is represented as:

[0043] .

[0044] Then, this aggregated vector is passed through a linear classification layer and projected onto a... In the output space of dimension:

[0045] ;

[0046] in, It is the number of tags. It is the weight matrix of the linear classification layer, used to represent the hidden states extracted by the BERT model. Projected onto the label space. It is the bias vector of the linear classification layer, used to adjust the result of the linear transformation.

[0047] To perform multi-label classification, the Sigmoid activation function is applied to each element. :

[0048] ;

[0049] in, Through linear classification layer The calculated result vector The various elements in It is the probability output vector of the BERT model.

[0050] For each , This indicates that for a given input, the label The probability of activation is determined by training the model using the standard binary cross-entropy (BCE) loss, which is the average of all labels.

[0051] ;

[0052] This loss treats each label independently, making it well-suited for multi-label learning tasks.

[0053] Optionally, this embodiment introduces label-based loss weights to address label imbalance:

[0054] ;

[0055] in, These are label-specific weighting coefficients, resulting in... This is the weighted loss function result.

[0056] Overall, this embodiment captures the rich syntactic and semantic features necessary for understanding sensitive and nuanced language patterns using the BERT model.

[0057] Step 2: Construct a Naive Bayes ensemble model for multi-label learning.

[0058] To complement deep learning models with interpretable and efficient lexical modeling, this embodiment constructs an ensemble of C independent Multinomial Naive Bayes (MNB) classifiers, each corresponding to one of the C labels. The processing flow of the Naive Bayes ensemble model is as follows: Figure 3 As shown.

[0059] Specifically, given an input text sequence First, extract its TF-IDF (Term Frequency - Inverse Document Frequency) representation:

[0060] ;

[0061] in, For vocabulary, For words exist TF-IDF score.

[0062] Specifically, the score is calculated as follows:

[0063] ;

[0064] in, (Word frequency) is the frequency of words. In the text The number of times it appears in the text divided by the number of times it appears in the text Total word count; (Inverse document frequency) is in logarithmic form (total number of documents in the corpus divided by the number of words contained in the document). (Number of documents).

[0065] For each tag A standalone MNB classifier estimates the posterior probability using Bayesian rules:

[0066] ;

[0067] in, In the text Under the conditions, it belongs to the label The posterior probability; In the label Under these conditions, text The likelihood probability of occurrence; It is a tag The prior probability, i.e., the label The probability of it appearing in the entire corpus.

[0068] In practice, use Indicates label Under the conditions To determine the probability of occurrence and ensure numerical stability, the logarithmic posterior needs to be calculated.

[0069] .

[0070] Here, using Smooth the training data (the training data consists of a large number of labeled text samples, each sample containing a text segment and one or more corresponding labels), and calculate... :

[0071] ;

[0072] in, Representing words In The number of times it appears in the labeled samples. It is a smoothing factor (usually set to 1).

[0073] After calculating the raw log probabilities for each class, they are further normalized using the Sigmoid function to convert them into pseudoprobabilities:

[0074] ;

[0075] in, The input text is processed using a Naive Bayes classifier. After processing, the resulting text belongs to the first... The predicted probability of each label (pseudo-probability normalized by the Sigmoid function).

[0076] Final Naive Bayes ensemble prediction vector yes:

[0077] .

[0078] It is worth noting that the NB component (which refers to the predictions of the Naive Bayes ensemble) is not updated during backpropagation; instead, it acts as an external signal that regularizes the BERT model through consistency loss, thereby providing robustness against noisy or short inputs where BERT may perform poorly.

[0079] Step 3: In the training phase, introduce consistency loss regularization.

[0080] To encourage consistent behavior between deep BERT models and shallow Naive Bayes ensembles, this implementation introduces a consistency regularization loss. This loss forces the predicted probabilities of the two models to align for each label. The training process is as follows: Figure 4 As shown.

[0081] set up This represents the probability output vector of a BERT-based classifier. For the corresponding output of the Naive Bayes set, the consistency loss is defined as the mean squared error between the two:

[0082] ;

[0083] in, For consistency loss, and for the sake of compactness, it can also be written in vector form:

[0084] ;

[0085] Intuitively, this loss term penalizes the difference between the two distributions on the labels predicted by BERT and the Naive Bayes component. Unlike the binary cross-entropy loss, which measures the distance to the ground truth label, the consistency loss encourages consistency between models, which helps improve generalization, especially when the Naive Bayes model captures complementary lexical patterns.

[0086] The overall training objective combines BERT's supervised classification loss and consistency loss, as follows:

[0087] ;

[0088] here, It is a non-negative hyperparameter that controls the relative importance of consistency regularization. A higher [value] indicates better consistency. Encourage a closer integration of BERT models with Naive Bayes predictions, while lower... The main focus is on learning basic truth labels.

[0089] Optionally, consistency loss can be applied selectively. For example, for uncertain samples with high model inconsistency, consistency loss can be emphasized, in which case a gating function can be used. To achieve:

[0090] ;

[0091] Such gating mechanisms can help avoid forcing an agreement when a pattern is clearly underperforming.

[0092] Step 4: In the inference phase, use a fusion strategy to jointly deploy the BERT model and the Naive Bayes ensemble model.

[0093] After training, this embodiment further deploys the BERT model and the Naive Bayes model jointly, and fuses their predictions in a probabilistic manner. The inference fusion process is as follows: Figure 5 As shown.

[0094] Suppose that for a given input , and These are the prediction probability vectors from BERT and Naive Bayes, respectively. The final prediction vector is calculated through a convex combination:

[0095] ;

[0096] in, and These are the predicted probability vectors from the BERT model and the Naive Bayes ensemble model, respectively. It is a fusion weight hyperparameter that balances the contribution of each model.

[0097] if If the prediction comes purely from BERT; if It uses only Naive Bayes; intermediate values ​​allow the ensemble to benefit from both models.

[0098] To transform these soft probabilities into binary label decisions for multi-label classification, a fixed threshold is applied to each label dimension:

[0099] ;

[0100] Alternatively, you can adjust the label-specific threshold on the validation set. :

[0101] ;

[0102] in, It is a soft probability obtained from convex combination calculation. It is a binary label decision; this adaptive threshold may be beneficial when the label distribution is highly unbalanced.

[0103] Therefore, the fusion strategy in this embodiment effectively combines the contextual semantic capabilities of BERT with the statistical reliability of Naive Bayes, making the final model more robust to lexical variations and rare label patterns.

[0104] Furthermore, to verify the effectiveness of the detection method of the present invention, classification tasks were performed on multiple datasets (openmalla-dataset, CommentToxicityOnGoogle, CommentToxicityOnJigsaw). Compared with existing methods such as SVM, RandomForest, XGBoost, CNN, and GCN, the method of the present invention showed outstanding performance in accuracy (ACC). For example, the ACC reached 0.9708 on the openmalla-dataset, and 0.9161 and 0.9183 on CommentToxicityOnGoogle and CommentToxicityOnJigsaw, respectively, demonstrating high classification accuracy.

[0105] Ablation experiments show that different combinations of modules such as BERT (BERT's sensitive text classification module), Beiyesi (Naive Bayes ensemble module for multi-label learning), and Loss (consistency loss regularization) in this invention affect performance.

[0106] When all three modules (M5) are present, on the CommentToxicityOnGoogle dataset, the ACC is 0.9161, the PUR (a metric for evaluating the quality of classification results) is 0.6132, and the F1 score is 0.6637. Compared to other module combinations, this shows the best performance, indicating that the fusion of these modules can effectively improve classification performance.

[0107] Then, the parameters of the fusion probability prediction module. Analysis revealed that when When the parameter value is 0.7, the ACC reaches 0.9161, the PUR is 0.6132, and the F1 score is 0.6637 on the CommentToxicityOnGoogle dataset, which is the best performance among all parameter values, indicating that appropriate parameter settings can further improve the performance of the method of the present invention.

[0108] In summary, the core of this invention is to construct a hybrid modeling framework that combines deep models (BERT) and shallow models (Naive Bayes ensemble). Through a two-level collaborative mechanism of regularization during training and fusion during inference, the two heterogeneous models complement each other.

[0109] Deep models are responsible for capturing rich syntactic and semantic features in the input text, understanding subtle language patterns in sensitive content, and solving deep semantic understanding problems. Shallow models, based on lexical statistical patterns, provide highly interpretable and computationally efficient prediction results, compensating for the lack of robustness of deep learning models in noisy input and short text scenarios. This two-level collaborative mechanism... machine During the training phase, consistency loss is used to force the two models to align their predictions, and during the inference phase, probability fusion is used to unify the output, balancing the model's generalization ability and prediction reliability. This effectively enhances the robustness against adversarial attacks such as code switching and significantly improves detection performance in low-resource languages.

[0110] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-label sensitive text detection method integrating BERT and Naive Bayes ensemble, characterized in that, Includes the following steps: Step 1: Extract deep semantic features from the input text using a pre-trained BERT model, perform BERT-based sensitive text classification, understand the subtle language patterns of sensitive content, and calculate the prediction probability of the BERT model. The BERT model was trained using the standard binary cross-entropy loss averaged across all labels: ； in, It is the number of tags. This indicates that for a given input label The probability of being activated. It is a tag The true value, It is the result of the loss function calculation; Label imbalance is addressed using label-based loss weights, as shown in the following formula: ； in, These are label-specific weighting coefficients, resulting in... This is the weighted result of the loss function; Step 2: Construct a Naive Bayes ensemble model for multi-label learning, and calculate the prediction probability of the Naive Bayes ensemble model based on lexical statistical rules. The Naive Bayes integration model is composed of A set of independent multinomial Naive Bayes classifiers, each classifier corresponding to One of the tags, the specific processing is as follows: Step 2.1: Given a sequence of input text Extract its TF-IDF representation: ； in, For vocabulary, For words exist TF-IDF score; Step 2.2, for each tag The posterior probability is estimated using a separate MNB classifier via Bayesian rules: ； in, Indicated in text Under the condition of belonging to label The posterior probability, Indicates in the label Under the condition of text The likelihood probability of occurrence Indicates label The prior probability; Step 2.3: Calculate the logarithmic posterior to maintain numerical stability. ； in, Indicates label Under the conditions The probability of occurrence; Step 2.4, Use Smooth training data, estimate : ； in, , Representing words and In The number of times it appears in the labeled samples. It is a smoothing factor; Step 2.5: Calculate the raw log probability for each class, normalize it using the Sigmoid function, and convert it into a pseudo-probability. ； in, To classify the input text using a Naive Bayes classifier After processing, the resulting text belongs to the first... The predicted probability of each label; Step 2.6: Obtain the final Naive Bayes ensemble prediction vector. : ； The Naive Bayes integrated prediction vector It is not updated during backpropagation and is used as an external signal to regularize the BERT model through consistency loss. Step 3: During the training phase, introduce consistency regularization loss, using the obtained Naive Bayes ensemble prediction vector as the external signal, to force the prediction probabilities of the BERT model and the Naive Bayes ensemble model to align with each label. Step 4: In the inference stage, the BERT model and the Naive Bayes ensemble model are jointly deployed and fused for prediction in a probabilistic manner to obtain multi-label sensitive text detection results.

2. The multi-label sensitive text detection method integrating BERT and Naive Bayes as described in claim 1, characterized in that, In step 1, the predicted probability of the BERT model is calculated as follows: Step 1.1: For a given input text sequence The maximum length is ,pass The function will assign each tag Mapping to context hidden vectors to generate hidden states : ； in, It is the hidden dimension of BERT; Step 1.2, Using special classification tokens The corresponding hidden state is used as a fixed-length representation of the entire sequence. The hidden vector corresponding to the token is extracted as follows: ； in, It is the hidden vector corresponding to the extracted token. It is a special classification token after the BERT model encodes the input text. The corresponding hidden state vector; Step 1.3: Project the aggregated vectors through a linear classification layer to a... In the output space of dimension: ； in, It is the weight matrix of the linear classification layer, used to represent the hidden states extracted by the BERT model. Projected onto the label space. It is the bias vector of the linear classification layer, used to adjust the linear transformation result; Step 1.4: Perform multi-label classification and apply the Sigmoid activation function to each element. : ； in, The result vector is obtained through linear classification layer calculation. The various elements in It is the probability output vector of the BERT model.

3. The multi-label sensitive text detection method integrating BERT and Naive Bayes as described in claim 1, characterized in that, The TF-IDF score mentioned in step 2.1 is calculated as follows: ； in, Indicating word frequency is the frequency of words. In the text The number of times it appears in the text divided by the number of times it appears in the text Total word count; Inverse document frequency (IVF) is the total number of documents in the corpus divided by the number of words contained in the document. The number of documents.

4. The multi-label sensitive text detection method integrating BERT and Naive Bayes as described in claim 1, characterized in that, In step 3, the consistency regularization loss is defined by the mean squared error and expressed as: ； Represented in vector form as follows: ； in, This represents the probability output vector of the BERT model. This represents the corresponding output vector of the Naive Bayes ensemble model. This results in a loss of consistency.

5. The multi-label sensitive text detection method integrating BERT and Naive Bayes as described in claim 4, characterized in that, For samples where the inconsistency between the BERT model and the Naive Bayes ensemble model exceeds a preset threshold, the consistency regularization loss is calculated using a gating function: ； in, For gated functions, For consistency loss, It is the result of the gating function being applied to the original consistency loss.

6. The multi-label sensitive text detection method integrating BERT and Naive Bayes as described in claim 5, characterized in that, The overall training objective, calculated by combining BERT's supervised classification loss and consistency loss, is expressed as: ； in, It is a non-negative hyperparameter used to control the relative importance of consistency regularization.

7. The multi-label sensitive text detection method integrating BERT and Naive Bayes as described in claim 6, characterized in that, In the inference process described in step 4, the prediction vector is calculated through convex combination: ； in, and These are the predicted probability vectors from the BERT model and the Naive Bayes ensemble model, respectively. It is the fusion weight hyperparameter, used to balance the contribution of each model.

8. The multi-label sensitive text detection method integrating BERT and Naive Bayes as described in claim 7, characterized in that, Apply a fixed threshold to each label dimension This transforms the soft probabilities into binary label decisions for multi-label classification: ； When the label distribution is highly unbalanced, adjust the label-specific threshold on the validation set. : ； in, It is a soft probability obtained from convex combination calculation. It is a binary label decision.

Images