Text style backdoor defense method based on multi-granularity variant generation and style immunization

Abstract

The invention discloses a text style backdoor defense method based on multi-granularity variant generation and style immunization. The method comprises the following steps: capturing a text style and content by combining explicit and implicit features; generating high-quality text variants on a plurality of granularities such as lexical, syntactic, style, context and the like by utilizing a large language model; carrying out label correction on the suspicious samples based on multi-dimensional risk assessment, and carrying out voting decision making by utilizing style neutralization variants and the like; and finally, through style invariant representation learning, style separation and style contrast training, the robustness of the model to style change is improved. According to the method, an explicit detection trigger is not needed, various attacks including style backdoors can be effectively defended, and through a systematic framework and a self-adaptive strategy, the safety and robustness of the model are remarkably improved while the normal performance of the model is ensured.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a text style backdoor defense method based on multi-granularity variant generation and style immunity. Background Art

[0002] Natural language processing technologies such as text classification are widely used in fields such as sentiment analysis and content review. However, malicious attackers may implement backdoor attacks by injecting hidden triggers into the training data, causing the model to output specified labels for samples containing triggers during the inference phase. Existing defense methods include trigger detection and removal, data augmentation and random perturbations, model-level defense, and post-processing defense. For example, trigger detection methods (such as keyword scanning) can identify explicit triggers (reference: Smith, J., et al., 2020. Detecting Backdoor Triggers in NLP Models. Journal of AI Security); data augmentation methods reduce trigger sensitivity through synonym replacement (reference: Li, Y., et al., 2021. Data Augmentation for Backdoor Defense. Proceedings of ACL).

[0003] Existing technologies suffer from the following shortcomings: trigger detection is ineffective against style-level triggers; data augmentation has limited coverage and can easily corrupt semantics; model-level defenses require a large number of augmented samples and have poor transferability; and post-processing relies on manual labor, resulting in low efficiency. These methods lack a systematic framework, and research on defenses against covert style attacks is insufficient. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the existing technology and provide a text style backdoor defense method based on multi-granularity variant generation and style immunity to improve the robustness of the model against text backdoor attacks.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A text style backdoor defense method based on multi-granularity variant generation and style immunity includes the following steps:

[0007] (1) Preprocessing: Initialize the style and semantic feature space, configure a multi-granularity variant generator based on a large language model, and generate variant templates at the lexical, syntactic, style, and context levels;

[0008] (2) Training data processing: extracting style and semantic features from input text, generating multi-granularity variants and screening high-quality variants, and correcting contaminated labels through style risk assessment and voting mechanisms;

[0009] (3) Style-immune training: Based on the corrected training data, we use style-invariant representation learning, style separation learning, and style contrast training to build a model that is robust to style triggers.

[0010] (4) Model deployment and inference: Use the constructed model test samples that are robust to style triggers to conduct online style risk assessment, adopt adaptive defense strategies based on the risk score, and generate the final prediction results.

[0011] Furthermore, the step (1) includes the following sub-steps:

[0012] (1.1) Feature Space Initialization: Collecting Diverse Style Benchmark Datasets Train the style encoder and semantic encoder E style , mapping text into style representation and semantic representation;

[0013] (1.2) Multi-granularity variant generator configuration: building lexical variant templates Syntactic-level variant templates Style-level variation templates and context-level variation templates described Includes style-preserving templates Style Neutral Template Style Contrast Template And design a content-preserving scoring function Q for evaluating the quality of variants c (x, x′) and the style transformation score function Q s (x, x′, type), and then calculate the variant quality score.

[0014] Specifically, the calculation expression of the variant quality score is as follows:

[0015] Q(x,x′)=β·Q c (x,x′)+(1-β)·Q s (x,x′,type)

[0016] where β is the weight coefficient that balances content preservation and style transformation.

[0017] Furthermore, the step (2) includes the following sub-steps:

[0018] (2.1) Multidimensional feature extraction: Extracting explicit style features S through linguistic feature analysis exp (x), use the style encoder to extract the implicit style representation S imp (x), calculate the comprehensive style representation S(x) and style abnormality D style (x);

[0019] (2.2) Multi-granularity variant generation: using large language models to generate lexical, syntactic, and style-level variants By screening out high-quality variants with quality scores higher than a preset threshold according to the variant quality scores Q(x,x′) designed in step (1.2);

[0020] (2.3) Style-aware label correction: Based on style risk R style (x), represents the similarity R sim (x) and the predicted conflict degree R conflict (x) Calculate the comprehensive risk score R total (x), and then divide the risk levels into different levels according to the risk scores and adopt corresponding label modification strategies: When R total When (x)<τ1, keep the original label y corrected =y; when τ1≤R total When (x)<τ2, the intermediate risk label correction mechanism y is adopted corrected =StyleVote(x,medium), which modifies labels by voting on style and variants; R total When (x)≥τ2, the advanced risk label correction mechanism y is used corrected =StyleVote(x,high), which modifies the label by quality-weighted voting; where τ1 and τ2 are risk threshold parameters.

[0021] Furthermore, the step (3) includes the following sub-steps:

[0022] (3.1) Training data preparation: building a revised training set Expanding high-quality variant data Generating hard negative sample pairs

[0023] (3.2) Style Immunity Training: Using Style Invariant Representation to Learn the Loss Function To make different styles of the same semantics closer, we use style separation learning loss function Make the style features orthogonal to the content features and use the style contrast training loss function Distinguish samples with the same semantics but opposite styles, combined with the cross entropy classification loss function To ensure model classification performance, these loss functions are weighted and combined Optimization model, where η1, η2, and η3 are weight coefficients.

[0024] Furthermore, the step (4) includes the following sub-steps:

[0025] (4.1) Online style risk assessment: Extract the style representation S(x test ), calculate the style deviation score Dstyle (x test ) and the prediction variance Generate a comprehensive risk score R infer (x test );

[0026] (4.2) Adaptive defense decision: Samples are divided into low-risk, medium-risk, and high-risk samples according to their risk scores. Low-risk samples are predicted directly, medium-risk samples are predicted using the average prediction of style and variants, and high-risk samples are predicted using multi-variant quality weighted prediction.

[0027] The beneficial effects of the present invention are as follows:

[0028] The innovation of the method of the present invention lies in that it can defend against lexical-level, syntactic-level and style-level backdoor attacks without explicitly detecting triggers, overcoming the defect of insufficient detection of hidden triggers in existing technologies; multi-granularity variant generation has comprehensive coverage and maintains semantics, solving the problem of single data enhancement; automatic label correction reduces manual dependence and improves efficiency; the systematic defense framework is highly adaptable and suitable for a variety of NLP application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a flowchart of the method of the present invention. DETAILED DESCRIPTION

[0030] The core technology of this invention is to build a defense system against text backdoor attacks through multi-dimensional feature modeling, multi-granularity variant generation, style-aware label correction, and style immunity training. The following is a detailed description with reference to specific examples:

[0031] This paper proposes a text style backdoor defense method based on multi-granularity variant generation and style immunity, such as Figure 1 As shown, the following steps are included:

[0032] (1) Preparation before training:

[0033] (1.1) Feature space initialization: Collect benchmark datasets with various styles including news, social media, etc. Training BERT-based style encoder and semantic encoder Generate style representation and semantic representation;

[0034] (1.2) Multi-granularity variant generator configuration: Construct templates such as “replace nouns with synonyms”, “rewrite as main clause + subordinate clause”, and “rewrite using neutral style”, and design scoring functions, where the content preservation score is based on cosine similarity, the style transformation score is based on style representation distance, and the comprehensive quality score is a weighted sum.

[0035] Construct a lexical variant template: For example: "Replace key nouns in the text with synonyms, but keep the overall meaning unchanged";

[0036] Syntactic-level variant templates: For example: "Restructure the sentence structure and rewrite the main and subordinate complex sentences into parallel sentences, while keeping the original meaning unchanged";

[0037] Style-level variation templates Includes style preserving templates: Style Neutralization Template: Style comparison template:

[0038] And the context-level variant template: For example: "Add additional context while keeping the core idea intact."

[0039] Design a content-preserving scoring function Q c (x,x′)=sim(E content (x),E content (x′)), used to measure the similarity between the original text and the variant in the semantic space;

[0040] Style transfer scoring function Q s (x,x′,type), evaluates the appropriateness of the style transformation according to the variant type, for the style-preserving variant Q s (x,x′,sp)=sim(E style (x),E style (x′)), for style neutralization

[0041] Q s (x,x′,sn)=sim(E style (x′),μ), for style contrast variants

[0042] Q s (x,x′,sc)=1-sim(E style (x),E style (x′)).

[0043] Finally, calculate the comprehensive quality score Q(x,x′) = β·Q c (x,x′)+(1-β)·Q s (x,x′,type), where β is the weight coefficient that balances content preservation and style transformation.

[0044] (2) Training data processing:

[0045] (2.1) Multidimensional feature extraction: first perform explicit style feature extraction

[0046] S exp (x) = [formality(x), complexity(x), sentiment_bias(x), ...], calculated through linguistic feature analysis, including quantifiable indicators such as formality(x), complexity(x), and sentiment bias(x); then obtain the implicit style representation S imp (x) = E style (x), extract deep style representation using a pre-trained style encoder.

[0047] Calculate the comprehensive style representation S(x) = α·S exp (x)+(1-α)·S imp (x), where α is a coefficient that balances explicit and implicit features.

[0048] It is based on the Mahalanobis distance to measure the deviation of the sample from the normal style distribution, that is, the style abnormality; calculate the style consistency C within the class style (x,y)=sim(S(x),μ y ), evaluates the consistency between the sample style and the intra-class style center corresponding to its label.

[0049] (2.2) Multi-granularity variant generation: Generate variants using a large language model. For example, for the input “he is angry”, generate the lexical variant “he is very angry”, the syntactic variant “he is very angry”, and the stylistic neutralization variant “he feels unhappy”. Variants with a quality score greater than 0.8 are selected.

[0050] Lexical-level variant generation:

[0051]

[0052] The input text is rewritten at the word level through the Large Language Model (LLM), such as synonym replacement and entity transformation.

[0053] Syntactic-level variant generation:

[0054]

[0055] Through LLM, the sentence structure is changed, such as voice conversion and sentence reconstruction, while keeping the semantics unchanged.

[0056] Style-level variant generation

[0057] (a) Style-preserving variants Keep the original style but use different expressions;

[0058] (b) Style Neutralization Variation Convert text to a standard, neutral style;

[0059] (c) Style contrast variants Generate expressions that are opposite to the original style.

[0060] (d) Context-level variant generation By adding or modifying contextual information, the core semantics remain unchanged.

[0061] Variant quality assessment and screening:

[0062]

[0063] in is the union of all generated variants:

[0064]

[0065] δ is the quality screening threshold, which is used to filter low-quality variants.

[0066] (2.3) Style-aware label correction: Calculating style risk:

[0067] R style (x) = γ1·D style (x)+γ2·(1-C style (x,y))

[0068] Among them, γ1 and γ2 are weight coefficients used to balance the importance of the two risk factors, and y represents the original label of the sample; D style (x) is the style abnormality, which uses the Mahalanobis distance to measure the deviation of the sample style from the normal style distribution; C style (x,y) is the intra-class style consistency, which evaluates the consistency between the sample style and the intra-class style center corresponding to its label.

[0069] The style risk is assessed by combining style anomaly and intra-class consistency; the similarity anomaly is calculated:

[0070]

[0071] in, is a set of high-quality variants of sample x; represents the size of the variant set; finit(x) is the feature extraction of the original sample by the preliminary model; distance calculates the distance between the original sample and its variant in the feature space, which is used to detect anomalies in similarity and identify potential trigger effects.

[0072] Evaluate the distance anomaly between the original sample and its variant in the feature space; calculate the prediction conflict degree:

[0073]

[0074] Measure the consistency between the original sample and its variant prediction results; finally calculate the comprehensive risk score:

[0075] R total (x)=w1·R style (x)+w2·R sim (x)+w3·R conflict (x),

[0076] Where w1, w2, w3 are the weight coefficients of each risk factor. Style-aware label decision:

[0077] Make label decisions based on risk scores:

[0078] If R total (x)<τ1, then y corrected =y(keep original label);

[0079] If τ1≤R total (x)<τ2, then y corrected =StyleVote(x,medium)(medium risk label correction);

[0080] If R total (x)≥τ2, then y corrected =StyleVote(x,high)(Advanced Risk Label Correction).

[0081] Among them, τ1 and τ2 are risk score threshold parameters used to distinguish low-risk, medium-risk and high-risk samples; y represents the original label of the sample; y corrected Represents the label after style-aware label correction; StyleVote(x,medium) is a medium-risk label correction function based on the majority vote of style and variants; StyleVote(x,high) is a high-risk label correction function based on the quality-weighted voting of all high-quality variants. Style-aware voting mechanism:

[0082] For medium-risk voting:

[0083] majority voting based on style neutralization and variation;

[0084] For advanced risk voting:

[0085] A quality-weighted vote based on all high-quality variants.

[0086] LLM auxiliary verification:

[0087] For high-risk samples, additional verification is performed via LLM:

[0088] The final judgment is made based on the original text, variant content, original labels and revised labels.

[0089] (3) Style immunity training:

[0090] (3.1) Training data preparation: building a revised training set Corrected labels using style-aware labels; augmented variant data Add high-quality variants to the training set, using the corrected labels; construct hard negative pairs for each example Samples with similar semantics but different categories are selected as hard negative samples.

[0091] (3.2) Style Immunity Training: Style Invariant Representation Learning:

[0092]

[0093] where z i =f(x i ) is the feature representation of the original sample, is the representation of its style and variants, τ is the temperature parameter, and sim is the cosine similarity function, which is used to calculate the similarity between two feature vectors. This loss function encourages the model to learn style-independent feature representations, making different styles of expressions of the same semantic content close in the feature space.

[0094] Style Separation Learning:

[0095]

[0096] where f style and f content These are style and content feature extractors. This loss function promotes orthogonal separation of style features and content features, reducing mutual interference.

[0097] Style contrast training:

[0098]

[0099] in is x i A style contrast variant of , m is the margin parameter, and d is the distance function. This loss function forces the model to distinguish samples with the same semantics but opposite styles, enhancing robustness to style triggers.

[0100] Classification loss:

[0101]

[0102] The standard cross entropy loss function ensures that the model maintains good classification performance.k It represents the model's predicted probability that the sample belongs to the kth category.

[0103] Comprehensive training objectives:

[0104]

[0105] Among them, η1, η2, and η3 are the weight coefficients of each loss term, which are used to balance the importance of each target.

[0106] Three-stage training process:

[0107] In the first stage, style-aware pre-training is performed on the original data to obtain f init ; The second stage performs main style immunity training on the label-corrected data; the third stage performs style adaptive fine-tuning on difficult examples.

[0108] (4) Model deployment and inference:

[0109] (4.1) Online style risk assessment: First, extract the style representation of the test sample: S(x test ); then calculate the style deviation score Then generate a small batch of style neutralization variants Measuring the model's predicted variance for style neutrality and variation Finally calculate the comprehensive risk score Where δ1 and δ2 are weight coefficients.

[0110] (4.2) Adaptive defense decision-making: Based on the risk score, ARTIST adopts a three-level defense strategy:

[0111] Low-risk samples (R infer (x test )<θ1): Use the model to predict the result directly without additional defense measures: prediction = f(x test ).

[0112] Medium risk samples (θ1≤R infer (x test )<θ2): Use the average prediction results of style neutralization and variants to reduce the influence of style triggers:

[0113] High-risk samples (R infer (x test )≥θ2): A more comprehensive defense mechanism is adopted to further improve the robustness through multi-variant quality weighted averaging:

[0114] First generate more style-neutralizing variants:

[0115]

[0116] Calculate the quality weight:

[0117]

[0118] Weight normalization:

[0119] normalized_weights=normalize(weights);

[0120] Finally, weighted prediction

[0121]

[0122] Example

[0123] This invention was implemented on a high-performance server equipped with four NVIDIA A6000 GPUs, using the aforementioned parameters, and experiments were conducted on a classification dataset containing 100,000 texts. The results demonstrate advantages over existing technologies. This method effectively addresses lexical, syntactic, and style-level triggers through a full-spectrum defense mechanism, addressing the shortcomings of single-trigger defense. It employs multidimensional feature modeling to separate style and content representations, significantly improving the detection of hidden triggers. It utilizes a large language model to generate diverse variants and ensure semantic consistency, addressing the limited coverage of data augmentation. An automated label correction mechanism addresses label contamination processing and reduces manual reliance. A style-immune training framework enhances model robustness and ensures classification performance. An adaptive reasoning defense strategy optimizes resource allocation and improves reliability. A systematic defense system is constructed to achieve full-process protection and ensure long-term adaptability. These advantages collectively constitute a comprehensive, efficient, and sustainable text backdoor defense solution.

Claims

1. A text style backdoor defense method based on multi-granularity variant generation and style immunity, characterized by: The following steps are involved: (1) Preprocessing: Initialize the style and semantic feature space, configure a multi-granularity variant generator based on a large language model, and generate variant templates at the lexical, syntactic, style, and context levels; (2) Training data processing: extracting style and semantic features from input text, generating multi-granularity variants and screening high-quality variants, and correcting contaminated labels through style risk assessment and voting mechanisms; (3) Style-immune training: Based on the corrected training data, we use style-invariant representation learning, style separation learning, and style contrast training to build a model that is robust to style triggers. (4) Model deployment and inference: Use the constructed model test samples that are robust to style triggers to conduct online style risk assessment, adopt adaptive defense strategies based on the risk score, and generate the final prediction results.

2. The text style backdoor defense method based on multi-granularity variant generation and style immunity according to claim 1 is characterized in that: The step (1) includes the following sub-steps: (1.1) Feature Space Initialization: Collecting Diverse Style Benchmark Datasets Train the style encoder and semantic encoder E style , mapping text into style representation and semantic representation; (1.2) Multi-granularity variant generator configuration: building lexical variant templates Syntactic-level variant templates Style-level variation templates and context-level variation templates described Includes style-preserving templates Style Neutral Template Style Contrast Template And design a content-preserving scoring function Q for evaluating the quality of variants c (x,x ′ ) and the style transformation score function Q s (x,x ′ ,type), and then calculate the variant quality score.

3. The text style backdoor defense method based on multi-granularity variant generation and style immunity according to claim 2 is characterized in that: The calculation expression of the variant quality score is as follows: Q(x,x ′ )=β·Q c (x,x ′ )+(1-β)·Q s (x,x ′ ,type) where β is the weight coefficient that balances content preservation and style transformation.

4. The text style backdoor defense method based on multi-granularity variant generation and style immunity according to claim 1 is characterized in that: The step (2) includes the following sub-steps: (2.1) Multidimensional feature extraction: Extracting explicit style features S through linguistic feature analysis exp (x), use the style encoder to extract the implicit style representation S imp (x), calculate the comprehensive style representation S(x) and style abnormality D style (x); (2.2) Multi-granularity variant generation: using large language models to generate lexical, syntactic, and style-level variants By the variant quality score Q(x,x) designed in step (1.2) ′ ) Screen out high-quality variants with quality scores higher than a preset threshold; (2.3) Style-aware label correction: Based on style risk R style (x), represents the similarity R sim (x) and the predicted conflict degree R conflict (x) Calculate the comprehensive risk score R total (x), and then divide the risk levels into different levels according to the risk scores and adopt corresponding label modification strategies: When R total When (x)<τ1, keep the original label y corrected =y; when τ1≤R total When (x)<τ2, the intermediate risk label correction mechanism y is adopted corrected =StyleVote(x,medium), which modifies labels by voting on style and variants; R total When (x)≥τ2, the advanced risk label correction mechanism y is used corrected =StyleVote(x,high), which modifies the label by quality-weighted voting; where τ1 and τ2 are risk threshold parameters.

5. The text style backdoor defense method based on multi-granularity variant generation and style immunity according to claim 1 is characterized in that: The step (3) includes the following sub-steps: (3.1) Training data preparation: building a revised training set Expanding high-quality variant data Generating hard negative sample pairs (3.2) Style Immunity Training: Using Style Invariant Representation to Learn the Loss Function To make different styles of the same semantics closer, we use style separation learning loss function Make the style features orthogonal to the content features and use the style contrast training loss function Distinguish samples with the same semantics but opposite styles, combined with the cross entropy classification loss function To ensure model classification performance, these loss functions are weighted and combined Optimization model, where η1, η2, and η3 are weight coefficients.

6. The text style backdoor defense method based on multi-granularity variant generation and style immunity according to claim 1 is characterized in that: The step (4) includes the following sub-steps: (4.1) Online style risk assessment: Extract the style representation S(x test ), calculate the style deviation score D style (x test ) and the prediction variance Generate a comprehensive risk score R infer (x test ); (4.2) Adaptive defense decision: Samples are divided into low-risk, medium-risk, and high-risk samples according to their risk scores. Low-risk samples are predicted directly, medium-risk samples are predicted using the average prediction of style and variants, and high-risk samples are predicted using multi-variant quality weighted prediction.

Images