A malicious text interception model training method based on group relative strategy optimization

Abstract

The present application relates to the technical field of natural language processing, in particular to a malicious text interception model training method based on group relative strategy optimization; the method comprises the following steps: constructing a text dataset containing category labels; using a group relative strategy optimization algorithm for reinforcement learning training, generating multiple candidate outputs with thinking chains for the same input, and calculating rewards using a double reward function, which combines rule rewards based on hard labels and process rewards based on large language model discrimination, calculates the relative advantage within the group, and optimizes the model parameters by combining strategy constraint loss to improve the depth and accuracy of harmful identification; a fast-slow adaptive training mechanism is introduced, the preference data is constructed by comparing the income difference between the outputs with and without thinking chains, and the model is trained using a preference optimization algorithm so that it can adaptively select the thinking mode according to the input complexity; a flexible malicious text interception model is obtained. The present application can balance the identification accuracy, reasoning interpretability and response efficiency.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a training method for a malicious text interception model based on group relative strategy optimization. Background Technology

[0002] The rapid development of Large Language Models (LLMs) has greatly enriched online information interaction, but it has also brought serious content security challenges. Malicious users may induce or "hijack" models to generate harmful content, such as harassing, deceptive, and inflammatory texts, through carefully constructed prompts. Existing malicious text interception training methods have obvious shortcomings: traditional methods mostly rely on rule templates and supervised fine-tuning classification models (such as BERT), which have a certain semantic understanding ability, but their generalization ability is weak and they are prone to misjudgment when facing new threats such as adversarial examples and jailbreak attacks.

[0003] Large-scale model-based identification methods typically employ human preference alignment techniques (such as DPO) combined with efficient fine-tuning (such as LoRA). While these methods can improve performance, they still heavily rely on a large amount of manually labeled preference data, resulting in high costs. Furthermore, they focus on overall generation alignment, making it difficult to achieve accurate and detailed security alignment in tasks requiring fine semantic discrimination, such as malicious content identification. As malicious text patterns continue to evolve, existing methods struggle to adapt to dynamic threats, limiting their identification accuracy and generalization capabilities.

[0004] The recent superior performance of models such as DeepSeek-R1 in complex inference tasks is attributed to the Group Relative Policy Optimization (GRPO) algorithm. This algorithm updates policies based on the relative advantages of candidate outputs within a group, providing a new approach to improve the discriminative performance of models in complex scenarios. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the background technology by proposing a training method for a malicious text interception model based on group-relative strategy optimization.

[0006] The technical solution of this invention: a method for training a malicious text interception model based on group-relative strategy optimization, comprising the following specific implementation steps: S1. Construct a text dataset with harmful and harmless category labels, and divide it into training set, validation set and test set; S2. A group relative policy optimization algorithm is used for reinforcement learning training to generate multiple candidate outputs containing thought chains for the same input. The reward value of each candidate output is calculated using a dual reward function. The model parameters are optimized based on the relative advantage within the group combined with the policy constraint loss to improve the depth and accuracy of harmful text recognition. S3. The dual reward function includes a result-oriented rule reward and a process-oriented model discrimination reward. The rule reward is calculated based on manually labeled hard tags and weighted from four dimensions: accuracy of judging harmful categories, accuracy of judging harmful subcategories, compliance of output format, and the proportion of Chinese in the thinking process. The model discrimination reward uses a large language model discriminator to score the logic, completeness, and readability of the thought chain. S4. Perform fast and slow adaptive reinforcement learning training. Use the model trained in step S2 to generate long thought chain output and no thought chain output for each sample. Calculate the reward value of the two and compare the difference in benefits. Construct preference pair data based on the difference in benefits and a preset threshold. Use the direct preference optimization algorithm to train the model so that it can adaptively select the thinking mode according to the complexity of the input text. S5. After training, a malicious text interception model is obtained. This model supports fully automatic adaptive mode and manual forced mode. It can automatically select or force the use of fast reasoning or slow thinking mode according to the input or according to the instruction, and output structured recognition conclusions.

[0007] Preferably, in step S2, the group relative strategy optimization algorithm includes the following steps: Input text data into the current model, and generate n different candidate outputs by independently sampling and adjusting the generation parameters. Each candidate output contains the thought process and the final conclusion. Calculate the reward value for each candidate output; Calculate the relative advantage of each candidate output within the group, where the relative advantage is the difference between the reward value of the candidate output and the average reward value within the group; Based on the relative advantage, construct the group relative reward loss, and combine it with the policy constraint loss to form the total loss function; The model parameters are updated by minimizing the total loss function, and this process is repeated iteratively until the model performance stabilizes.

[0008] Preferably, the policy constraint loss is the KL divergence loss, which is used to constrain the distribution difference between the updated policy model and the policy model before the update.

[0009] Preferably, in step S3, the rule reward is calculated by weighted summation, where the weight of each sub-item is configurable and the score of each sub-item is a binary score.

[0010] Preferably, in step S3, the discriminator used for model discrimination reward is a large language model with a larger number of parameters than the target model. The discriminator scores the input text and thought chain content from three dimensions: logic, completeness and readability, and takes the average value as the reward.

[0011] Preferably, in step S4, the specific steps for constructing the preference pair data include: For each training sample, generate slow thinking candidate outputs containing long thought chains and fast reasoning candidate outputs without thought chains. Calculate the reward value for each of the two candidate outputs; Calculate the payoff difference between slow-thinking candidates and fast-reasoning candidates; If the difference in returns is greater than the preset bias threshold, then the slow thinking candidate will be used as the preferred output and the fast reasoning candidate will be used as the non-preferred output. If the payoff difference is less than the negative bias threshold, then the fast reasoning candidate is taken as the preferred output and the slow thinking candidate is taken as the non-preferred output.

[0012] Preferably, in step S5, the manual forced mode is achieved by attaching specific tags: attaching an end-of-thinking tag after the input text to force fast reasoning; attaching a start-of-thinking tag after the input text to force slow thinking and output the complete thought chain.

[0013] Preferably, in step S1, the constructed dataset includes subclass labels for harmful categories, and the subclasses include at least one of pornography, violence and threats, hate speech, illegal information, privacy violations, harassment and abuse, dangerous behavior inducement, and false information.

[0014] Preferably, in step S2, the thought chain and conclusion of the candidate output are organized according to a preset format: the thought chain content is located between a pair of start and end thinking labels, and the final conclusion is located after the end thinking label.

[0015] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial technical effects: This invention designs a training method for a malicious text interception model based on group relative policy optimization. First, by employing a training framework combining a group relative policy optimization algorithm with a dual reward function, the model can deeply mine the latent semantics and contextual relationships of the text, significantly improving the accuracy and generalization ability in identifying concealed and complex malicious content. Second, by introducing a process-oriented model discrimination reward, the model generates logically clear and comprehensively analytical thought chains, enhancing the interpretability and reliability of decisions and facilitating manual review and system auditing. Third, by constructing preference pairs based on payoff differences and utilizing a direct preference optimization algorithm to achieve fast-slow adaptive inference, the model can... The system intelligently balances recognition effectiveness and response efficiency, quickly outputting conclusions for simple text and initiating deep analysis for complex text, thus achieving the optimal trade-off between efficiency and effectiveness in real-world applications. Furthermore, the entire training method is based on single-round labeled data and configurable rule rewards, effectively reducing data labeling and model iteration costs and enhancing the feasibility and engineering potential of the solution. The resulting model supports both fully automatic adaptive and manual forced application modes, flexibly adapting to business scenarios with different security levels and response latency requirements, providing a complete and innovative technical path for building an efficient, reliable, and interpretable malicious text interception system. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method for training a malicious text interception model based on group relative strategy optimization proposed in this invention. Figure 2 This is a schematic diagram illustrating the principle of the long thought chain reinforcement learning training stage proposed in this invention. Figure 3 This is a schematic diagram illustrating the principle of the fast-slow adaptive reinforcement learning stage proposed in this invention. Detailed Implementation

[0017] Example 1, as Figure 1 As shown, the present invention proposes a method for training a malicious text interception model based on group-relative strategy optimization, which includes the following specific implementation steps: S1. Collect text from online sources and define a classification system that includes "harmful / harmless" and specific harmful subcategories (such as violence and pornography). Perform manual single-round category labeling to construct a dataset, which is then divided into training, validation, and test sets to provide low-cost and clear supervision signals for subsequent training. Specifically: We collect raw text data extensively from potentially harmful information dissemination channels such as public internet forums, social media, and instant messaging tools; at the same time, we can introduce known malicious instruction sets, adversarial attack samples, and jailbreak attack templates. The collected data is initially cleaned to remove duplicate, invalid, and overly verbose content. Define a clear classification system for harmful content, which should include at least one binary label, namely "harmful" or "harmless"; To improve the model's fine-grained recognition capability, it is preferable to further subdivide the "harmful" category; For example, they can be divided into subcategories such as: "pornography", "violent threats", "hate speech / discrimination", "illegal information", "privacy violations", "harassment and abuse", "dangerous behavior inducement", and "false information"; specific categories can be adjusted and expanded according to actual business needs. Based on the defined category system, the cleaned text data is manually labeled; the labeling process is a single-round judgment, that is, for each text, it is labeled whether it is "harmful" and the specific harmful subclass (if harmful); this labeling method is significantly less costly than complex labeling tasks that require labeling multiple rounds of dialogue preferences or generating content quality ranking; after the labeling is completed, sampling quality inspection is required to ensure the consistency of the labeling. The labeled dataset is randomly divided into training, validation, and test sets, for example, in a ratio of 70%:15%:15%. The training set is used for updating model parameters, the validation set is used for hyperparameter tuning and monitoring the training process, and the test set is used for objective evaluation of the final model performance. The final dataset samples are shown in Table 1: Table 1 Sample Dataset Table .

[0018] S2. Reinforcement learning training based on the GRPO (Group Relative Policy Optimization) algorithm is used to optimize long thought chain reasoning. This involves having the target model generate multiple candidate outputs containing thought chains for the same input, calculating the reward for each candidate using a dual reward function, and calculating the relative advantage within each group. A total loss function is constructed by combining the group relative reward loss and the KL divergence policy constraint. The model parameters are iteratively updated by minimizing this loss to improve the depth and accuracy of harmful identification. Specifically: Choose an open-source Basic Large Language Model (LLM) as the target model to be trained. Its parameters are ; For example, a model with a parameter range of 7B to 13B can be selected as a starting point; Define a standardized input prompt template; For example, “Please determine whether the following input text contains harmful content and explain your reasoning; Text: [Text to be identified]; Please put your thought process in…” <think> and< / think> "Between the labels, and place the final conclusion afterward"; Here, represents a command marker used for structured model output and controlling model behavior; that is, a key control symbol with specific functions and meanings. Define the desired output format: <think> [Detailed thought process behind model generation, including text semantic analysis, potential hazard profiling, and comparison with hazard category definitions, etc.]< / think> [Final conclusion, such as: "harmless" or "harmful; [specific subclass]"]; The GRPO training iteration process is executed as follows: A1. State and Action Sampling: The i-th text data... After formatting the Prompt template as described above, input the current target model. By adjusting generation parameters (such as temperature coefficient and Top-p sampling), n different candidate outputs (e.g., n=4 or 8) are independently sampled from the model strategy. Each candidate output They all contain a chain of thought and a final conclusion; A2. Reward Calculation: For each candidate output Calculate reward value This reward value is calculated by the "dual reward function" detailed in subsequent step S3, which comprehensively evaluates the accuracy of the output conclusion (based on the hard label of step S1) and the quality of the thought process. A3. Advantage Calculation: Calculate the relative advantage of each action within the group, as follows: ; in, Indicates relative advantage, characterizes specific actions The difference between the reward and the average reward for this set of actions; A4. Loss Function Construction and Optimization: Constructing the total loss function for GRPO And perform gradient updates: Group relative reward loss Encourage the model to increase the probability of producing outputs with a high relative advantage. ; Policy-constrained (KL divergence) loss Preventing new strategies Deviating too much from the old strategy (i.e., the model before this parameter update), ensuring training stability: ; Total loss ; in, This represents the policy probability, i.e., in the state (data). Below, the current strategy model Generate candidate output The probability of; This represents the KL divergence (Kullback-Leibler divergence), which measures the difference between two probability distributions. This represents the old policy distribution, the policy model before the last parameter update (the old model) in state. The output probability distribution is as follows; This represents the distribution of the new policy, where the current policy model to be optimized is in state. The output probability distribution is as follows; Indicates the KL constraint coefficient; A5. Parameter Update: Calculation Regarding model parameters The gradient is calculated, and the parameters are updated using an optimizer (such as AdamW); after a batch is completed, the parameters are updated. For the current , used for calculating KL constraints in the next iteration; Repeat steps A1 to A5 above for multiple epochs until the model’s reward score or harmful identification accuracy on the validation set stabilizes.

[0019] S3. Design a dual reward function for the harmful content identification task. The result-oriented rule reward is based on manually labeled hard tags, which perform deterministic scoring based on four dimensions: harmful category judgment, harmful subcategory judgment, output format compliance, and Chinese content ratio. The process-oriented model discrimination reward uses a large-parameter LLM discriminator to score the logic, completeness, and readability of the thought process. The weighted sum of the two rewards provides a comprehensive feedback signal for GRPO training. Figure 2 As shown, specifically: Reward Function 1: Define Outcome-Oriented Rule-Based Rewards (0~1.0 points): Based on manually labeled hard tags and preset formatting rules, it provides deterministic feedback, the calculation of which includes four configurable weighted sub-items: Accuracy of Hazard Classification Judgment (Weight) ): Analyze the final conclusion part of candidate output 'a' and extract the judgment of "harmful / harmless"; compare it with the true label. If they are completely consistent (e.g., the model outputs "harmful" and the true label is "harmful", or the model outputs "harmless" and the true label is "harmless"), then a score is awarded. ;otherwise ; Accuracy of harmful subclass identification (weight) ): This only applies when the true label is harmful and contains subcategories; it parses the harmful subcategory descriptions in the model's conclusions and performs semantic matching (such as string inclusion or similarity calculation) with the true harmful subcategory labels; if a match is successful, a score is awarded. ;otherwise If the model does not output a subclass or the true label is harmless, no points will be awarded for this item. Output format compliance (weight) ): Check if output 'a' strictly satisfies: contains one and only one complete pair. <think> and< / think> Tags; the thought content is located between these tags; the final conclusion is located after the tags; the length of the thought content does not exceed the preset token limit. (e.g., 512); if all conditions are met, the score is awarded. If any one of them is not satisfied, then ; Readability of the thought process (weight) ):calculate <think> and< / think> The percentage of Chinese characters in the text between tags; if the percentage is not lower than a preset threshold. If the reading comprehension score is 70%, then the reading comprehension is considered satisfactory, and the score is [score missing]. ;otherwise ; Ultimately, result-oriented rule rewards are obtained. : ; Reward Function 2: Defining a Procedural Model for Reward Judgment (1~1.0 points) This reward aims to evaluate the quality of the thought chain itself, requiring a high-performance LLM with a larger number of parameters as the discriminator. (e.g., Qwen3-32B, GPT-4, etc.): Discriminant input construction: Combine the original query text s and the thought chain part CoT(a) of the candidate output to form a hint for the discriminator; For example, “Please evaluate the quality of the following thought process in analyzing text ‘[s]’; thought process: [CoT(a)]; please score it in terms of logic, completeness and readability (0.0-1.0) and give the final average score”; Multidimensional scoring: discriminator Output scores in three dimensions based on the instructions: Logic : Are the reasoning steps interconnected? Are there any contradictions between the premises and the conclusion? Do they conform to common sense or domain knowledge? Integrity Does it cover the key suspicious points in the text? Does it consider the possibility of different interpretations? Is the analysis comprehensive? readability : Is the expression clear and fluent? Is the structure distinct? Are there any serious grammatical errors or confusing expressions? For stability and cost considerations, it can be used in advance. The training set generates thought chains, which are then batch-scored and cached, or the scores are updated every few steps during training. Ultimately, a process-oriented model discriminative reward is obtained. : ; Rewards based on results And process-oriented model for discriminating rewards The weighted summation yields the GRPO training signal, which provides a comprehensive feedback signal.

[0020] S4. Perform fast-slow adaptive reinforcement learning training. Use the GRPO model to generate two candidate outputs for each sample: one with a long thought chain and one without. Calculate their rewards. Construct preference pairs by comparing the difference in rewards between the two outputs with a threshold. Train the model using these preference pairs based on the DPO algorithm. This allows the model to learn to adaptively select between fast reasoning and slow thinking modes based on input complexity, or to be forced to switch between them by external instructions, thereby balancing recognition performance and efficiency. Figure 3 As shown, specifically: S41. Prepare training data, using samples x and their labels from the training set. For each sample data x, perform the following procedure: B1. Generate candidate pairs: Slow Thinking Candidates : Use the model trained in step S2 (called Generated in the normal way, i.e., the input Prompt ends with... <think>, get output <think> [CoT]< / think> [in conclusion]; Fast reasoning candidate Force the model to perform fast inference; modify the input Prompt by adding the following to the end:< / think> Tags are used to indicate that the thinking process has ended, such as: "...text: [x]; please judge; <think>< / think> The model will be forced to generate direct conclusions that begin with phrases like "harmful" or "pornographic." B2. Calculate absolute return: using the reward function from step S3. Calculate separately , Reward value , : and ; B3. Identify Preference Pairs: Calculate the relative payoffs of the thought chain: Profitability difference : ; Introduce a bias threshold (For example (), used to fine-tune preferences for efficiency or effectiveness; like This indicates that for this sample, the improvement brought by using long thought chains exceeds the threshold, therefore the preference output is... The non-preference output is ; like This indicates that using the thought chain method is not very beneficial for this sample, and may even introduce errors or delays due to overthinking. Therefore, the preferred output is... The non-preference output is ; If the absolute value of the difference in returns is not greater than If the sample does not constitute a valid preference pair, it can be skipped. S42. Execute the DPO algorithm training, using the preferences constructed above on the dataset. For the model (initialization The DPO algorithm's loss function is used for training. as follows: ; in, This represents the Sigmoid function.

[0021] S5. After completing all the above steps S1~S4, the final deployable model is obtained. Its application is flexible: Fully automatic adaptive mode: Users only need to input the text to be recognized (no special labels required), and the model will automatically decide whether to use fast reasoning or generate a complete thought chain based on its internalized strategy, and output the final conclusion; for simple and obvious malicious text, it may output a conclusion quickly; for complex and hidden text, it will start slow thinking. Manual Force Mode: If you need to force fast reasoning to pursue the ultimate efficiency, then append a label as an instruction after the input text; If you need to force slow thinking to obtain detailed analysis evidence (such as in a review or audit scenario), then append the text after entering it. <think>Tags serve as instructions; Output results: Regardless of the mode, the model ultimately outputs structured conclusions, which are easy for downstream systems to process. For example, the "whether it is harmful" and "harm type" fields can be directly parsed.

[0022] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.< / think>

Claims

1. A method for training a malicious text interception model based on group-relative strategy optimization, characterized in that, The specific implementation steps include the following: S1. Construct a text dataset with harmful and harmless category labels, and divide it into training set, validation set and test set; S2. A group relative policy optimization algorithm is used for reinforcement learning training to generate multiple candidate outputs containing thought chains for the same input. The reward value of each candidate output is calculated using a dual reward function. The model parameters are optimized based on the relative advantage within the group combined with the policy constraint loss to improve the depth and accuracy of harmful text recognition. S3. The dual reward function includes a result-oriented rule reward and a process-oriented model discrimination reward. The rule reward is calculated based on manually labeled hard tags and weighted from four dimensions: accuracy of judging harmful categories, accuracy of judging harmful subcategories, compliance of output format, and the proportion of Chinese in the thinking process. The model discrimination reward uses a large language model discriminator to score the logic, completeness, and readability of the thought chain. S4. Perform fast and slow adaptive reinforcement learning training. Use the model trained in step S2 to generate long thought chain output and no thought chain output for each sample. Calculate the reward value of the two and compare the difference in benefits. Construct preference pair data based on the difference in benefits and a preset threshold. Use the direct preference optimization algorithm to train the model so that it can adaptively select the thinking mode according to the complexity of the input text. S5. After training, a malicious text interception model is obtained. This model supports fully automatic adaptive mode and manual forced mode. It can automatically select or force the use of fast reasoning or slow thinking mode according to the input or according to the instruction, and output structured recognition conclusions.

2. The method for training a malicious text interception model based on group relative strategy optimization according to claim 1, characterized in that, In step S2, the group relative policy optimization algorithm includes the following steps: Input text data into the current model, and generate n different candidate outputs by independently sampling and adjusting the generation parameters. Each candidate output contains the thought process and the final conclusion. Calculate the reward value for each candidate output; Calculate the relative advantage of each candidate output within the group, where the relative advantage is the difference between the reward value of the candidate output and the average reward value within the group; Based on the relative advantage, construct the group relative reward loss, and combine it with the policy constraint loss to form the total loss function; The model parameters are updated by minimizing the total loss function, and this process is repeated iteratively until the model performance stabilizes.

3. The method for training a malicious text interception model based on group relative strategy optimization according to claim 2, characterized in that, The policy constraint loss is the KL divergence loss, which is used to constrain the distribution difference between the updated policy model and the policy model before the update.

4. The method for training a malicious text interception model based on group relative strategy optimization according to claim 3, characterized in that, In step S3, the rule reward is calculated by weighted summation, where the weight of each sub-item is configurable and the score of each sub-item is a binary score.

5. The method for training a malicious text interception model based on group relative strategy optimization according to claim 1, characterized in that, In step S3, the discriminator used for model discrimination reward is a large language model with a larger number of parameters than the target model. This discriminator scores the input text and thought chain content from three dimensions: logic, completeness and readability, and takes the average value as the reward.

6. The method for training a malicious text interception model based on group relative strategy optimization according to claim 1, characterized in that, In step S4, the specific steps for constructing the preference pair data include: For each training sample, generate slow thinking candidate outputs containing long thought chains and fast reasoning candidate outputs without thought chains. Calculate the reward value for each of the two candidate outputs; Calculate the payoff difference between slow-thinking candidates and fast-reasoning candidates; If the difference in returns is greater than the preset bias threshold, then the slow thinking candidate will be used as the preferred output and the fast reasoning candidate will be used as the non-preferred output. If the payoff difference is less than the negative bias threshold, then the fast reasoning candidate is taken as the preferred output and the slow thinking candidate is taken as the non-preferred output.

7. The method for training a malicious text interception model based on group relative strategy optimization according to claim 6, characterized in that, In step S5, the manual forced mode is achieved by attaching specific tags: attaching an end-of-thinking tag after the input text to force fast reasoning; attaching a start-of-thinking tag after the input text to force slow thinking and output the complete thought chain.

8. The method for training a malicious text interception model based on group relative strategy optimization according to claim 1, characterized in that, In step S1, the constructed dataset contains subclass labels for harmful categories, including at least one of the following: pornography, violence and threats, hate speech, illegal information, privacy violations, harassment and abuse, dangerous behavior inducement, and false information.

9. The method for training a malicious text interception model based on group relative strategy optimization according to claim 1, characterized in that, In step S2, the thought chain and conclusion of the candidate output are organized according to a preset format: the thought chain content is located between a pair of start and end thinking labels, and the final conclusion is located after the end thinking label.

Images