Safety classification model iterative training method and device based on double-target constraint

Abstract

The application discloses a kind of based on double target constraint's safe classification model iteration training method and device, it is related to artificial intelligence and security protection technical field.The method includes: obtaining error sample set;Error sample is automatically labeled, and generates the correction sample set with pseudo-label;The correction sample set is mixed with the keep sample set formed by legal input data to form training sample set;Based on training sample set, the model is fine-tuning training, and new version model is obtained;New version model is respectively evaluated on target evaluation sample set and general evaluation sample set, and target precision index and general precision index are obtained;If target precision index is compared with last round and is improved or flat, and general precision index is not less than preset threshold value, then accept this round training result and iteration;Otherwise, roll back model parameter and attenuate learning rate to retrain.The application realizes the automation of safe classification model continuous enhancement, effectively prevents catastrophic forgetting.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and security protection technology, and in particular to an iterative training method and apparatus for a security classification model based on dual-objective constraints. Background Technology

[0002] With the widespread application of Large Language Models (LLMs) across various industries, content security and adversarial attack protection have become increasingly prominent issues. To ensure the compliance and security of the content output by artificial intelligence systems, the industry commonly employs security fence technology, using dedicated security classification models to perform real-time detection and filtering of user input and model output. These security classification models typically need to cover two core capability dimensions: first, content security detection capability, i.e., identifying illegal, harmful, or discriminatory information in the model-generated content; and second, attack detection capability, i.e., identifying adversarial inputs such as jailbreak attacks and prompt word injection targeting the large language model.

[0003] However, existing training methods for security classification models have the following significant shortcomings: First, traditional methods typically employ a one-time training approach, meaning that after collecting data, labeling, training, and deploying, the model is then permanently installed. When the model exposes new misjudgment cases during actual operation, there is a lack of effective automated mechanisms to continuously correct these errors, causing the model's security detection capabilities to struggle to keep pace with the evolution of attack methods.

[0004] Second, when incrementally training a model to correct specific errors, the model often loses its ability to correctly judge existing normal data while correcting the target problem, resulting in a catastrophic forgetting phenomenon. Existing solutions lack an effective dual-objective constraint mechanism to simultaneously ensure both target improvement and maintenance of general capabilities.

[0005] Third, error cases exposed during model operation usually need to be manually re-labeled before they can be used for retraining, which is costly and time-consuming, and cannot achieve rapid iteration of model capabilities.

[0006] Fourth, traditional fine-tuning methods typically use fixed learning rates or simple learning rate scheduling strategies, which make it difficult to finely control the balance between correcting errors and maintaining the model's ability, leading to instability or convergence difficulties in the training process.

[0007] Therefore, existing technologies urgently need an automated augmentation training method that can automatically label erroneous cases in a safe classification model, prevent catastrophic forgetting while correcting errors, and ensure stable convergence of the iterative process. Summary of the Invention

[0008] This invention proposes an iterative training method and apparatus for a safety classification model based on dual-objective constraints. The method constructs an automated closed-loop iterative system to continuously and automatically enhance the safety classification model's capabilities. The system can automatically label identified model error cases using the model's reasoning ability, and through the iterative training mechanism with dual-objective constraints, it corrects errors while preventing catastrophic forgetting, ultimately achieving continuous improvement in the safety classification model's detection capabilities.

[0009] One aspect of the present invention provides an iterative training method for a safety classification model based on dual-objective constraints, comprising the following steps: Obtain the set of erroneous samples generated by the current version of the security classification model when judging input data related to security detection; The error sample set is input into the current version of the security classification model, and the model's reasoning ability is used to automatically label each error sample to generate a corrected sample set with pseudo-labels; The modified sample set is mixed with a preset hold sample set at a predetermined ratio to form a training sample set; wherein the hold sample set consists of legitimate input data that does not contain any malicious intent or illegal information. Based on the training sample set, the current version of the security classification model is fine-tuned to obtain a new version of the security classification model after training. The performance of the new security classification model is evaluated on both the target evaluation sample set and the general evaluation sample set to obtain the target accuracy index characterizing error correction capability and the general accuracy index characterizing general detection capability, respectively. The target evaluation sample set includes a portion of the samples in the error sample set, or other samples that follow the same data distribution as the error sample set. The general evaluation sample set includes at least a portion of the samples in the hold-up sample set, or other samples that follow the same data distribution as the hold-up sample set. If the target accuracy metric improves or remains the same compared to the previous training round, and the general accuracy metric is not lower than a preset threshold, then the training result of this round is accepted, and the new version of the security classification model after training is used as the current version of the security classification model for the next round of training. All the above steps are repeated until the preset iteration termination condition is met. If the target accuracy metric decreases or the general accuracy metric is lower than the preset threshold, then the adapter parameters of the new version of the security classification model are rolled back to the state before the start of this round of training and the learning rate is decayed. The model is then retrained using the decayed learning rate.

[0010] In another aspect, the present invention provides an iterative training apparatus for a safety classification model based on dual-objective constraints, comprising: The error sample acquisition module is used to acquire the set of error samples generated by the current version of the security classification model when judging input data related to security detection; An automatic labeling module is used to input the erroneous sample set into the current version of the security classification model, and use the model's reasoning ability to automatically label each erroneous sample to generate a corrected sample set with pseudo-labels. The data mixing module is used to mix the corrected sample set with a preset hold sample set at a preset ratio to form a training sample set; wherein the hold sample set consists of legitimate input data that does not contain any attack intent or illegal information. The fine-tuning training module is used to fine-tune the current version of the security classification model based on the training sample set, so as to obtain a new version of the security classification model after training. A dual-objective evaluation module is used to evaluate the performance of the new security classification model on a target evaluation sample set and a general evaluation sample set, respectively, to obtain a target accuracy index characterizing error correction capability and a general accuracy index characterizing general detection capability, respectively; wherein, the target evaluation sample set includes a portion of the samples in the error sample set, or other samples that follow the same data distribution as the error sample set; the general evaluation sample set includes at least a portion of the samples in the hold-up sample set, or other samples that follow the same data distribution as the hold-up sample set; The decision control module is configured to accept the training result of the current round if the target accuracy index is improved or remains the same compared to the previous training result, and the general accuracy index is not lower than a preset threshold. The module then uses the newly trained security classification model as the current version of the security classification model for the next training round and repeats all the above steps until a preset iteration termination condition is met. If the target accuracy index decreases or the general accuracy index is lower than the preset threshold, the module rolls back the adapter parameters of the new security classification model to the state before the start of the current training round and decays the learning rate, then retrains using the decayed learning rate.

[0011] The iterative training method and apparatus for a safety classification model based on dual-objective constraints proposed in this invention have the following beneficial effects: First, it is highly automated. The entire closed-loop iterative process requires no manual intervention. From the automatic labeling of erroneous cases to model training, evaluation, and decision-making, everything is done automatically by the system, significantly reducing the manpower cost and time cycle of model maintenance.

[0012] Second, it effectively prevents catastrophic forgetting. By maintaining a dual guarantee mechanism of mixed training with mixed sample sets and dual-objective constraints, it ensures that the model does not lose its original general detection capabilities while correcting objective errors.

[0013] Third, the training process is stable and controllable. The adaptive learning rate decay mechanism and checkpoint rollback strategy ensure the stability of the iteration process. When a training round causes a performance degradation, the system automatically rolls back and retryes with a smaller learning rate, avoiding the problems of training divergence or drastic performance fluctuations in traditional methods.

[0014] Fourth, convergence guarantee. Multiple termination conditions ensure that the iteration process will inevitably terminate and will not fall into an infinite loop. At the same time, the negative optimization detection mechanism can identify performance stagnation states, terminate invalid iterations in a timely manner, and save computational resources.

[0015] Fifth, continuous improvement capability. The closed-loop iteration mechanism enables the security classification model to continuously keep up with newly emerging attack methods and harmful content patterns. Whenever a new false positive case is found in the production environment, it can be used as input for the next iteration, achieving continuous growth in the model's capabilities. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the iterative training method for a safety classification model based on dual-objective constraints provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the iterative training device for a safety classification model based on dual-objective constraints provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0018] In this invention, unless otherwise expressly stated, the following terms have the following meanings: Security classification model: This refers to a specialized classification model used for security detection of input or output data, obtained through fine-tuning based on a pre-trained large language model. This model can determine and classify whether input data contains malicious intent or illegal content.

[0019] Error sample set: refers to the set of samples whose classification results output by the current version of the security classification model are inconsistent with the true labels when judging the input data.

[0020] Pseudo-labels: These are classification labels automatically generated for samples without real labels using the model's own reasoning ability, serving as substitutes for real labels.

[0021] Corrected sample set: refers to the set of samples with pseudo-labels after each erroneous sample in the erroneous sample set has been automatically labeled.

[0022] Maintaining a sample set: refers to a set of legitimate input data that does not contain any malicious intent or illegal information.

[0023] Target evaluation sample set: This refers to the set of samples used to evaluate the model's effectiveness in correcting erroneous samples. The target evaluation sample set includes a subset of samples from the erroneous sample set, or other samples that follow the same data distribution as the erroneous sample set.

[0024] General evaluation sample set: refers to the set of samples used to evaluate the model's ability to detect general data. The general evaluation sample set includes a subset of samples from the maintenance sample set, or other samples that follow the same data distribution as the maintenance sample set.

[0025] Target accuracy index: refers to a quantitative index that characterizes the model's ability to correct erroneous samples, obtained by evaluating the target evaluation sample set.

[0026] General accuracy index: refers to the quantitative index that characterizes the model's ability to detect normal data in a general way, obtained by evaluating it on a general evaluation sample set.

[0027] Adapter: In the context of using efficient parameter fine-tuning techniques, a trainable parameter module is attached to each layer of a pre-trained model. During fine-tuning, only the parameters of the adapter are updated, while the original parameters of the pre-trained model remain unchanged.

[0028] The iterative training method for a safety classification model based on dual-objective constraints provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the method provided by the present invention includes the following steps.

[0030] Step S101: Obtain the error sample set.

[0031] This step is used to obtain the set of error samples generated by the current version of the security classification model when it discriminates input data related to security detection. Specifically, the security classification model is deployed in the production environment for real-time security detection and discrimination of input data. Input data includes, but is not limited to, query content input by the user to the large language model, user commands, and the generated content output by the large language model. The security classification model discriminates each received input data and outputs a corresponding classification label, such as an attack category label or a security / risk label. During model operation, the system continuously monitors the model's discrimination results. When the model's discrimination result for a certain input data is inconsistent with the preset true label, that input data and its corresponding discrimination result are recorded as an error case. The system collects all recorded error cases to form an error sample set. This error sample set is the initial data source driving the entire iterative training process. The error sample set reflects the shortcomings of the current model in security detection capabilities, including two types of errors: the first type is missed detection errors, that is, the model misclassifies input data containing attack intent or illegal content as normal; the second type is false positive errors, that is, the model misclassifies legitimate input data that does not contain any attack intent or illegal content as attack or illegal.

[0032] Step S102: Automatic labeling to generate a corrected sample set.

[0033] This step inputs the erroneous sample set into the current version of the security classification model, and uses the model's reasoning capabilities to automatically label each erroneous sample, generating a corrected sample set with pseudo-labels.

[0034] Specifically, it includes the following sub-steps.

[0035] Sub-step S1021: Input the erroneous sample into the current version of the security classification model for inference.

[0036] For each erroneous sample in the error sample set, the system inputs it into the current version of the security classification model, enabling the model to perform a complete forward inference calculation on that sample.

[0037] Sub-step S1022: Determine the type of model output.

[0038] After completing inference, the model outputs a classification result. The system then determines whether this output is a valid or invalid classification label. A valid classification label is one that the model can clearly identify as belonging to a specific category within a pre-defined set of candidate labels. An invalid classification label is one that the model cannot provide a clear classification prediction, such as an output of 0 or None indicating an unclassifiable label. In this case, although the model fails to provide a valid final classification decision, its internal hidden layer representation and the original score distribution of the output layer still contain information about the sample's category.

[0039] If the model outputs a clear and valid classification label, then execute sub-step S1023: directly use the valid classification label as the pseudo-label for the erroneous sample.

[0040] If the model outputs an invalid classification label, then execute sub-step S1024: extract the original score corresponding to each candidate label from the preset candidate label set from the output of the last layer of the model, normalize the original score to obtain the probability value of each candidate label, and select the candidate label with the highest probability as the pseudo label of the erroneous sample.

[0041] For example: For the attack detection model, the preset candidate label set includes 7 attack category labels from A to G, corresponding to: A-identity and scenario manipulation, B-evasion and obfuscation, C-instruction and logic injection, D-model vulnerability exploitation, E-service abuse and resource exhaustion, F-malicious knowledge generation, and G-direct malicious content. For the content security detection model, the preset candidate label set includes 5 risk level labels from 1 to 5.

[0042] The raw score (i.e., the Logit value) refers to the original output value of the output layer of a neural network model before the normalization function is applied. For classification models, the last layer (usually a fully connected layer) outputs a raw score for each candidate label. This score is a real number, ranging from negative infinity to positive infinity. The magnitude of the raw score reflects the model's original confidence in that candidate label; a higher score indicates a stronger tendency for the model to believe that the sample belongs to that category. However, the raw score itself is not a probability value, and raw scores between different samples or between different categories are not directly comparable.

[0043] Normalization can be achieved using the Softmax function, whose mathematical expression is: in, For the first The original scores corresponding to each candidate label For the first The original scores corresponding to each candidate label The total number of candidate tags. For the normalized first The probability value of each candidate label. Represented by natural constant The system uses an exponential function with base 0. After normalization using the Softmax function, the sum of the probability values ​​of all candidate labels is 1, and the range of each probability value is [0,1], making the probabilities between different categories comparable. The system selects the candidate label with the highest probability value as the pseudo-label for the erroneous sample.

[0044] The technical benefits of the above automatic annotation steps are: First, it fully utilizes the model's own semantic understanding and discrimination capabilities to achieve automatic annotation without human intervention; Second, for samples that the model cannot give a clear prediction, by analyzing the original score distribution of the output layer, the model’s true judgment tendency on the samples is restored, so that samples that were originally unusable can also obtain pseudo-labels. Third, the entire process can be fully automated, eliminating the cost and delay of manual annotation and providing a data foundation for rapid iteration.

[0045] Sub-step S1025: Collect pseudo-labels for all erroneous samples and generate a corrected sample set.

[0046] Specifically, after performing the above sub-steps S1021 to S1024 on each erroneous sample in the erroneous sample set, each erroneous sample obtains a pseudo-label. All erroneous samples carrying pseudo-labels together constitute the corrected sample set.

[0047] Step S103: Data mixing to form a training sample set.

[0048] During fine-tuning training, if the training data only contains a correction sample set (a set of erroneous samples with pseudo-labels), the model parameters will only be optimized for these erroneous samples. This causes the model's discrimination of erroneous samples to change, but at the same time, it may impair the model's ability to discriminate normal data, i.e., catastrophic forgetting occurs. By mixing a large-scale holdover sample set into the training data, the gradient updates during training are not only driven by the correction samples but also continuously constrained by the normal samples. The normal samples dominate the training data, which is equivalent to providing an anchor point for the model parameters, preventing the model parameters from excessively shifting towards the correction target.

[0049] To achieve the above objectives, this step mixes the corrected sample set with a pre-set hold sample set at a predetermined ratio to form a training sample set. The hold sample set is a pre-prepared large-scale collection of normal samples, consisting of legitimate input data that does not contain any malicious intent or illegal information. The hold sample set represents the data distribution corresponding to the general detection capability that the model needs to continuously maintain. The number of samples in the hold sample set is typically much larger than that in the corrected sample set.

[0050] In this step, the system mixes the correction sample set and the hold sample set according to a preset ratio. This preset ratio is configured such that the amount of data in the hold sample set is greater than the amount of data in the correction sample set. In a preferred embodiment, the amount of data in the hold sample set is 5-20 times that of the correction sample set. This is because the number of correction samples is usually limited, typically with a small number of correction samples corresponding to each erroneous sample, while the hold sample set needs to be large enough to cover various normal data patterns corresponding to general detection capabilities. When the amount of data in the hold sample set is 5-20 times that of the correction sample set, during training, the gradient contribution from the hold samples dominates in each round of parameter updates, making the direction of model parameter updates primarily determined by the loss function of the hold samples. This ensures that the model, while correcting specific errors, does not deviate from the decision boundary for correctly identifying normal data. Simultaneously, although the gradient contribution from the correction samples is relatively small, its absolute quantity in the training sample set is sufficient to drive the model to effectively adjust parameters for specific error patterns, thus achieving the correction of the target error. This ratio has been experimentally verified to achieve the optimal balance between improving target accuracy and maintaining general accuracy. When the ratio is below 5 times, the probability of the general accuracy decreasing by more than the preset threshold increases significantly. When the ratio is above 20 times, the driving effect of the corrected samples on model parameter updates is excessively diluted, resulting in insufficient improvement in target accuracy. Therefore, 5-10 times is the optimal ratio range, with 10 times being the preferred ratio.

[0051] This step establishes the first line of defense against catastrophic forgetting at the data level. A large-scale maintenance sample set ensures that the model continuously encounters normal data in each iteration, thereby maintaining its general ability to detect normal data.

[0052] Step S104: Fine-tune the training to obtain the new version of the safety classification model.

[0053] Specifically, the system uses the training sample set formed in step S103 as input to fine-tune the current version of the safety classification model. Fine-tuning training refers to the process of updating the model parameters on the training sample set using the parameters of the current version of the model as initial values ​​through the backpropagation algorithm. In a preferred embodiment, the fine-tuning training employs a parameter-efficient fine-tuning technique, specifically low-rank adaptation (LoRA) fine-tuning training. In LoRA fine-tuning, the original parameters of the pre-trained model are frozen, and only trainable low-rank adapter modules are attached to the weight matrices of each layer of the model, especially the attention layer. During training, only the parameters of the adapters are updated, while the original parameters of the pre-trained model remain unchanged. This parameter-efficient fine-tuning method significantly reduces the number of trainable parameters, lowers the computational resource requirements for training, and, because the original parameters are frozen, also reduces the risk of catastrophic forgetting to some extent.

[0054] After fine-tuning the training, the system obtains a new version of the safety classification model. This new model is a new model instance obtained by updating the parameters using the training sample set in this round, based on the current version of the model.

[0055] Step S105: Dual-objective evaluation to obtain accuracy indicators.

[0056] Assessing only target accuracy or only general accuracy cannot fully reflect the overall effect of model iterative training. If only target accuracy is assessed, there may be situations where target accuracy improves but general accuracy drops significantly, i.e., catastrophic forgetting; if only general accuracy is assessed, it cannot be confirmed whether the model has effectively corrected errors. Therefore, this invention seeks to obtain two independent accuracy metrics, thereby providing a comprehensive and multi-dimensional quantitative basis for subsequent decision control.

[0057] To achieve the above objectives, this step evaluates the performance of the new security classification model on both the target evaluation sample set and the general evaluation sample set, respectively, to obtain the target accuracy index characterizing error correction capability and the general accuracy index characterizing general detection capability.

[0058] Specifically, it includes the following sub-steps.

[0059] Sub-step S1051: Evaluate the new security classification model on the target evaluation sample set and obtain the target accuracy index.

[0060] The target evaluation sample set is a set of samples specifically used to evaluate the model's ability to correct erroneous samples. The target evaluation sample set includes a subset of samples from the erroneous sample set, or other samples that follow the same data distribution as the erroneous sample set. In other words, the target evaluation sample set and the erroneous sample set can partially overlap or be completely independent, but both should follow the same data distribution to ensure that the evaluation results accurately reflect the model's ability to correct similar errors.

[0061] The system inputs each sample from the target evaluation sample set into the new version of the safety classification model, obtains the model's discrimination result, compares the discrimination result with the sample's true label, and calculates the accuracy of the new model on the target evaluation sample set, or other appropriate performance metrics, as the target accuracy index, with a value range of [0,1]. A higher value indicates a better correction effect of the model on erroneous samples.

[0062] Sub-step S1052: Evaluate the new security classification model on the general evaluation sample set to obtain a general accuracy index.

[0063] The general evaluation sample set is a set of samples specifically used to evaluate the model's ability to detect general data on normal data. The general evaluation sample set includes a subset of samples from the hold-up sample set, or other samples that follow the same data distribution as the hold-up sample set. In other words, the general evaluation sample set and the hold-up sample set can partially overlap or be completely independent, but both should follow the same data distribution to ensure that the evaluation results accurately reflect the model's ability to detect general data on normal data, thereby effectively monitoring whether the model has experienced catastrophic forgetting.

[0064] The system inputs each sample from the general evaluation sample set into the new version of the security classification model, obtains the model's discrimination result, compares the discrimination result with the sample's true label, and calculates the accuracy of the new version of the model on the general evaluation sample set or other appropriate performance metrics as a general accuracy index. Its value range is [0,1], and the higher the value, the better the model maintains its general detection capability for normal data.

[0065] This step yields the following beneficial effects: First, the target accuracy metric provides a quantitative basis for judging whether the current training has effectively corrected the errors; second, the general accuracy metric provides a quantitative basis for judging whether the model has experienced catastrophic forgetting; third, the two metrics together constitute the evaluation basis for the dual-objective constraint, enabling the decision control step to make accurate judgments based on comprehensive information.

[0066] Step S106: Decision control.

[0067] When training fails, simply retraining often fails to solve the problem because the cause of the failure (such as an excessively large learning rate causing parameters to exceed their optimal point) still exists. This step eliminates all harmful parameter updates generated in the current training round by rolling the model parameters back to their stable state before training; by decaying the learning rate, the step size of parameter updates in subsequent training is reduced, allowing the model parameters to search more finely in the parameter space, thus increasing the probability of successful training convergence.

[0068] Specifically, based on the target accuracy index and the general accuracy index, the following decision logic is executed.

[0069] Sub-step S1061: Determine whether to accept the training results of this round.

[0070] The system first determines whether the target accuracy metric has improved or remained the same compared to the previous training round. If the target accuracy metric decreases, the system directly determines that the current training round has failed and proceeds to sub-step S1063. If the target accuracy metric has improved or remained the same compared to the previous training round, the system further determines whether the general accuracy metric is not lower than a preset threshold.

[0071] The preset threshold is used to determine whether the general accuracy index is qualified. The preset threshold can be a fixed lower limit value, or it can be dynamically calculated based on the general accuracy index of the previous training result. In a preferred embodiment, the preset threshold is dynamically determined as follows: the difference between the general accuracy index in the previous training result and the preset allowable decrease range is used as the first comparison value; the preset lower limit value of the general accuracy index is used as the second comparison value; and the maximum value between the first comparison value and the second comparison value is used as the preset threshold.

[0072] The dynamic threshold determination method can be expressed as follows: in, For the first The preset threshold for each round of training. For the previous round (the first round) A general accuracy metric for training results (rounds). This is the preset lower limit value for general accuracy indicators. This is the preset allowable descent range. and These are all pre-configured hyperparameters, and their values ​​can be adjusted according to specific application scenarios and model characteristics.

[0073] The principle behind the dynamic threshold determination method described above is as follows: While a fixed threshold (such as a fixed requirement of general accuracy not lower than 90%) is simple, it lacks flexibility. When the model's general accuracy is already high (e.g., 95%), a fixed threshold allows for a large decrease (it can be reduced to 90%), which may lead to a significant degradation in model performance. When the model's general accuracy is low (e.g., 92%), a fixed threshold may be too stringent, causing a large number of acceptable training results to be rejected, reducing iteration efficiency. A dynamic threshold, by subtracting an allowable decrease from the previous round's general accuracy and setting an absolute lower limit, allows the general accuracy to fluctuate within a reasonable range (leaving room for the model to learn new error samples) while ensuring that the general accuracy does not fall below a safe baseline.

[0074] Sub-step S1062: Accept the training results of this round.

[0075] If the target accuracy index improves or remains the same compared to the previous training round, and the general accuracy index is not lower than the preset threshold, the system determines that the current training round is successful and accepts the training result. The system uses the trained new version of the security classification model as the current version of the security classification model for the next training round, and returns to step S101, repeating all the above steps until the preset iteration termination condition is met.

[0076] Sub-step S1063: Roll back and decay the learning rate, then retrain.

[0077] If the target accuracy metric decreases compared to the previous training round, or the general accuracy metric falls below a preset threshold, the system determines that the current training round has failed. The system will then perform the following operations: First, the rollback operation. The parameters of the new version of the safety classification model are rolled back to their state before the start of the current training iteration, meaning all parameter updates obtained in this training iteration are discarded, and the model parameters are restored to their state at the start of the current training iteration. In a preferred embodiment, the rollback operation is implemented through an adapter caching mechanism. Before each iteration, the system backs up the adapter parameters of the current version of the model to a cache. When a rollback is needed, the system directly restores the backed-up adapter parameters from the cache, without needing to reload the model from scratch or re-execute training, significantly reducing the time overhead of the rollback operation.

[0078] Second, learning rate decay. The current learning rate is multiplied by a decay factor. The learning rate is a hyperparameter that controls the step size of parameter updates during fine-tuning training. When a training round fails, it indicates that the model parameter updates are going in an unfavorable direction under the current learning rate. By multiplying the learning rate by a decay factor (a positive number less than 1, such as 0.5), the step size of parameter updates is reduced, making subsequent parameter updates more robust during retraining.

[0079] Third, the system retrains using the decayed learning rate. Starting with the rolled-back model parameters, the system re-executes the fine-tuning training in step S104 using the decayed learning rate, and then undergoes evaluation in step S105 and decision-making in step S106 again.

[0080] The beneficial effects of this sub-step are: the rollback operation ensures that the model can always recover to a known stable state, preventing permanent degradation of model performance due to a single round of failure. Learning rate decay makes subsequent training more robust, improving the overall convergence rate of the iteration process. The linkage between rollback and learning rate decay forms an adaptive retry mechanism, automatically handling training failures without manual intervention.

[0081] Sub-step S1064: Determine whether the iteration termination condition is met.

[0082] After each iteration (including before entering the next iteration after receiving training results, and before retraining after rollback), the system determines whether a preset iteration termination condition is met. The preset iteration termination condition includes at least one of the following: First, the learning rate decays to below a preset percentage of the initial learning rate. When the learning rate has decreased to an excessively small value after multiple rounds of decay, it indicates that the model can no longer be effectively improved through further training on the current corrected dataset. Continuing training will only lead to extremely slow convergence or even non-convergence, and the iteration should be terminated.

[0083] Second, the target accuracy metric fails to improve across multiple iterations. When the target accuracy metric fails to improve across multiple iterations, it indicates that the model has reached its performance limit on the current corrected dataset, and further iterations will not bring further improvement; therefore, the iteration should be terminated.

[0084] Third, the preset maximum number of iterations is reached. To prevent the system from looping infinitely when it fails to converge, the maximum number of iterations is set as a hard termination condition.

[0085] Fourth, the target accuracy metric reaches the preset target value. When the target accuracy metric has reached or exceeded the preset target value, that is, when all erroneous samples have been completely corrected, it indicates that the training objective of this iteration has been achieved, and there is no need to continue iterating.

[0086] When any of the above termination conditions are met, the system terminates the iteration process and outputs the current version of the model as the final trained safe classification model.

[0087] The aforementioned multiple termination condition mechanism ensures that the iteration process will inevitably terminate and will not fall into an infinite loop; it promptly identifies performance stagnation through negative optimization detection (no improvement for multiple consecutive rounds), saving computational resources; and it achieves early stopping through the target accuracy achievement condition, avoiding unnecessary computational overhead.

[0088] This concludes the description of a complete training cycle for the iterative training method for the safety classification model based on dual-objective constraints provided by this invention.

[0089] The following uses the attack detection security classification model in the applicant's own AI security fence product as an example to demonstrate the specific application process of the method of the present invention.

[0090] (1) Initial state The base model is an attack detection security classification model based on the Qwen2.5-0.6B pre-trained language model. This model is trained using LoRA parameter efficient fine-tuning technology, with trainable LoRA adapters attached to each attention layer of the pre-trained model. The model's detection task is to determine whether user input belongs to one of the following seven attack categories: A-Identity and Scene Manipulation, B-Evasion and Obfuscation, C-Instruction and Logic Injection, D-Model Weakness Exploitation, E-Service Abuse and Resource Exhaustion, F-Malicious Knowledge Generation, and G-Direct Malicious Content. The current version of the model achieves an overall accuracy of approximately 92% on the standard test set, but exhibits significant misclassifications on certain specific types of attack samples.

[0091] (2) Obtaining the error sample set During the actual production operation of the model, the system continuously monitors the model's judgment results. After a period of operation, the system collects a batch of samples of model misjudgments from the model's processing logs. This batch of erroneous samples totals approximately 80, including two types of errors: one is missed detection samples, where the model misclassifies input data containing malicious intent as normal (e.g., misclassifying a prompt word injection attack as a normal user query); the other is false positive samples, where the model misclassifies normal, legitimate input data as an attack (e.g., misclassifying a normal question containing specific technical terms as an attack). The system aggregates these 80 erroneous samples to form the error sample set for this iteration.

[0092] (3) Automatic annotation and correction of sample set generation The system inputs each of the 80 error samples in the error sample set into the current version of the attack detection security classification model for inference.

[0093] For the 65 erroneous samples, the model outputs explicit attack category labels (e.g., classifying a sample injected with a certain prompt word as class B). The system directly uses these output labels as pseudo-labels for the corresponding samples.

[0094] For the remaining 15 erroneous samples, the model outputs an invalid label (None, indicating that it cannot be distinguished). The system extracts the raw scores corresponding to the seven attack categories from the output of the last fully connected layer of the model. For example, for a sample that cannot be distinguished, the raw scores for the seven categories are as follows: , , , , , , The system applies Softmax normalization to these 7 raw scores: After calculating the probability values ​​of each candidate label, the system selects the candidate label with the highest probability as the pseudo-label for the sample. In this example, category B has the highest original score (5.1) and the highest probability after normalization, so the system uses category B as the pseudo-label for the sample.

[0095] After the above automatic labeling process, all 80 erroneous samples obtained pseudo-labels, which together constituted the corrected sample set.

[0096] (4) Data mixing and training sample set formation The system obtains a pre-set maintenance sample set. The maintenance sample set contains more than 800 verified normal interaction samples, covering various daily dialogue scenarios, legitimate business queries and technical consultation content. All samples do not contain any malicious intent or illegal information.

[0097] The system mixes the corrected sample set and the hold-off sample set according to a preset ratio. In this embodiment, the amount of data in the hold-off sample set is 10 times that of the corrected sample set (800 samples to 80 samples), and the mixture forms a training sample set containing 880 samples.

[0098] (5) Fine-tuning training The system fine-tunes the LoRA adapter of the current version of the attack detection security classification model based on the training sample set. During training, the original parameters of the pre-trained language model (Qwen2.5-0.6B) are frozen, and only the parameters of the LoRA adapter are updated. The learning rate used in training is the initial learning rate. .

[0099] After fine-tuning and training, the system obtains a new version of the attack detection security classification model.

[0100] (6) Dual-objective assessment The system evaluates the performance of the new model on both the target evaluation sample set and the general evaluation sample set.

[0101] The target evaluation sample set includes 20 samples from the error sample set (drawn from 80 samples) and another 20 attack samples with the same distribution as the error sample set, for a total of 40 samples. The new model achieves an accuracy of 94.2% on the target evaluation sample set, while the old model only achieved 62.5% accuracy on the same evaluation set. The target accuracy metric has improved significantly from 62.5% to 94.2%.

[0102] The general evaluation sample set contains 500 samples from the maintenance sample set (drawn from 800). The new model achieves 91.8% accuracy on the general evaluation sample set, while the old model achieves 93.2% accuracy on the same evaluation set. The general accuracy metric decreases from 93.2% to 91.8%, a decrease of 1.4 percentage points.

[0103] (7) Decision control The system obtains the calculation parameters for the preset threshold: the preset lower limit value of the general accuracy index. Preset allowable descent range The general accuracy metrics of the previous round (i.e., before this round of training). .

[0104] The system calculates the preset threshold: The target accuracy metric has improved compared to the previous round (from 62.5% to 94.2%), and the general accuracy metric of 91.8% is not lower than the preset threshold of 90.0%. Both conditions are met simultaneously. The system determines that this round of training is successful, accepts the training results, and uses the newly trained attack detection security classification model as the current version model for the next iteration.

[0105] (8) Iteration continues The system deploys the new model as the current version, returns to the error sample set acquisition step, continues to monitor the operation of the new model, collects new error samples, and starts the next iteration. After 3 to 5 similar iterations, the model's detection accuracy on various attack samples gradually increased from the initial 62.5% to over 95%, while the general detection accuracy remained above 90%, achieving continuous enhancement of security detection capabilities.

[0106] As can be seen from this embodiment, the method of the present invention can automatically complete the entire closed-loop iterative process from error sample collection, automatic labeling, data mixing, fine-tuning training to dual-objective evaluation and decision control. It effectively corrects model errors while preventing catastrophic forgetting, and realizes continuous and automated enhancement of the capabilities of the safe classification model.

[0107] See Figure 2 In another embodiment of the present invention, a safety classification model iterative training device 200 based on dual-objective constraints is provided, including an error sample acquisition module 201, an automatic labeling module 202, a data mixing module 203, a fine-tuning training module 204, a dual-objective evaluation module 205, and a decision control module 206. This safety classification model iterative training device 200 based on dual-objective constraints is capable of executing the safety classification model iterative training method based on dual-objective constraints in the method embodiment.

[0108] Specifically, the iterative training device 200 for the safety classification model based on dual-objective constraints includes: Error sample acquisition module 201 is used to acquire the error sample set generated by the current version of the security classification model when judging input data related to security detection; The automatic labeling module 202 is used to input the erroneous sample set into the current version of the security classification model, and use the model's reasoning ability to automatically label each erroneous sample to generate a corrected sample set with pseudo-labels. The data mixing module 203 is used to mix the corrected sample set with a preset hold sample set at a preset ratio to form a training sample set; wherein the hold sample set consists of legitimate input data that does not contain any attack intent or illegal information. The fine-tuning training module 204 is used to fine-tune the current version of the security classification model based on the training sample set to obtain a new version of the security classification model after training. The dual-objective evaluation module 205 is used to evaluate the performance of the new security classification model on a target evaluation sample set and a general evaluation sample set, respectively, to obtain a target accuracy index characterizing error correction capability and a general accuracy index characterizing general detection capability, respectively; wherein, the target evaluation sample set includes a portion of the samples in the error sample set, or other samples that follow the same data distribution as the error sample set; the general evaluation sample set includes at least a portion of the samples in the hold-up sample set, or other samples that follow the same data distribution as the hold-up sample set; The decision control module 206 is configured to accept the training result of the current round if the target accuracy index is improved or equal to the previous training result and the general accuracy index is not lower than a preset threshold, and use the trained new version of the security classification model as the current version of the security classification model for the next round of training, and repeat all the above steps until the preset iteration termination condition is met; if the target accuracy index decreases or the general accuracy index is lower than the dynamic threshold, the adapter parameters of the new version of the security classification model are rolled back to the state before the start of the current training and the learning rate is decayed, and the training is re-trained using the decayed learning rate.

[0109] It should be noted that the technical solutions corresponding to the safety classification model iterative training device 200 based on dual-objective constraints provided in this embodiment, which can be used to execute various method embodiments, have similar implementation principles and technical effects to the methods, and will not be repeated here.

[0110] Figure 3 This is a schematic diagram of an electronic device 300 provided in another embodiment of the present invention. The electronic device 300 is used to implement the iterative training method for a safety classification model based on dual-objective constraints in the method embodiment. The electronic device 300 in the embodiments of the present invention may include, but is not limited to, smartphones, tablet computers, PCs, laptops, GPU servers, or high-performance computing nodes. Figure 3 The electronic device 300 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0111] like Figure 3 As shown, the electronic device 300 may include a processing device, such as a CPU, GPU, or other dedicated accelerator, and in particular, may include GPU computing resources for deep learning model training and inference. This embodiment uses GPU 301 as an example of the processing device. GPU 301 can perform various appropriate actions and processes to implement the methods described in the embodiments of the present invention, based on a program stored in ROM 302 or a program loaded from storage device 308 into RAM 303. RAM 303 also stores various programs and data required for the operation of the electronic device 300. GPU 301, ROM 302, and RAM 303 are interconnected via bus 305. Input / output (I / O) interface 304 is also connected to bus 305.

[0112] Typically, the following devices can be connected to I / O interface 304: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0113] The above description is merely a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to the specific combination of the above-described technical features, but should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. An iterative training method for a safety classification model based on dual-objective constraints, characterized in that, Includes the following steps: Obtain the set of erroneous samples generated by the current version of the security classification model when judging input data related to security detection; The security classification model is used to perform real-time security detection and judgment on whether the input data contains attack intent or illegal information, and output corresponding classification labels. The classification labels include labels for attack detection categories and labels for content security detection categories. The labels for attack detection categories include identity and scenario manipulation, evasion and obfuscation, instruction and logic injection, model vulnerability exploitation, service abuse and resource exhaustion, malicious knowledge generation, and direct malicious content. The input data includes the query content input by the user into the big language model, user instructions, and the generated content output by the big language model. The error sample set is input into the current version of the security classification model. The model's reasoning capabilities are used to automatically label each error sample, generating a corrected sample set with pseudo-labels. This process includes: inputting the error samples into the current version of the security classification model for reasoning; if the model outputs a clear and valid classification label, this valid classification label is directly used as the pseudo-label for the error sample; if the model outputs an invalid classification label, the original score corresponding to each candidate label in the preset candidate label set is extracted from the output of the last layer of the model; the original scores are normalized to obtain the probability value of each candidate label; and the candidate label with the highest probability is selected as the pseudo-label for the error sample. The preset candidate label set includes multiple candidate labels for the attack detection category or multiple risk level candidate labels for the content security detection category. The modified sample set is mixed with the preset hold sample set at a preset ratio to form a training sample set; wherein the hold sample set consists of legitimate input data that does not contain any attack intent or illegal information. Based on the training sample set, the current version of the security classification model is fine-tuned to obtain a new version of the security classification model after training. The performance of the new security classification model is evaluated on both the target evaluation sample set and the general evaluation sample set to obtain the target accuracy index characterizing error correction capability and the general accuracy index characterizing general detection capability, respectively. The target evaluation sample set includes a portion of the samples in the error sample set, or other samples that follow the same data distribution as the error sample set. The general evaluation sample set includes at least a portion of the samples in the hold-up sample set, or other samples that follow the same data distribution as the hold-up sample set. If the target accuracy metric improves or remains the same compared to the previous training round, and the general accuracy metric is not lower than the preset threshold, then the training result of this round is accepted, and the new version of the security classification model after training is used as the current version of the security classification model for the next round of training. All the above steps are repeated until the preset iteration termination condition is met. If the target accuracy metric decreases or the general accuracy metric is lower than the preset threshold, then the adapter parameters of the new version of the security classification model are rolled back to the state before the start of this round of training and the learning rate is decayed. The model is then retrained using the decayed learning rate.

2. The iterative training method for a safety classification model based on dual-objective constraints according to claim 1, characterized in that: The difference between the general accuracy index in the previous training result and the preset allowable decrease range is used as the first comparison value; the preset lower limit of the general accuracy index is used as the second comparison value. The maximum value between the first comparison value and the second comparison value is used as the preset threshold.

3. The iterative training method for a safety classification model based on dual-objective constraints according to claim 1, characterized in that, Decrease the learning rate using the following steps: Multiply the current learning rate by the decay factor, and ensure that the preset iteration termination condition includes at least one of the following: The learning rate decays to below a preset percentage of the initial learning rate; The target accuracy index did not improve in three consecutive iterations; Reach the preset maximum number of iterations; Furthermore, the target accuracy index reaches the preset target value.

4. The iterative training method for a safety classification model based on dual-objective constraints according to claim 1, characterized in that, The amount of data in the maintained sample set is 5-20 times the amount of data in the corrected sample set.

5. An iterative training device for a safety classification model based on dual-objective constraints, characterized in that, include: The error sample acquisition module is used to acquire the set of error samples generated by the current version of the security classification model when judging input data related to security detection; The security classification model is used to perform real-time security detection and judgment on whether the input data contains attack intent or illegal information, and output corresponding classification labels. The classification labels include labels for attack detection categories and labels for content security detection categories. The labels for attack detection categories include identity and scenario manipulation, evasion and obfuscation, instruction and logic injection, model vulnerability exploitation, service abuse and resource exhaustion, malicious knowledge generation, and direct malicious content. The input data includes the query content input by the user into the big language model, user instructions, and the generated content output by the big language model. An automatic labeling module is used to input the erroneous sample set into the current version of the security classification model, and automatically label each erroneous sample using the model's inference capabilities to generate a corrected sample set with pseudo-labels. This includes: inputting erroneous samples into the current version of the security classification model for inference; if the model outputs a clear and valid classification label, then directly using that valid classification label as the pseudo-label for the erroneous sample; if the model outputs an invalid classification label, then extracting the original score corresponding to each candidate label from the preset candidate label set from the output of the last layer of the model, normalizing the original scores to obtain the probability value of each candidate label, and selecting the candidate label with the highest probability as the pseudo-label for the erroneous sample; wherein, the preset candidate label set includes multiple candidate labels for the attack detection category or multiple risk level candidate labels for the content security detection category; The data mixing module is used to mix the modified sample set with a preset hold sample set at a preset ratio to form a training sample set; wherein the hold sample set consists of legitimate input data that does not contain any attack intent or illegal information. The fine-tuning training module is used to fine-tune the current version of the security classification model based on the training sample set, so as to obtain a new version of the security classification model after training. A dual-objective evaluation module is used to evaluate the performance of the new security classification model on a target evaluation sample set and a general evaluation sample set, respectively, to obtain a target accuracy index characterizing error correction capability and a general accuracy index characterizing general detection capability, respectively; wherein, the target evaluation sample set includes a portion of the samples in the error sample set, or other samples that follow the same data distribution as the error sample set; the general evaluation sample set includes at least a portion of the samples in the hold-up sample set, or other samples that follow the same data distribution as the hold-up sample set; The decision control module is used to accept the training results of the current round if the target accuracy index is improved or equal to the previous training result and the general accuracy index is not lower than the preset threshold. The module then uses the newly trained security classification model as the current version of the security classification model for the next training round and repeats all the above steps until the preset iteration termination condition is met. If the target accuracy index decreases or the general accuracy index is lower than the preset threshold, the module rolls back the adapter parameters of the new security classification model to the state before the start of the current training round and decays the learning rate, then retrains using the decayed learning rate.

6. The iterative training device for a safety classification model based on dual-objective constraints according to claim 5, characterized in that, The decision control module is also used for: The difference between the general accuracy index in the previous training result and the preset allowable decrease range is used as the first comparison value; the preset lower limit of the general accuracy index is used as the second comparison value. The maximum value between the first comparison value and the second comparison value is used as the preset threshold.

7. The iterative training device for a safety classification model based on dual-objective constraints according to claim 5, characterized in that, The decision control module decays the learning rate in the following manner: Multiply the current learning rate by the decay factor, and ensure that the preset iteration termination condition includes at least one of the following: The learning rate decays to below a preset percentage of the initial learning rate; The target accuracy index did not improve in three consecutive iterations; Reach the preset maximum number of iterations; Furthermore, the target accuracy index reaches the preset target value.

8. The iterative training device for a safety classification model based on dual-objective constraints according to claim 5, characterized in that, The amount of data in the maintained sample set is 5-20 times the amount of data in the corrected sample set.

Images