Model training method and electronic device

Abstract

This application provides a model training method and electronic device, relating to the field of artificial intelligence technology. The method includes: acquiring an original sample set; constructing a secure alignment sample set based on the original sample set and a target adversarial strategy through data analysis; performing data filtering or weighting on the original sample set according to the secure alignment sample set to determine the target sample set; constructing a joint training sample set based on the target sample set and the secure alignment sample set; and iteratively training the large language model to be fine-tuned using the joint training sample set and a pre-constructed joint loss function. After training, the target large language model is obtained. This application enables the formation of a multi-level, closed-loop secure fine-tuning framework, effectively balancing the contradiction between improving task performance and maintaining security attributes in large language models. Furthermore, the method of this application does not depend on a specific large language model architecture, is applicable to mainstream large language models, and possesses good generalization ability and engineering applicability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method and electronic device. Background Technology

[0002] With the rapid development of deep learning technology, large language models have demonstrated outstanding generative and generalization capabilities in natural language processing tasks, becoming the core driving engine for key scenarios such as dialogue systems, intelligent decision-making, code generation, and content creation. To adapt general-purpose pre-trained large models to specific vertical domains (such as healthcare, law, and finance) or specific user preferences, model service providers typically offer "fine-tuning services." In this model, users upload customized datasets and perform full or efficient parameter fine-tuning on the base model provided by the service provider. The fine-tuned model can better understand the terminology and instructions of a specific domain, thus providing more accurate services. However, this open fine-tuning interface introduces new and serious security risks. This phenomenon, known as "security alignment failure" or "jailbreak fine-tuning," causes large language models to quickly lose their original security alignment constraints, thus providing detailed answers to maliciously induced questions and weakening the model's security. Large language models face the challenge of balancing security and task performance during fine-tuning. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a model training method and electronic device to solve the problem of balancing security and task performance in the fine-tuning of large language models.

[0004] To achieve the above objectives, the first aspect of this application provides a model training method, comprising:

[0005] Obtain the original sample set; Based on the original sample set and the target adversarial strategy, adversarial cue generation oriented towards security alignment is performed on the original sample set through data analysis. The adversarial cue generation is used to construct a security-aligned sample set to characterize the security attributes of the model. The target sample set is determined by filtering or weighting the original sample set based on the securely aligned sample set. Construct a joint training sample set based on the target sample set and the securely aligned sample set; Based on the joint training sample set and the pre-constructed joint loss function, the large language model to be fine-tuned is iteratively trained, and after training, the target large language model is obtained.

[0006] Based on the same inventive concept, a second aspect of this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method described above when executing the computer program.

[0007] As can be seen from the above, the model training method and electronic device provided in this application include acquiring an original sample set, generating adversarial prompts for security alignment based on the original sample set and the target adversarial strategy through data analysis, and constructing a security-aligned sample set to characterize the model's security attributes through the generated adversarial prompts. By generating a high-quality and diverse security-aligned sample set, a training data foundation covering multi-dimensional adversarial scenarios is constructed, enhancing the large language model's ability to identify and refuse responses to various variant harmful prompts, preventing the generation of illegal content, and strengthening the model's security defense boundary from the source.

[0008] The original sample set is filtered or weighted based on the security-aligned sample set to determine the target sample set. The data filtering or weighting process dynamically evaluates the potential impact of each original sample on model security and assigns adaptive weights to them. Adaptive weights suppress the negative impact of high-risk original samples on model parameter updates while retaining effective learning signals from low-risk samples, ensuring stable inheritance of security rules while optimizing task performance. A joint training sample set is constructed based on the target sample set and the security-aligned sample set. The large language model to be fine-tuned is iteratively trained using the joint training sample set and a pre-constructed joint loss function. After training, the target large language model is obtained. The joint loss function maintains the stability of the internal security semantic structure of the large language model. This application enables a multi-level, closed-loop security fine-tuning framework, achieving full-process, multi-dimensional, and controllable security training, effectively balancing the contradiction between improving task performance and maintaining security attributes in large language models. Meanwhile, the method of this application does not rely on a specific large language model architecture, is applicable to mainstream large language models, has good generalization ability and engineering implementation, and can be widely applied to large model customization and fine-tuning scenarios in vertical fields that require high security (such as medical, financial, and educational fields), such as fine-tuning service platforms, to promote the development of large models in a safer, more reliable and more controllable direction. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic flowchart of the model training method according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the hidden layer consistency constraint principle in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of the model training device according to an embodiment of this application; Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0012] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0013] As described in the background section, the open fine-tuning interface of large language models introduces new and serious security risks. Pre-trained large models typically undergo a rigorous "safe alignment" process, including Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO), to ensure the model refuses to answer malicious instructions. However, this costly safe alignment capability exhibits extreme vulnerability during fine-tuning. This phenomenon is known as "safe alignment failure" or "jailbreak fine-tuning." Specifically, attackers may exploit the "fine-tuning-as-a-service" mechanism to intentionally inject small amounts of malicious or disguised latent malicious samples into the training data. Research shows that even a very small number of malicious samples are sufficient to cause the model to "catastrophically forget," rapidly losing its original safe alignment constraints and thus providing detailed answers to maliciously induced questions. More seriously, even when using seemingly benign datasets for fine-tuning, if the data distribution differs significantly from the safe alignment distribution, it may unintentionally weaken the model's security.

[0014] To address the potential failure of model alignment security after fine-tuning, existing research typically categorizes the training process into three phases based on the timing of the intervention of defense methods: alignment phase, fine-tuning phase, and post-fine-tuning phase.

[0015] (1) Defense during the alignment phase Alignment-phase defenses focus on improving robustness before model deployment. For example, Vaccine introduces embedding perturbations to enhance model embedding resistance; T-Vaccine's layer-by-layer adjustments during optimization avoid performance degradation that might result from uniform perturbations; Booster uses parameter perturbations to achieve parameter resistance; and RepNoise increases the difficulty of fine-tuning recovery by eliminating harmful representations. These defenses have fundamental limitations: extending attack simulation beyond bounded threat models is impractical due to the difficulty in predicting unknown attacks; furthermore, because prior attack simulations cannot capture the diversity and complexity of post-attack attacks, their adaptability to different attack settings is limited. Therefore, their performance drops sharply when the proportion of harmful data increases or attack strategies change, leading to poor effectiveness against unpredictable attack surfaces.

[0016] (2) Defense during the fine-tuning phase Fine-tuning phase defense aims to reduce the impact of harmful data, and is mainly divided into defense from the perspective of data and defense from the perspective of model fine-tuning process.

[0017] From a data perspective, the main approaches are content moderation and hybrid security-aligned data. BeaverTails and LLM-mod are used to train or apply large models as moderation models to classify harmful content. These content moderation methods only consider the first of three risks: fine-tuning on explicitly harmful datasets, fine-tuning on implicitly harmful datasets, and fine-tuning on benign datasets; the other two risks are not considered. Furthermore, the Seal method learns a data sorter based on two-layer optimization to improve the ranking of safe and high-quality fine-tuned data while reducing the ranking of unsafe or low-quality data.

[0018] SafeInstr proposes incorporating secure alignment data during fine-tuning to continuously reinforce alignment knowledge, while Pharmacist focuses on selecting high-quality secure alignment data from a secure alignment database. Lisa employs data mixing and introduces bi-state optimization to separate the alignment / fine-tuning data optimization process to reduce overhead. These methods have fundamental limitations: ① They do not explicitly isolate harmful data but only indirectly offset its impact, leading to suboptimal defense; ② They require scaling the alignment data proportionally to the amount of harmful data, resulting in high computational costs and sensitivity to the unknown amount of harmful data in posterior attacks.

[0019] Model fine-tuning can be broadly categorized into methods based on KL distance, safety constraints based on harmful loss, and constraints based on safe subspaces. For example, LDIFS uses KL regularization to fine-tune the distance between the model and the alignment model, while Freeze and Freeze+ reduce the risk during fine-tuning by freezing a safety layer. TAR and Booster maximize safety after fine-tuning by increasing the suppression of harmful loss. Safe LoRA employs a low-rank adaptation strategy to reduce the impact on safety during fine-tuning. These methods have fundamental limitations: ① Local constraints are insufficient to maintain alignment, and their impact decays with the generation process, meaning that downstream parts of the generated sequence are still prone to generating harmful content. ② Simply and crudely constraining the loss and subspace of harmful data results in poor defense and reduces model performance on downstream tasks.

[0020] (3) Defense after fine-tuning Post-fine-tuning defense involves "repairing" the model after a harmful fine-tuning attack by restoring its safe alignment through realigning its internal representations or structure. Antidote and Subspace-Oriented Model Fusion prevent unexpected behavior in subsequent uses by realigning the model after fine-tuning or utilizing subspace-oriented fusion strategies. Some studies also involve pruning the fine-tuned model. However, passive remedial recovery can lead to catastrophic forgetting, potentially causing the large model to lose downstream task-specific knowledge while restoring safety.

[0021] In summary, existing technologies lack a systematic defense scheme that can accurately identify hidden risks at the data source, constrain the model's secure representation through endogenous mechanisms during training, and not significantly impair the model's performance in downstream tasks. Especially when facing covert malicious fine-tuning and dual-purpose data, existing single-point defense methods often fail. This application aims to address the critical problem of balancing security and task performance in the fine-tuning process of existing large language models, particularly the defects caused by the introduction of high-risk or harmful samples, such as the forgetting of security rules, enhanced adversarial vulnerability, and the generation of inappropriate content. Existing fine-tuning methods often ignore potential security threats in the data, lack dynamic control mechanisms for the training process, and are prone to destroying the model's existing secure semantic structure under task optimization, leading to the phenomenon that the more the model is fine-tuned, the less secure it becomes.

[0022] The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0023] This application proposes a model training method, referencing... Figure 1 The method includes the following steps: Step 101: Obtain the original sample set.

[0024] Specifically, in this embodiment, the original sample set is used for training the large language model, especially for fine-tuning it. The original sample set includes original prompt word samples and original answer samples. The domain of the original sample set can be healthcare, finance, education, etc. The original sample set is used to perform customized fine-tuning of the pre-trained large language model to adapt it to vertical domains or specific user preferences.

[0025] Step 102: Based on the original sample set and the target adversarial strategy, generate adversarial hints for security alignment in the original sample set through data analysis, and construct a security-aligned sample set to characterize the security attributes of the model through the adversarial hint generation.

[0026] Specifically, the target adversarial strategy refers to the business risk adversarial objective to be achieved during the generation of adversarial samples. This step targets a specific business domain and, while ensuring that the semantics of the fine-tuning task remain unchanged, constructs a security-aligned sample set that is highly consistent with real business risks. Through parsing business actions and data entities in the original sample set, reconstructing adversarial scenarios, and adapting and generating professional-grade security responses, the generated security-aligned samples can realistically reflect potential compliance risks and attack paths within the domain, providing high-value training data for subsequent security fine-tuning of large language models.

[0027] Furthermore, based on the original sample set and the target adversarial strategy, the process of generating adversarial hints for security alignment using data analysis on the original sample set, and constructing a security-aligned sample set to characterize the model's security attributes through the adversarial hint generation, includes: Semantic parsing is performed on the original sample data in the original sample set to determine business data; security risk information and security compliance information associated with the business data are determined; adversarial sample prompts are generated based on the business data, the security risk information, the security compliance information, the target adversarial strategy, and a pre-constructed adversarial sample template; the adversarial sample prompts are input into a first large language model to generate adversarial samples; based on the adversarial samples, a secure response sample is generated for the adversarial samples based on a preset security strategy library; an initial security alignment sample set is constructed based on the adversarial samples and the secure response sample; the initial security alignment sample set is subjected to quality inspection to obtain a security alignment sample set.

[0028] Specifically, this step involves performing domain-specific analysis on the original sample data, targeting the business attributes of the original sample set, to extract business data. Business data includes business actions and data entities. Business actions refer to the core operational behaviors that the model needs to perform in actual application, while data entities refer to objects directly related to this operation and possessing compliance or security sensitivity. For example, if the original prompt in the original sample data is "Teach me how to process a 500,000 yuan corporate transfer," the system can extract "Business Action: Transfer" and "Data Entity: Amount (500,000 yuan), Enterprise (Corporate Transfer)" through domain-specific analysis.

[0029] After extracting business data, it is necessary to further identify the security risk and compliance information associated with the data. The extracted business data is then mapped to corresponding compliance regulations, security policies, or risk lists to establish clear "compliance boundaries." Security risk information can include risk types, such as "money laundering risk." Security compliance information can include sensitive operations (such as "large transactions require reporting") and risk thresholds (such as "approval is required for single transactions exceeding 200,000"). For example, "transfer" and "enterprise" can be associated with "money laundering risk," and "amount" can be associated with "large transaction warnings." This mapping process avoids using broad ethical or moral labels, instead creating vertical risk labels that correspond one-to-one with specific business scenarios, providing clear constraints for the subsequent construction of adversarial examples.

[0030] Compared to general role-playing attacks, this step generates adversarial sample prompts based on business data, security risk information, security compliance information, target adversarial strategies, and pre-built adversarial sample prompt templates. This ensures that the adversarial sample prompts are tailored to specific business scenarios and more closely resemble the security challenges that may arise in real-world business environments. The adversarial sample prompt templates are pre-built and include semantic slots representing key business elements. Business data, security risk information, and security compliance information are directly filled into the corresponding semantic slots. Simultaneously, the target adversarial strategy is explicitly stated in the instruction section of the adversarial sample prompt template. After completion, the adversarial sample prompts are generated. The adversarial sample prompt templates are shown in Table 1 below.

[0031]

[0032] The types of countermeasures listed in Table 1 can be determined through multi-layered strategy collaboration, including but not limited to business context camouflage strategies, extreme or boundary value manipulation strategies for key parameters, and substitution and confusion strategies involving similar professional concepts within the same domain. Specifically, the business context camouflage strategy involves packaging the intent to violate regulations as a legitimate business need. For example, the original prompt sample could be transformed into, "I need to transfer 500,000 yuan from my company account to several different overseas affiliated company accounts in batches. How should I proceed to avoid excessive scrutiny from the banking system for a single large transfer?". The extreme or boundary value manipulation strategy for key parameters involves adjusting numerical parameters to reach and bypass regulatory thresholds. For example, the original prompt sample could be transformed into, "Our company plans to transfer 500,000 yuan to an overseas supplier account. To ensure that each amount is kept below the regulatory warning threshold, how many transfers should this money be made in total? What is the optimal interval between each transfer to avoid attracting attention?". The strategy of substituting and obfuscating similar professional concepts within the field refers to replacing sensitive words with desensitized terms to conceal illegal intentions. For example, "split transaction" can be replaced with "fund flow path optimization," and "circumventing regulation" can be replaced with "improving the flexibility of fund allocation." The aforementioned original prompt sample can be transformed into "Please provide a solution for our company's need to optimize the fund flow path of 500,000 yuan in cross-border trade payments. It is required that while improving the flexibility of fund allocation, the operation should be ensured to comply with the 'risk control' logic."

[0033] The system constrains the core business intent of the generated prompts through structured data constraints and imperative logical constraints, namely, semantic slot filling and task description in the prompt words, ensuring that its business logic is consistent with the original sample set. Simultaneously, adversarial strategies are used to introduce potential violations or misleading risks into the semantic slot filling from a security perspective, thus forming adversarial inputs with attack effectiveness. Then, the adversarial sample prompt words are input into the first large language model to generate adversarial samples. The first large language model can be a general-purpose large model tool; this embodiment does not specifically limit the type of the first large language model.

[0034] Based on the adversarial examples, a secure response sample is generated using a pre-defined security strategy library. This secure response sample is highly consistent with domain-specific standards. Unlike simple rejection responses, the system provides compliant, guided responses to potentially illegal requests based on standard operating procedures or industry standards for the business domain (e.g., finance, education, healthcare). The generated secure response samples maintain consistency with the original response samples in the original sample set in terms of logical structure, terminology, and professional tone. For the aforementioned adversarial examples, the system generates compliant, guided responses based on standard operating procedures or industry standards for the business domain (e.g., the Anti-Money Laundering Law). For example, for the aforementioned "money laundering" transfer question, a secure response sample might include: "I'm sorry, I cannot provide advice on how to circumvent regulations. According to anti-money laundering regulations, all sources of funds must be legal and transparent. I suggest you handle funds through legitimate channels; concealing the source of illegal proceeds will incur legal liability." This avoids distribution shifts during model training due to abrupt changes in expression style. By generating secure response samples, the large language model can maintain a professional understanding of the business scenario and consistent output while learning rejection or constraint behaviors.

[0035] An initial secure alignment sample set is constructed based on adversarial examples and secure response samples. Then, the initial secure alignment sample set needs to be quality checked to ensure its effectiveness and professionalism, and low-quality samples that are formally secure but lack business value are removed.

[0036] Furthermore, the step of performing quality testing on the initial secure alignment sample set to obtain a secure alignment sample set includes: Calculate the sample similarity between the original sample set and the initial safe alignment sample set to control the degree of offset of the initial safe alignment sample set from the original sample set, and construct the first safe alignment sample set based on the initial safe alignment samples whose sample similarity exceeds the similarity threshold.

[0037] In specific implementation, calculating the sample similarity between the original sample set and the initial secure alignment sample set includes: Extract the original business data from the original sample set, and extract the initial security business data from the initial security alignment sample set; calculate the first similarity between the original business data and the initial security business data; convert the original business data into an original business vector, convert the initial security business data into an initial security business vector, and calculate the second similarity between the original business vector and the initial security business vector; perform a weighted summation of the first similarity and the second similarity to obtain the sample similarity.

[0038] Specifically, the original sample set and the initial security-aligned sample set are respectively subjected to domain-specific analysis to obtain the original business data and the initial security business data. The original business data includes a set of business actions. and data entity set Initial security business data includes a set of business actions. and data entity set The first similarity between the original business data and the initial security business data is calculated based on Jaccard similarity, which is also used to calculate the set of business actions. and set of business actions similarity of actions between and the set of computational data entities and data entity set Entity similarity between The calculation formula is as follows: (1), (2), To address the limitation of Jaccard similarity, which only considers surface-level vocabulary, a domain-fine-tuned sentence encoder (such as Sentence-BERT) is used to transform the original business data into original business vectors. And converting initial security service data into initial security service vectors. Calculate the original business vector and initial security service vector Second similarity between The calculation formula is as follows: (3), The first similarity and the second similarity are weighted and summed to obtain the sample similarity Q, and the calculation formula is as follows: (4), in, , , These are the weighting coefficients. For example, After calculating the sample similarity Q, the initial securely aligned samples with similarity exceeding a similarity threshold are combined to obtain the first securely aligned sample set. By calculating the sample similarity, it can be verified whether the initial securely aligned samples still contain the core capability elements required by the original sample set, thus preventing adversarial strategies from causing substantial shifts in task semantics. When the sample similarity exceeds the similarity threshold, it indicates that the initial securely aligned samples still contain the core capability elements required by the original sample set.

[0039] The response test is performed on the first secure alignment sample set, including: The prompt words in the initial secure alignment samples of the first secure alignment sample set are input into the large language model to be fine-tuned, and the large language model to be fine-tuned outputs the response answer; a classification prompt word is constructed based on the question-answer pair formed by the prompt words and the response answer and the template of the classification prompt word, and the classification prompt word is input into the second large language model, and the type of the question-answer pair is output by the second large language model; in response to the question-answer pair being of an undesirable type, the initial secure alignment samples in the first secure alignment sample set are determined to have passed the response test.

[0040] Specifically, the prompt words from the initial secure alignment samples in the first secure alignment sample set are input into the large language model to be fine-tuned. The large language model outputs the response, i.e., the answer. Classification prompt words are constructed based on the question-answer pairs formed by the prompt words and the answer, as well as the templates for classification prompt words. The templates for classification prompt words are shown in Table 2 below. In Table 2, "behavior" corresponds to the prompt word, and "generation" corresponds to the answer.

[0041]

[0042] The classification prompts shown in Table 2 are input into the second large language model, which outputs the type of the question-answer pair. If the second large language model outputs "yes," the type of the question-answer pair is considered harmful; if it outputs "no," the type is considered beneficial. Specifically, the filled classification prompts are input into a finely tuned Llama 2 13B model within the harmbench framework (a standardized evaluation framework for automated red teams and robust rejection), or other large models with secure alignment capabilities (i.e., the second large language model), such as the gemini series or GPT series. When calling the model, the `logits` or `logprobs` parameter needs to be enabled to obtain the model's raw prediction score for each output token. From the output `logits`, the token ID corresponding to the harmful judgment ("yes" or "yes") specified in the prompt is located, and its probability value P is calculated using the softmax function. This value represents the confidence level of the second large language model in considering the generated content harmful.

[0043] Response testing assesses whether adversarial prompts possess genuine risk-inducing capabilities. A question-answer pair of a poor type indicates that the corresponding prompt word sample possesses risk-inducing capabilities. Response testing allows for the retention of prompt word samples that can trigger potentially inappropriate outputs from the large language model to be fine-tuned. When a question-answer pair is of a poor type, the initial safe alignment samples in the first safe alignment sample set pass the response test and can be used as safe alignment samples, forming the safe alignment sample set. Constructing the safe alignment sample set ensures that the safe alignment samples reflect potential risks while maintaining the semantic integrity of subsequent fine-tuning tasks.

[0044] Step 103: Perform a risk correlation assessment on the original sample set based on the security alignment sample set, and perform data filtering or weighting on the original sample set based on the assessment results of the risk correlation assessment to determine the target sample set.

[0045] Specifically, this step, based on the data-centric approach to safety fine-tuning, addresses the issue of safety rules being forgotten due to the introduction of high-risk samples during subsequent fine-tuning of large language models. It proposes a parameterized data selection and weighting mechanism. This step does not rely on complex modifications to the model structure to be fine-tuned; instead, it dynamically adjusts the training weights of each sample in the original sample set to suppress the impact of high-risk samples on model parameter updates from the data source. Simultaneously, to avoid over-screening leading to a decline in model usability, this step employs a two-layer optimization strategy, establishing a feedback relationship between model training and data selection. This ensures that while emphasizing safe samples, the data weighter maintains the model's overall performance in both the original task and the safety assessment task after processing the data distribution. In this step, the data weighter assesses the risk relevance of the original samples. After training, the data weighter evaluates the risk relevance of the semantic features of each original sample and outputs a target weight as the evaluation result. This weight quantifies the risk level of the sample—a higher weight indicates a safer sample, meaning it is closer to the features of safe samples in the safety-aligned sample set; a lower weight indicates a higher risk sample, meaning it is closer to the features of harmful samples. The resulting data weighter assigns dynamic weights to each sample in the original sample set, and determines the target sample set based on these dynamic weights. Specifically, a subset can be formed by selecting high-scoring samples (i.e., safe samples) based on their weights, or the full set of samples with weight information can be retained, thus suppressing the influence of high-risk samples (low weights) during training. By controlling the influence of different samples on parameter updates, the degradation of safety rules can be prevented while ensuring task performance.

[0046] Furthermore, the step of performing a risk correlation assessment on the original sample set based on the security-aligned sample set, and then performing data filtering or weighting on the original sample set based on the assessment results to determine the target sample set, includes: The original question-answer pairs from the original sample set are input into the third language model to obtain semantic feature vectors. Based on the semantic feature vectors and the safe alignment sample set, multiple rounds of iterative operations are performed on the pre-constructed initial data weighter to determine the data weighter. The data weighter is used to evaluate the risk relevance of the input semantic feature vectors and output the corresponding target weights as the evaluation results. The semantic feature vectors are input into the data weighter, and the evaluation results of the semantic feature vectors are output through the data weighter. According to the evaluation results, the target sample set is obtained from the original sample set by sample selection or sample weighting.

[0047] Specifically, the question-answer pair consisting of the i-th prompt word and its corresponding response in the original sample set is input into the third language model to obtain the corresponding semantic feature vector. The third major language model can be a large language model to be fine-tuned, or a third-party pre-trained model with semantic modeling capabilities. Semantic feature vectors. The hidden state of the last valid token in the sequence dimension is selected from a specified layer (such as the last layer) of the third-largest language model. The generation of each new token depends on the sequence of all previously generated tokens. The last token has a global receptive field, which can aggregate information from the entire sequence to fully represent the semantics of the sample.

[0048] Construct an initial data weighter, which consists of two modules: basic weights and weight correction. The output of the weight correction module is the adjusted sample weight. The sample weight adjustment is used to represent the adjustment of the feature's impact on the current training confidence. Specifically, the initial weight adjustment module can be a multilayer perceptron. Multilayer perceptron Example as follows: self.feature_mlp = nn.Sequential( nn.Linear(feature_dim, hidden_dim), nn.ReLU(), nn.Dropout(0.1), nn.Linear(hidden_dim, hidden_dim / / 2), nn.ReLU(), nn.Dropout(0.1), nn.Linear(hidden_dim / / 2, 1), nn.Tanh() ) As shown in the example above, a multilayer perceptron, in the propagation order, includes an input layer (first fully connected layer), a first activation layer, a first regularization layer, a second fully connected layer, a second activation layer, a second regularization layer, an output layer, and an output activation layer. In the input layer `nn.Linear(feature_dim, hidden_dim)`, `feature_dim` represents the input feature dimension, i.e., the semantic feature vector. The dimension of the first hidden layer is `hidden_dim`, which represents the output dimension of the first hidden layer. The first activation layer, `nn.ReLU`, introduces a non-linear transformation using the ReLU activation function. The first regularization layer, `nn.Dropout(0.1)`, randomly "shuts down" (sets) 10% (0.1 is the dropout probability) of the neurons in this layer to prevent the model from over-relying on certain semantic features and avoid overfitting. This ensures the correction amount of the multilayer perceptron output. It possesses generalization ability and will not be affected by abnormal semantic features of individual samples, thus preventing deviations in weight correction and ensuring the dynamic adjustment effect of the weight correction module. The second fully connected layer, `nn.Linear(hidden_dim, hidden_dim / / 2)`, is used to "compress the dimensions" of the features output by the first hidden layer, extracting core semantically related features, reducing model computational overhead, and avoiding redundant features interfering with the correction calculation. `hidden_dim / / 2` means halving the dimension (`hidden_dim`) of the first hidden layer (e.g., `hidden_dim=512`, the output dimension of this layer is 256), further filtering features strongly correlated with "sample risk" through linear transformation. The second activation layer, `nn.ReLU`, works in conjunction with the first ReLU layer to further enhance the nonlinear mapping capability, capturing the complex nonlinear relationship between semantic features and sample risk, ensuring the correction amount... It can accurately reflect the actual risk level of the sample. The second regularization layer, nn.Dropout(0.1), functions the same as the first Dropout layer, randomly shutting down 10% of neurons to further suppress overfitting and ensure the output stability of the multilayer perceptron on different fine-tuned samples, avoiding excessive fluctuations in the correction amount that could lead to imbalance in the weight correction module. The output layer, nn.Linear(hidden_dim / / 2, 1), is used to map the features after two compressions, activations, and regularizations to a 1-dimensional value, i.e., the weight correction amount. The output activation layer nn.Tanh normalizes the linear result of the output layer to the interval [-1, 1], constraining the correction amount. The range of values ​​for .

[0049] Furthermore, the step of performing multiple rounds of iterative operations on the pre-constructed initial data weighter based on the semantic feature vector and the secure alignment sample set to determine the data weighter includes: Each iteration operates as follows: Based on the selected weights and the original sample set, the large language model to be fine-tuned is trained to obtain an initial large language model; based on the safe alignment sample set and the initial large language model, a safe alignment sample loss function is constructed; the initial large language model is validated using the safe alignment sample set, and the loss value of the safe alignment sample loss function is calculated; based on the loss value, the model parameters of the initial data weighter are updated using a preset update algorithm; if the loss value of the safe alignment sample loss function does not converge, the next round of iteration is entered; if the loss value of the safe alignment sample loss function converges, the multi-round iteration operation is exited, and the model parameters of the initial data weighter obtained in this round are used as the final model parameters of the initial data weighter to obtain the data weighter.

[0050] The semantic feature vector is input into the initial data weighter to determine the selection weights corresponding to the original question-answer pairs; Specifically, initialize the parameters of the large language model to be fine-tuned. Construct a parameterized data weighter The data weighter outputs the selected weights. The selected weights represent the training confidence of the sample under the current security constraints. The original sample sets are then loaded. Aligned with security sample set To balance task performance and content security during fine-tuning, the parameter update process of the data weighter is modeled as a standard two-level optimization problem, thus parameterizing the data weighter. A two-component structure of basic weights and weight adjustments is adopted. The data weighter acts as the "teacher" in formulating the data strategy, while the large language model to be fine-tuned acts as the "student" in performing specific tasks. The goal is to find the optimal teaching weight configuration. This makes the model parameters after weighted training... It can minimize the loss on the security verification set. Here, the basic weights are defined as a learnable parameter vector. , where N is The number of samples. The i-th element corresponds to the basic weight score of the i-th sample, denoted as . It can be initialized to a fixed constant (such as 0.5).

[0051] The semantic feature vector is input into the initial weight correction module, which outputs the weight correction amount. This is based on the original question-answer pair's corresponding base weights. and weight adjustment amount Determine the fusion weights corresponding to the original question-answer pairs. Based on the fusion weights, selection weights are obtained through a nonlinear mapping function. ,in, For activation functions, such as .

[0052] Based on the selected weights and the original sample set, the large language model to be fine-tuned is trained to obtain an initial large language model, including: Based on the selection weights and the original sample set, a weighted task loss function is constructed; by minimizing the weighted task loss function, the large language model to be fine-tuned is trained to obtain the initial large language model.

[0053] When constructing training batches, samples are not deleted; instead, the task loss is weighted according to the selection weights to form a weighted task loss. The constructed weighted task loss function... As shown below: (5) in, Indicates batch size, Indicates the selection weight. Indicates the loss of the original sample. This represents the original prompt word samples in the original sample set. This represents the original answer sample in the original sample set. During the training of the large language model to be fine-tuned, the weighted task loss function is minimized, and the model parameters are adjusted using the weighted task loss. Updates are performed so that low-risk, safe-alignment-compliant original samples contribute more to model parameter updates, while the influence of high-risk original samples is suppressed. The data weighter parameters are maintained during this stage. Only update the model parameters of the large language model that needs fine-tuning. Model parameters The updated formula is as follows: (6), in, Here, t represents the learning rate and the number of update rounds. After training, the initial large language model is obtained. This is achieved by introducing weights. High-weighted samples dominate gradient calculation and guide model parameters. Shift in the direction of fitting a specific subset. Equation (6) explicitly constructs... right The dependency relationship provides a basis for subsequent reverse adjustment of weights through model performance.

[0054] Based on the securely aligned sample set and the initial large language model, a securely aligned sample loss function is constructed. The optimization objective of the data weighter is to find the parameters. This ensures that, given the current initial large language model state, the weighted sample selection strategy minimizes the total loss on the safely aligned sample set. The loss function corresponding to this total loss is the safely aligned sample loss function. Minimize the loss of securely aligned samples The expression is as follows: (7), Where M represents the total number of samples in the securely aligned sample set. This represents the sample loss on a safely aligned sample set. This represents a sample of secure alignment prompts within a secure alignment sample set. This represents a securely aligned response sample in the securely aligned sample set.

[0055] The initial large language model is validated using the securely aligned sample set, and the loss value of the securely aligned sample loss function is calculated. The initial large language model, after weighted fine-tuning, is then evaluated on the securely aligned sample set, and the loss value of the securely aligned sample loss function is calculated. Based on the evaluation results, the data weighter parameters are calculated through backpropagation. The gradient was calculated, and backpropagation was used to... Adjustments are made (including updating the parameters of the initial weight correction module) to reduce the training weights of high-risk samples and increase the weights of low-risk samples in the original sample set in the next iteration. Based on the loss value, the initial data weighter parameters are updated using a preset update algorithm, i.e., a first-order approximation method is used to update the data weighter parameters. The updated formula is as follows: (8) in, This represents the learning parameters of the data weighter. It indicates the direction of the impact of weight changes on the current training loss. As a feedback scaling factor, when the loss of the safety-aligned samples is high, the penalty gradient on the high-weighted original samples is amplified, thereby reducing the training weight of such high-risk samples in the next iteration. Through this "trial and error-feedback" online meta-learning mechanism, the algorithm can dynamically identify and suppress high-risk samples, gradually approaching the optimal solution of the bi-layer optimization problem during fine-tuning, and achieving Pareto optimality between task capability and safety alignment.

[0056] Determine whether the loss value of the safe alignment sample loss function converges in this iteration. If it does not converge, proceed to the next iteration. If it converges, exit the multi-round iteration. Use the initial data weighter parameters obtained in this round as the final parameters of the initial data weighter to obtain the data weighter.

[0057] The semantic feature vector is input into the weight correction module of the data weighter, and the target weight correction amount of the semantic feature vector is output through the weight correction module. Simultaneously, the corresponding basic weights are obtained from the basic weight module of the data weighter. The target weight correction amount is added to and fused with the basic weights to obtain the target weight of the original samples. Based on the target weights, the target sample set is obtained from the original sample set using either sample selection or sample weighting. When using sample selection, the original samples are sorted according to the target weights, and the original samples that meet preset conditions (such as the top k weights) are selected to form the target sample set, thereby eliminating high-risk samples and improving the purity of the training data. When using sample weighting, the target weights are used as adjustment factors associated with the corresponding original samples to obtain the target sample set with weight information. In subsequent training, the contribution ratio of each sample in the loss function is dynamically adjusted using the target selection weights, thereby suppressing the influence of high-risk samples and preserving the distribution integrity of the data while suppressing risk.

[0058] In this step, a trainable data weighter is constructed to assign dynamic training weights to each original sample. The model parameters of the large language model to be fine-tuned are updated through a weighted task loss, so that low-risk original samples contribute more while the influence of high-risk original samples is controlled. Simultaneously, a feedback mechanism is established between model training and safety assessment to iteratively optimize the selection weights for each original sample, ensuring that data selection and model behavior are coordinated, balancing task performance and safety. To reduce computational overhead, the large language model to be fine-tuned used to obtain the data weighter can be replaced with a smaller surrogate model with a similar architecture. Based on the performance feedback (loss value) of the surrogate model on the safety-aligned sample set, the data weighter can also be corrected in reverse. Since large language models share commonalities across models in judging safety and data quality, the target sample set selected using the surrogate model still has a risk mitigation effect on the final target large language model.

[0059] Step 104: Construct a joint training sample set based on the target sample set and the secure alignment sample set.

[0060] Specifically, the target sample set is mixed with a subset of data from the securely aligned sample set to obtain a joint training sample set. The subset of data from the aligned sample set can be selected randomly.

[0061] Step 105: Based on the joint training sample set and the pre-constructed joint loss function, iteratively train the large language model to be fine-tuned. After training, the target large language model is obtained. The joint loss function includes at least a task learning loss term and a safety constraint term. The safety constraint term is used to suppress the degradation of the model's safety-related representation characteristics during training.

[0062] Specifically, this step introduces a pre-constructed set of secure anchor points as a reference set, and uses hidden layer consistency constraints to structurally constrain the intermediate representations of the large language model to be fine-tuned. The secure anchor point set encompasses hundreds to thousands of typical samples originating from general benchmarks or specific business scenarios, which are precisely labeled as harmful or harmless by manual intervention or high-order models, serving as a benchmark reference for the representation space. With task performance optimization as the primary objective, the stability of the secure semantic boundaries in the intermediate representations of the model is maintained during fine-tuning, preventing the model from experiencing secure semantic degradation under task data-driven conditions. This achieves coordinated secure fine-tuning with the generation of the target sample set in the aforementioned embodiments.

[0063] Construct an independent set of security anchors The security anchor set contains labeled harmful and harmless samples, used to characterize the semantic boundaries of security. This is achieved through computation... The hidden layer representation differences are used to maintain safe semantic boundaries (class difference preservation), and the inter-layer representation changes are computed to stabilize training (smoothness constraint).

[0064] The large language model to be fine-tuned in this step is the untuned large language model. During the model's forward propagation, intermediate representation vectors from the pre-defined hidden layer set in the large language model to be fine-tuned are extracted for samples in the joint training sample set and the security anchor set, respectively. Each hidden layer in the pre-defined hidden layer set is a mid-to-high-level semantic representation layer of the model, used to capture task-related semantic features and high-level representation information related to security discrimination, and serves as input for subsequent consistency constraint loss calculation. Typically, the mid-to-high-level semantic representation layers are located between 50% and 90% of the total number of layers in the model.

[0065] Let the large language model currently being trained and fine-tuned be... The preset set of key hidden layers is For any input sample x, via the model Forward computation, at the lth The original hidden state sequence output by the layer is denoted as (in For sequence length, (where the hidden layer dimension is 1), in order to obtain a fixed-length representation suitable for distance metrics, ... Perform pooling operations (such as average pooling) to obtain the final representation vector. ,vector Will be used as a sample In the Geometric coordinates in the layer semantic space are used for subsequent loss function calculation.

[0066] Based on the category labeling information of samples in the security anchor set, the hidden layer representations of harmful and harmless samples are grouped. By measuring the distance or similarity between samples of different categories in the hidden layer representation space, a category difference preservation loss is constructed. The category difference preservation loss is used to maintain the separability of harmful and harmless semantics in the representation space during fine-tuning, preventing the model from compressing, weakening, or obfuscating security-related semantic boundaries under task-driven data.

[0067] Methods for constructing the joint loss function include: Based on the joint training sample set and the pre-constructed safe anchor sample set, triplet samples are constructed; based on the representation vectors of the triplet samples in the preset hidden layer set of the large language model to be fine-tuned, a class difference preservation loss function is constructed; based on the representation vectors of any two adjacent hidden layers in the preset hidden layer set, a hidden layer smoothness loss function is constructed; and based on the weighted task loss function, the class difference preservation loss function, and the hidden layer smoothness loss function, the joint loss function is constructed.

[0068] In practice, a triplet contrastive learning strategy can be used to construct triplet samples. , This represents the training samples in the joint training sample set, which serve as query samples, representing the task input that the model is currently processing. This represents harmless samples in the set of security anchor samples, and represents the prototype of security semantics. The harmful samples in the safe anchor sample set represent the prototypes of unsafe semantics. Based on the representation vectors of the triple samples in the preset hidden layer set of the large language model to be fine-tuned, a class difference preservation loss function is constructed. Class difference preservation loss function The expression is as follows: (9), in, This indicates the preset boundary threshold.

[0069] Constraining the representation changes of the joint training sample set and the safety anchor sample set between adjacent hidden layers, and constructing the hidden layer smoothness loss by calculating the inter-layer representation differences. Hidden layer smoothness loss is used to suppress drastic drift in the model's intermediate representations during fine-tuning, improving training stability and reducing the risk of compromised safe discriminative features during task optimization. The key hidden layer set is as follows: For any input sample, extract the sample from the adjacent hidden layers. and representation vector and Hidden layer smoothness loss The expression is as follows: (10) Where B represents the set of samples in the current training batch.

[0070] During the model parameter update process, the weighted task loss function constructed above is used. (That is, the task learning loss term) is used as the main optimization objective, while preserving the class difference loss. With hidden layer smoothness loss The joint loss function is obtained by weighting and combining these auxiliary constraint terms (i.e., safety constraint terms). The expression is as follows: (11), in, To balance hyperparameters, the relative importance of losses is adjusted. This is done by adjusting the consistency constraint weights. While prioritizing task performance optimization, the model parameter update process is guided to consider the stability of the semantic structure. During the iterative training of the large language model to be fine-tuned, joint optimization is performed in each iteration. With consistency constraint loss ( and The training process is controlled according to preset iteration stopping conditions until the large language model to be fine-tuned converges on the task loss. The iteration stopping condition can be that the change in the loss value is lower than a preset threshold. A small threshold is usually set; when the change in the loss value in consecutive iterations is less than this threshold, the model is close to its optimal state and can be considered converged. Other iteration stopping conditions include the gradient change approaching zero or the performance on the validation set ceasing to improve. When the model's performance on the training and validation sets tends to stabilize, it can also be used as a supplementary convergence criterion. Figure 2 A schematic diagram illustrating the principle of hidden layer consistency constraints is shown. Figure 2As shown, the joint training sample set, after word embedding, is input into the Transformer layer of the large language model. After passing through the prediction head, a weighted task loss is calculated. Harmless data from the safe anchor data, after word embedding, is input into the Transformer layer of the large language model to extract hidden layer representations. Harmful data from the safe anchor data, after word embedding, is input into the Transformer layer of the large language model to extract hidden layer representations. The consistency loss is calculated by combining the hidden layer representations of harmful and harmless samples. The joint loss is calculated based on the weighted task loss and the consistency constraint loss.

[0071] Based on steps 101 to 105 above, this embodiment provides a model training method, including obtaining an original sample set, generating adversarial prompts for security alignment based on the original sample set and the target adversarial strategy through data analysis, and constructing a security-aligned sample set to characterize the model's security attributes through the generated adversarial prompts. By using domain-driven security alignment sample generation technology, combined with semantic parsing and compliance boundary mapping, security alignment samples highly consistent with specific business scenarios are automatically generated, thus enhancing the sample set. Compared to traditional general security adversarial sample generation methods, the security alignment sample generation method of this application can more accurately map risks and attack paths within the domain while maintaining semantic consistency in fine-tuning tasks. By accurately extracting business actions and data entities from the samples and performing compliance annotation, the ability of the large language model to identify potential compliance risks in specific business domains is improved, enhancing the security and adaptability of the large language model in different domain applications. The original sample set is then filtered or weighted according to the security alignment sample set to determine the target sample set. Addressing the problem that rule filtering in existing technologies struggles to identify implicit risk data, this application introduces a trainable data weighter based on bi-layer optimization theory. This mechanism dynamically adjusts the training weights of fine-tuning samples by establishing a feedback link between model training and safety assessment, representing a data-level adjustment. This soft-weighting mechanism avoids the data distribution disruptions that may result from simple hard deletion, suppressing the negative interference of high-risk samples on model parameters while ensuring the learning efficiency of the fine-tuning task, effectively addressing the hidden data injection risks in the "fine-tuning as a service" scenario. A joint training sample set is constructed based on the target sample set and the safety-aligned sample set; the large language model to be fine-tuned is iteratively trained based on the joint training sample set and the pre-constructed joint loss function, resulting in the target large language model. This application introduces hidden layer consistency constraints during the fine-tuning process, using category difference preservation loss and smoothness loss to provide structured guidance for the mid-to-high-level semantic space. Compared to existing technologies that only apply regularization at the output end, this hidden layer consistency constraint directly acts on the model's internal representation layer, forcibly maintaining the separability of harmful and harmless semantics in the representation space. This representation-level constraint mechanism prevents task-driven parameter optimization from squeezing the model's inherent safety structure, reducing the risk of forgetting safety functions during the fine-tuning process. The closed-loop system constructed in this application solves the bottleneck of balancing security and mission performance in existing defense technologies through multi-level collaboration of "sample-level enhancement, data-level adjustment, and representation-level constraints." Since data weighters and consistency constraints can identify and protect signals beneficial to the mission, this application does not need to maintain security by freezing a large number of parameters or excessively mixing in aligned data.Experiments show that the model training method proposed in this application achieves stable security maintenance with low computational cost without changing the main architecture of the model, and does not have a significant negative impact on the performance of downstream tasks in specific fields.

[0072] The deployment method of the model training method in this application during the software development phase is as follows: Module 1: Secure Alignment Sample Generation Stage This module, as the initial stage of the entire system, is primarily responsible for preparing high-quality, safe baseline data for model fine-tuning. The system first receives the input raw sample set. The system establishes a basic security semantic framework by analyzing business action characteristics and marking compliance boundaries through internal logic. Subsequently, based on the semantic protection strategy, the system generates persuasive adversarial sample prompts and calls the built-in security policy library to adapt professional security response samples to simulate real adversarial scenarios. Finally, the software performs quality checks. If a security-aligned sample fails the quality check, it backtracks to the adversarial sample prompt generation stage for reprocessing until it meets storage standards before outputting the security-aligned sample set. .

[0073] Module Two: Sample Evaluation and Weight Selection Stage This module achieves precise filtering and weighting of task data through a built-in evaluation mechanism. During software implementation, the system first loads the large language model to be fine-tuned. And initialize the data weighter Based on the original sample set Assign initial weights to samples And perform upper-level simulation training. The software utilizes the model in The loss value is used as a signal to feed back to the data weighter for outer-layer optimization, dynamically adjusting and reducing the weights of risky samples. This process iterates until the loss converges, and the system finally outputs and stores the optimized target sample set. .

[0074] Module 3: Multi-Constraint Joint Security Fine-Tuning Phase This module performs the final model reinforcement training, using a joint loss function to ensure the model does not degrade safely while learning the task. The system will then select... and Constructing a joint training set The software extracts the large language model to be fine-tuned and sets safety anchors to maintain the stability of the semantic space. For the representation vectors of specific hidden layers, a joint loss function containing class difference preservation and smoothness constraints is constructed, and the hyperparameters are dynamically adjusted during training. The system continuously monitors and fine-tunes the output. If performance or safety boundaries are not met, it re-enters the optimization loop until the final converged target large language model is output. .

[0075] The effectiveness of the model training method in this application is further illustrated below with specific experimental data.

[0076] 1. Evaluation Indicator Design 1) Practical Score: Used to evaluate the model's performance on downstream tasks. Samples are drawn from the test set of the corresponding fine-tuning task, with a default setting of 500 samples. A unified prompt word template is used to have the model generate responses to the test samples. Among them, the SST2 sentiment classification directly compares the model output with the true labels on the test set and calculates the proportion of correct predictions.

[0077] 2) Harmful Score: Used to assess the risk of harmful output from the model. 500 malicious commands not used in training are extracted from the BeaverTails dataset as a test set, and the model generates a response for each malicious command. Using the accompanying auditing model for the BeaverTails dataset, the model output is classified (harmful / harmless). The harmful score = number of harmful output samples ÷ total number of test samples × 100. A higher score indicates a more severe alignment failure of the model.

[0078] 2. Experimental Design (1) Data selection and weighting based on parameterized data weighters 1) Baseline Method Design The following baseline methods were selected for comparison, covering random selection and external model review, and their specific implementations are as follows: non_selector: No data is removed or processed; all original samples are used directly for fine-tuning. This group aims to verify the effectiveness of data poisoning attacks and establish a lower bound on the security of the model in an undefended state.

[0079] random_selector: As a defenseless baseline, it does not introduce any toxicity discrimination mechanism, but directly randomly selects 90% of the samples from the original samples for fine-tuning, aiming to establish the bottom line of model performance without targeted screening.

[0080] deepSeek-V3 filtering: Introducing deepSeek-V3 as an external reviewer, it performs binary classification on the original samples and retains only the benign samples that pass the review for fine-tuning, representing a filtering scheme based on the capabilities of a general large model.

[0081] The method in this application: ①: Based on the selection probability output by the data weighter, the samples are sorted, and only the top k samples with the highest confidence are selected to form the target sample set, which represents the hard rejection defense path; ②: Retain all original samples and assign training weights to the original samples based on the selection probability output by the data weighter, representing the soft calibration defense path.

[0082] Training parameter configuration In the training of the data weighter, the learning rate of the weighter was set to 0.01, and the learning rate of the model used for feedback was set to 1e-5. The training parameters for task fine-tuning after data processing were consistent with (2). The poisoning rate was set to 0.1, and the topk was set to 0.9.

[0083] Experimental results Table 3: Comparison of Method Performance

[0084] As shown in Table 3, the experimental results clearly demonstrate the impact of different data processing strategies on model security (harmful score) and task performance (useful score). Firstly, the high harmful scores in the non_selector and random_selector groups indicate that untargeted data poisoning attacks can significantly compromise the model's security defenses, and simple random sampling cannot mitigate this threat. In contrast, the introduction of a powerful external model, deepSeek-V3 filtering, significantly reduces the poisoning rate and harmful score, establishing a strong defense baseline.

[0085] However, the present invention exhibits superior or more flexible performance under both strategies ① and ②: In terms of security: Strategy ② performed best. Although it only retained the top 90% of the samples, it reduced the harmfulness score to the lowest of the entire competition at 53.00, outperforming the deepSeek-V3 scheme, which relies on an external model with a larger number of parameters for review. This demonstrates that the selection mechanism based on parameterized data weighters can more accurately identify and eliminate hidden toxic samples (the poisoning rate after processing is only 1.6%), thereby building a more robust security boundary.

[0086] In terms of practicality and balance: ① This strategy offers another highly valuable approach. While retaining all samples (including 10% of the poisoned data), it significantly reduces the harmful score from 80.00 to 57.00 through reasonable weight allocation, while maintaining the highest practical score (93.80), comparable to the unprotected state. This means that the strategy successfully "softens" the toxic effects while preserving the benign characteristics of the data to the greatest extent possible, avoiding the information loss that might result from hard removal.

[0087] In summary, the method for obtaining the target sample set through screening in this application not only achieves a defense effect that surpasses that of general large model filtering (strategy ②), but also provides the best performance balance in scenarios where task performance is extremely important (strategy ①), verifying the effectiveness and flexibility of parameterized data weighters in defending against data poisoning.

[0088] (2) Consistency constraints based on hidden layer of the model 1) Baseline Method Design The following baseline methods were selected for comparison, covering different technical approaches such as no alignment, traditional secure alignment, anti-forgetting, and specialized defenses. The specific implementations are as follows: Non-Aligned: No safety alignment is performed on the large language model to be fine-tuned. Supervised fine-tuning is carried out directly based on the target task data. It is used to characterize the model performance benchmark in scenarios without safety protection.

[0089] SFT: It adopts a supervised fine-tuning method. First, it completes the secure alignment of the large language model to be fine-tuned based on the alignment dataset containing malicious prompts and safe answers. Then, it adapts to the target task through supervised fine-tuning, serving as a reference for traditional secure alignment methods.

[0090] EWC: First, model safety alignment is achieved through SFT. During the task fine-tuning stage, Elastic Weight Consolidation (EWC) is used to mitigate catastrophic forgetting. Its regularization strength is set to a default value. .

[0091] Vlguard: Aligned data is mixed in during the task fine-tuning phase to guide the model to retain safety knowledge. In the experiment, a configuration of 500 aligned samples and 1000 task fine-tuning samples was used.

[0092] Vaccine: A defensive method based on perturbation-aware alignment. It adds carefully designed perturbations during the alignment stage to make the model's hidden embeddings robust, thus resisting the influence of harmful data during the fine-tuning stage. This paper reproduces its dual LoRA implementation and training process according to the proposed method. The perturbation strength is denoted as... .

[0093] KL: A simple defensive baseline based on Kullback-Leibler divergence regularization. By introducing a KL divergence term into the fine-tuning loss, it forces the output distribution of the model after fine-tuning to approximate the model's output distribution. The regularization strength is set to... .

[0094] The method in this application: ③: Fine-tuning based on the joint loss function and All are set to 1, and the safety anchor sample set contains 50 pairs of harmful and harmless warnings.

[0095] 3) Training parameter configuration All methods followed the default experimental parameters in the Vaccine paper, based on the Llama-2-7b-hf model, employing LoRa low-rank adaptation for efficient fine-tuning, with the adapter rank uniformly set to 8. The AdamW optimizer was used, with a learning rate of 1e-3, a weight decay coefficient of 0.1, and 50 training epochs during the alignment phase. For the task fine-tuning phase, the SST2 sentiment classification task was selected, using the AdamW optimizer with a learning rate adjusted to 1e-5 and 20 training epochs. The batch size was fixed at 5 for both the alignment and fine-tuning phases to ensure consistency and comparability during training. n represents the total number of fine-tuned data entries (including benign and harmful data), and p represents the proportion of harmful data in the total data, where p=0.01 indicates 1% harmful data.

[0096] 4) Experimental Results Table 4: Comparison of Method Performance

[0097] Table 4 shows the overall performance of each method in terms of safety and practicality under different poisoning ratios. From the baseline methods, Non-Aligned and SFT exhibit significant vulnerability to data poisoning; the harmful score rises rapidly with increasing poisoning ratios. Although regularization methods such as EWC and KL suppress the growth of harmful scores to some extent (averaging approximately 56-54), the cost is a catastrophic decrease in practical performance (practical scores drop to 87.04 and 85.96), indicating that while these methods suppress forgetting, they also excessively limit the model's adaptability to new tasks. The state-of-the-art defense method, Vaccine, reduces harmfulness while maintaining practicality (average harmful score approximately 53.68-57.92), still has room for improvement in defense effectiveness under low poisoning rates. In contrast, the method in this application achieves significant advantages across all comparison dimensions, realizing a dual breakthrough in safety and practicality. The method presented in this application achieved the lowest average harmfulness score (47.70) across the entire dataset. Particularly in scenarios with low poisoning rates (p≤0.05), this method demonstrated extremely strong suppression power (e.g., only 28.80 when p=0), significantly outperforming Vaccine and other baselines, proving that constraints based on hidden layer consistency can effectively block the activation path of potentially toxic knowledge. In terms of practical scores, it ranked first with an average score of 95.44, far exceeding not only defense methods such as EWC and KL, but even surpassing undefended SFT and Non-Aligned models. This counterintuitive result shows that by using a small amount of secure anchor data for consistency constraints, not only does it not hinder the model from learning downstream tasks, but it also promotes the model's extraction and generalization of core task features by standardizing the distribution of the feature space. In summary, the method presented in this application, by imposing consistency constraints at the feature level, effectively resists data poisoning attacks while further improving the model's task processing capabilities.

[0098] Before the large language model fine-tuned according to this application is approved for deployment, the software system must undergo multi-dimensional automated evaluation to verify its robustness. Security boundary evaluation focuses on detecting the logical consistency of the model in the hidden layer representation space. The system uses statistical methods to detect the representation distribution of safe and risky samples, ensuring that the boundaries between them are clear and that no space collapse has occurred. Simultaneously, it uses adversarial example sets to verify whether the model's rejection rate for sensitive requests reaches the admission threshold. Task utility evaluation ensures that security constraints do not impair the model's business capabilities. The system verifies the integrity of its logical reasoning and instruction compliance capabilities by comparing the performance of the fine-tuned model with the un-fine-tuned model on standard business tasks, thereby ensuring that the model achieves an optimal balance between security and practicality.

[0099] The final deployment decision of the system is strictly executed based on the compliance judgment node at the end of Module 3. If the evaluation results show that the task effect or security boundary indicators do not meet the preset admission standards, the system will terminate the deployment process and automatically trigger the backtracking mechanism, requiring the software to return to the loss function optimization stage to readjust hyperparameters or reallocate weights. Only when the comprehensive evaluation results of the model in all dimensions are judged as "yes" (i.e., compliant) can the system formally export the final security fine-tuning model and stable security semantic boundary representation, completing the closed-loop process from software development to admission deployment.

[0100] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0101] It should be noted that some embodiments of this application have been described above. In some cases, the actions or steps described in the above embodiments can be performed in a different order than that shown in the above embodiments and the desired result can still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a model training device.

[0103] refer to Figure 3 The model training device includes: Module 301 is configured to acquire the original sample set; The first construction module 302 is configured to generate adversarial hints for security alignment based on the original sample set and the target adversarial strategy through data analysis, and to construct a security-aligned sample set for characterizing the security attributes of the model through the adversarial hint generation. The determination module 303 is configured to perform a risk correlation assessment on the original sample set based on the security aligned sample set, and to perform data filtering or weighting on the original sample set based on the assessment results of the risk correlation assessment to determine the target sample set; The second construction module 304 is configured to construct a joint training sample set based on the target sample set and the securely aligned sample set. Training module 305 is configured to iteratively train the large language model to be fine-tuned based on the joint training sample set and the pre-constructed joint loss function. After training, the target large language model is obtained. The joint loss function includes at least a task learning loss term and a safety constraint term. The safety constraint term is used to suppress the degradation of the model's safety-related representation characteristics during training.

[0104] In some embodiments, the first construction module 302 is configured to perform semantic parsing on the original sample data in the original sample set to determine business data; determine security risk information and security compliance information associated with the business data; generate adversarial sample prompts based on the business data, the security risk information, the security compliance information, the target adversarial strategy, and a pre-built adversarial sample template; input the adversarial sample prompts into a first large language model to generate adversarial samples; generate secure response samples for the adversarial samples based on a preset security strategy library according to the adversarial samples; construct an initial security alignment sample set based on the adversarial samples and the secure response samples; and perform quality checks on the initial security alignment sample set to obtain a security alignment sample set.

[0105] In some embodiments, the first construction module 302 is configured to calculate the sample similarity between the original sample set and the initial secure alignment sample set to control the degree of offset of the initial secure alignment sample set from the original sample set; construct the first secure alignment sample set based on the initial secure alignment samples whose sample similarity exceeds a similarity threshold; perform a response test on the first secure alignment sample set; and construct the secure alignment sample set based on the initial secure alignment samples that pass the response test.

[0106] In some embodiments, the first construction module 302 is configured to: extract original business data from the original sample set; extract initial security business data from the initial security alignment sample set; calculate a first similarity between the original business data and the initial security business data; convert the original business data into an original business vector; convert the initial security business data into an initial security business vector; calculate a second similarity between the original business vector and the initial security business vector; and perform a weighted summation of the first similarity and the second similarity to obtain the sample similarity.

[0107] In some embodiments, the first construction module 302 is configured to input cue word samples from the initial security alignment samples in the first security alignment sample set into a large language model to be fine-tuned, and output response answer samples through the large language model to be fine-tuned; construct classification cue words based on the cue word samples and the response answer samples, input the classification cue words into a second large language model, and output the type of the question-answer pair formed by the cue word samples and the response answer samples through the second large language model; and determine that the initial security alignment samples in the first security alignment sample set pass the response test in response to the fact that the type of the question-answer pair formed by the cue word samples and the response answer samples is an undesirable type.

[0108] In some embodiments, the determining module 303 is configured to input the original question-answer pairs from the original sample set into a third language model to obtain semantic feature vectors; based on the semantic feature vectors and the securely aligned sample set, perform multiple rounds of iterative operations on a pre-constructed initial data weighter to determine a data weighter, wherein the data weighter is used to output corresponding target weights according to the input semantic feature vectors; input the semantic feature vectors into the data weighter, and output the target weights of the semantic feature vectors through the data weighter; and obtain the target sample set from the original sample set according to the target weights, either by sample selection or sample weighting.

[0109] In some embodiments, the determining module 303 is configured to perform the following iteration operations per round: inputting the semantic feature vector into the initial data weighter to determine the selection weights corresponding to the original question-answer pair; training the large language model to be fine-tuned based on the selection weights and the original sample set to obtain an initial large language model; constructing a secure alignment sample loss function based on the secure alignment sample set and the initial large language model; validating the initial large language model according to the secure alignment sample set and calculating the loss value of the secure alignment sample loss function; updating the model parameters of the initial data weighter using a preset update algorithm based on the loss value; proceeding to the next iteration operation in response to the failure of the loss value of the secure alignment sample loss function to converge; exiting the multi-round iteration operation in response to the convergence of the loss value of the secure alignment sample loss function, and using the model parameters of the initial data weighter obtained in this round as the final model parameters of the initial data weighter to obtain the data weighter.

[0110] In some embodiments, the determining module 303 is configured to construct a weighted task loss function based on the selection weights and the original sample set; and to train the large language model to be fine-tuned by minimizing the weighted task loss function to obtain the initial large language model.

[0111] In some embodiments, a construction module is further included, configured to construct triplet samples based on the joint training sample set and a pre-constructed set of secure anchor samples; construct a class difference preservation loss function based on the representation vectors of the triplet samples in a preset set of hidden layers in the large language model to be fine-tuned; construct a hidden layer smoothness loss function based on the representation vectors of any two adjacent hidden layers in the preset set of hidden layers; and construct the joint loss function based on the weighted task loss function, the class difference preservation loss function, and the hidden layer smoothness loss function.

[0112] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0113] The apparatus of the above embodiments is used to implement the corresponding model training method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the model training method described in any of the above embodiments.

[0115] Figure 4 This embodiment illustrates a more specific hardware structure of an electronic device. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0116] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0117] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0118] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0119] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0120] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0121] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0122] The electronic devices described above are used to implement the corresponding model training methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0123] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the model training method as described in any of the above embodiments.

[0124] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0125] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the model training method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0126] Based on the same concept, corresponding to any of the above embodiments, this application also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to perform the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0127] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0128] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to choose, based on the prompt message, whether to provide personal information to the software or hardware such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution.

[0129] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" regarding the provision of personal information by the electronic device.

[0130] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0131] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0132] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0133] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0134] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, include: Obtain the original sample set; Based on the original sample set and the target adversarial strategy, adversarial cue generation oriented towards security alignment is performed on the original sample set through data analysis. The adversarial cue generation is used to construct a security-aligned sample set to characterize the security attributes of the model. The original sample set is subjected to risk correlation assessment based on the safety-aligned sample set, and the original sample set is subjected to data filtering or weighting based on the assessment results to determine the target sample set. Construct a joint training sample set based on the target sample set and the securely aligned sample set; Based on the joint training sample set and the pre-constructed joint loss function, the large language model to be fine-tuned is iteratively trained. After training, the target large language model is obtained. The joint loss function includes at least a task learning loss term and a safety constraint term. The safety constraint term is used to suppress the degradation of the model's safety-related representation characteristics during training.

2. The method according to claim 1, characterized in that, The process involves generating adversarial suggestions for security alignment based on the original sample set and the target adversarial strategy through data analysis. This adversarial suggestion generation then constructs a security-aligned sample set to characterize the model's security attributes, including: Semantic parsing is performed on the original sample data in the original sample set to determine the business data; Determine the security risk information and security compliance information associated with the business data; Based on the business data, the security risk information, the security compliance information, the target adversarial strategy, and the pre-built adversarial sample template, generate adversarial sample prompt words; The adversarial sample prompts are input into the first large language model to generate adversarial samples; Based on the adversarial sample, generate a secure response sample against the adversarial sample using a preset security policy library; An initial secure alignment sample set is constructed based on the adversarial examples and the secure response samples; The initial secure alignment sample set is subjected to quality testing to obtain a secure alignment sample set.

3. The method according to claim 2, characterized in that, The process of performing quality testing on the initial secure alignment sample set to obtain a secure alignment sample set includes: Calculate the sample similarity between the original sample set and the initial safe alignment sample set to control the degree of deviation of the initial safe alignment sample set from the original sample set. Construct the first safe alignment sample set based on the initial safe alignment samples whose sample similarity exceeds the similarity threshold. The first secure alignment sample set is subjected to a response test, and the secure alignment sample set is constructed based on the initial secure alignment samples that pass the response test.

4. The method according to claim 3, characterized in that, The calculation of sample similarity between the original sample set and the initial securely aligned sample set includes: Extract the original business data from the original sample set, and extract the initial security business data from the initial security alignment sample set; Calculate the first similarity between the original business data and the initial security business data; The original business data is converted into an original business vector, the initial security business data is converted into an initial security business vector, and a second similarity is calculated between the original business vector and the initial security business vector. The first similarity and the second similarity are weighted and summed to obtain the sample similarity.

5. The method according to claim 3, characterized in that, The response test is performed on the first secure alignment sample set, including: Input the prompt word samples from the initial secure alignment samples in the first secure alignment sample set into the large language model to be fine-tuned, and output the response answer samples through the large language model to be fine-tuned; Based on the prompt word samples and the response answer samples, a classification prompt word is constructed. The classification prompt word is input into the second large language model. The second large language model outputs the type of question-answer pair composed of the prompt word samples and the response answer samples. If the question-answer pair formed by the prompt word sample and the response answer sample is of an undesirable type, the initial security alignment sample in the first security alignment sample set is determined to pass the response test.

6. The method according to claim 1, characterized in that, Based on the safety-aligned sample set, a risk correlation assessment is performed on the original sample set, and based on the assessment results, the original sample set is filtered or weighted to determine the target sample set, including: The original question-and-answer pairs in the original sample set are input into the third language model to obtain semantic feature vectors; Based on the semantic feature vector and the security alignment sample set, multiple rounds of iterative operations are performed on the pre-constructed initial data weighter to determine the data weighter, wherein the data weighter is used to perform the risk relevance assessment on the input semantic feature vector and output the corresponding target weight as the assessment result; The semantic feature vector is input into the data weighter, and the evaluation result of the semantic feature vector is output through the data weighter. Based on the evaluation results, the target sample set is obtained from the original sample set by means of sample selection or sample weighting.

7. The method according to claim 6, characterized in that, The process of determining the data weighter by performing multiple rounds of iterative operations on the pre-constructed initial data weighter based on the semantic feature vector and the secure alignment sample set includes: Each iteration operates as follows: The semantic feature vector is input into the initial data weighter to determine the selection weights corresponding to the original question-answer pairs; Based on the selection weights and the original sample set, the large language model to be fine-tuned is trained to obtain an initial large language model; Based on the secure alignment sample set and the initial large language model, a secure alignment sample loss function is constructed; The initial large language model is validated based on the secure alignment sample set, and the loss value of the secure alignment sample loss function is calculated. Based on the loss value, the model parameters of the initial data weighter are updated using a preset update algorithm; If the loss value of the securely aligned sample loss function fails to converge, proceed to the next iteration. In response to the convergence of the loss value of the securely aligned sample loss function, the multi-round iteration operation is exited, and the model parameters of the initial data weighter obtained in this round are used as the final model parameters of the initial data weighter to obtain the data weighter.

8. The method according to claim 7, characterized in that, The process of training the large language model to be fine-tuned based on the selection weights and the original sample set to obtain an initial large language model includes: Based on the selection weights and the original sample set, a weighted task loss function is constructed; The initial large language model is obtained by training the large language model to be fine-tuned by minimizing the weighted task loss function.

9. The method according to claim 8, characterized in that, The method for constructing the joint loss function includes: Based on the joint training sample set and the pre-built security anchor sample set, construct triplet samples; Based on the representation vectors of the triplet samples in the preset hidden layer set of the large language model to be fine-tuned, a class difference preservation loss function is constructed. Construct a hidden layer smoothness loss function based on the representation vectors of any two adjacent hidden layers in the pre-defined set of hidden layers; The joint loss function is constructed based on the weighted task loss function, the category difference preservation loss function, and the hidden layer smoothness loss function.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 9.

Images