A large language model security alignment method and system based on multi-language self-distillation

Abstract

This invention discloses a secure alignment method and system for large language models based on multilingual self-distillation, belonging to the fields of artificial intelligence and natural language processing. The method instantiates a teacher model and a student model from the same large language model. The student model only receives queries in the target low-resource language, while the teacher model additionally receives English queries and thought chain instructions to stimulate secure reasoning capabilities in high-resource languages. A dual-perspective security weighting mechanism is proposed to jointly evaluate teacher confidence and student deviation, adaptively adjusting the token-level penalty weight. It supports both online and offline self-distillation modes, achieving cross-language security capability transfer through weighted KL divergence optimization. This invention requires no response data and achieves superior security performance on multiple large language models and multilingual jailbreaking benchmarks, particularly demonstrating strong generalization ability on low-resource languages ​​not seen in the training set, while maintaining model versatility and eliminating the problem of existing methods relying on high-cost response data generation.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and in particular to a secure alignment method and system for large language models based on multilingual self-distillation. Background Technology

[0002] Large language models have become an important application worldwide, enabling users from different language and cultural backgrounds to benefit from the development of artificial intelligence. However, existing large language models suffer from a serious imbalance in multilingual security alignment: the models exhibit strong security protection capabilities in high-resource languages ​​(such as English), effectively identifying and rejecting malicious requests, but are extremely vulnerable to jailbreak attacks in low-resource languages ​​(such as Swahili and Javanese), rendering their security protection mechanisms ineffective.

[0003] Existing multilingual secure alignment methods primarily rely on supervised fine-tuning or preference optimization techniques. A patent application with publication number CN120745837A discloses a reasoning-based multilingual security protection framework; this scheme performs knowledge distillation on a basic large language model through supervised fine-tuning and combines it with constraint alignment optimization to achieve cross-language knowledge transfer. However, this scheme requires generating high-quality response data for each target language for supervised fine-tuning, which is costly and difficult, especially for low-resource languages. A patent application with publication number CN121683936A discloses a dynamic self-distillation method for large language models oriented towards text generation; this scheme achieves self-distillation training through confidence-driven loss optimization and a confidence-quality joint evaluation framework. However, this scheme does not address multilingual security scenarios and cannot solve the cross-language secure alignment problem.

[0004] In addition, while existing training-free methods (such as representation engineering and activation guidance) reduce the dependence on response data, they suffer from weak generalization ability and capability degradation across datasets, making it difficult to maintain stable security performance in diverse real-world application scenarios.

[0005] Therefore, a technical solution is needed that can effectively transfer the security protection capabilities of large language models in high-resource languages ​​to low-resource languages ​​without requiring any response data, while maintaining strong generalization and versatility. Summary of the Invention

[0006] The purpose of this invention is to provide a secure alignment method and system for large language models based on multilingual self-distillation, in order to solve the following technical problems existing in the prior art: existing large language models suffer from serious imbalances in multilingual secure alignment; the models have strong security protection capabilities in high-resource languages, but are extremely vulnerable to jailbreak attacks in low-resource languages; existing secure alignment methods rely on generating high-quality response data for each target language for supervised fine-tuning or preference optimization, which is costly and difficult; existing training-free methods suffer from weak generalization ability and capability degradation across datasets.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a secure alignment method for large language models based on multilingual self-distillation. The method is executed by a computing device, including a processor and a memory, and includes the following steps: the processor instantiates a teacher model and a student model from the same large language model. The teacher model and the student model have the same model structure and initial parameters, distinguished only by input context information. The student model only receives queries in the target low-resource language, while the teacher model additionally receives queries in the high-resource language version and thought chain instructions. The processor samples the output sequence from either the teacher model or the student model according to a sampling strategy, including online and offline strategies. Based on the output sequence, the processor calculates the penalty weight of each token using a dual-view secure weighting mechanism, which is based on the top-K of the teacher model. The entropy of the token set is used to calculate the confidence weight of the teacher model, and the deviation weight of the student model is calculated based on the probability of the token selected by the teacher model. The two weights are then multiplied to obtain the penalty weight. The processor constructs an optimization objective function based on the penalty weight and the KL divergence between the probability distributions of the teacher and student models. This objective function performs a weighted sum of the KL divergences for each token and takes its expectation over all samples. The processor updates only the parameters of the student model according to the optimization objective function, while the parameters of the teacher model remain frozen. The training data for this method only contains query pairs between the target low-resource language query and the high-resource language version query, without any response data. This method achieves security capability transfer from the high-resource language to the target low-resource language without generating any response data, thus reducing data generation costs.

[0008] By instantiating teacher and student models from the same large language model and differentiating them through differentiated contextual conditions, the teacher model additionally receives queries and thought chain instructions in the high-resource language, stimulating the model's inherent secure reasoning ability in the high-resource language. The student model only receives queries in the target low-resource language, representing the model's secure ability in the low-resource language. Cross-language secure capability transfer is achieved through self-distillation training. The dual-view security weighting mechanism adaptively applies higher penalty weights to security-critical tokens and lower penalty weights to non-critical tokens by jointly evaluating teacher confidence and student deviation, improving the accuracy of security alignment. The method supports two self-distillation modes: online and offline. The online strategy samples the output sequence from the student model, and the teacher model provides real-time supervision of the output generated by the student, improving the accuracy of security alignment. The offline strategy samples the output sequence from the teacher model, and the teacher model parameter freeze provides a stable supervision signal. The method requires no response data at all, relying solely on multilingual query pairs to achieve security alignment, eliminating the problem of existing methods relying on high-cost response data generation.

[0009] Furthermore, the calculation process of the dual-view security weighting mechanism includes: calculating the confidence weight of the teacher model, which is calculated based on the normalized entropy of the teacher model's top-K token set; calculating the deviation weight of the student model, which is calculated based on the probability of the student model selecting the token by the teacher model; calculating the original weight of the t-th token, which is the product of the confidence weight and the deviation weight; normalizing the original weight to obtain a normalized penalty weight; and applying a stop-gradient operation to the normalized penalty weight during training. Through the joint evaluation of the confidence weight from the teacher's perspective and the deviation weight from the student's perspective, security-critical tokens can be accurately identified and subject to higher penalty weights, improving the accuracy and robustness of the distillation process.

[0010] Preferably, the K value in the top-K Token set is 16, 32, or 64, with a preferred K value of 32. This range of K values ​​ensures the accuracy of the teacher model's confidence assessment while avoiding the problem of the full vocabulary entropy calculation being diluted by long-tail probabilities. A K value of 32 achieves the optimal balance between security performance and versatility.

[0011] Furthermore, the sampling process of the offline strategy includes: sampling the output sequence from the teacher model; wherein the parameters of the teacher model are frozen during the sampling process to provide a stable supervision signal, and the offline strategy provides a stable supervision signal by freezing the parameters of the teacher model. The offline strategy can fully utilize the stable and secure capabilities of the teacher model in high-resource languages ​​to provide a high-quality supervision signal for the student model.

[0012] Furthermore, the sampling process of the online strategy includes sampling the output sequence from the student model; wherein the teacher model provides real-time supervision of each token generated by the student model, and the online strategy provides real-time supervision of the output sequence generated by the student model itself through the teacher model, thereby improving the accuracy of security alignment. By providing real-time supervision of the output generated by the student model itself, the online strategy can more accurately correct the unsafe behavior of the student model in low-resource languages.

[0013] Furthermore, the thought chain instruction guides the teacher model to perform secure reasoning in the high-resource language and generate a final response using the original language of the target low-resource language query. The thought chain instruction explicitly requires the teacher model to analyze the security of the query using the high-resource language; if the query is insecure, it should be politely rejected; if the query is secure, assistance should be provided, and the final response should always be generated using the original query language, effectively stimulating the model's inherent secure reasoning ability in the high-resource language.

[0014] Preferably, the high-resource language is English, or for models with a high proportion of Chinese pre-training data, the high-resource language is Chinese. English has the highest proportion of pre-training data for most large language models and has the strongest security protection capabilities; for models with a high proportion of Chinese pre-training data, using Chinese as the high-resource language can also achieve effective cross-language security capability transfer.

[0015] Furthermore, the training data comes from a multilingual security query dataset, which covers high-resource, medium-resource, and low-resource languages. The training data contains only multilingual query pairs and no response data. The multilingual security query dataset covers 14 categories of security issues, including 7 typical security scenarios, 6 command attack scenarios, and 1 common-sense security test. Translation quality is ensured through professional translation and proofreading, providing high-quality training data for cross-language security alignment.

[0016] This invention also provides a secure alignment system for a large language model based on multilingual self-distillation, comprising: a model instantiation module configured to instantiate a teacher model and a student model from the same large language model, wherein the teacher model and the student model have the same model structure and initial parameters, distinguished only by input context information, wherein the student model only receives queries in the target low-resource language, and the teacher model additionally receives queries in the high-resource language version and thought chain instructions; the model instantiation module includes a student context construction unit and a teacher context construction unit; a sampling module configured to sample output sequences from the teacher model or the student model according to a sampling strategy, wherein the sampling strategy includes an online strategy and an offline strategy; and a dual-view secure weighting module configured to calculate the penalty weight of each token, wherein the dual-view secure weighting module jointly evaluates the confidence weight of the teacher model and the... The deviation weights of the student model are described. The dual-view safety weighting module includes a top-K entropy calculation unit, a deviation calculation unit, and a weight normalization unit. The optimization module is configured to construct an optimization objective function based on the penalty weights and KL divergence, and update only the parameters of the student model according to the optimization objective function, while keeping the parameters of the teacher model frozen. The model instantiation module is connected to the sampling module via a data connection, and the model instantiation module outputs the model branches and context information of the teacher model and the student model to the sampling module via a data connection. The sampling module is connected to the dual-view safety weighting module via a data connection, and the sampling module outputs the output sequence and token-level probability distribution to the dual-view safety weighting module via a data connection. The dual-view safety weighting module is connected to the optimization module via a data connection.

[0017] The system implements a complete process of multilingual self-distillation secure alignment through modular design. The model instantiation module is responsible for instantiating teacher and student models from the same large language model and constructing differentiated contexts. The sampling module is responsible for sampling output sequences according to online or offline strategies. The dual-view secure weighting module is responsible for calculating token-level penalty weights. The optimization module is responsible for constructing the optimization objective function and updating student model parameters. Information is transferred between the modules through data connections, forming a complete data flow from input query to model parameter update.

[0018] Furthermore, the student context construction unit of the model instantiation module is configured to provide only the target low-resource language query to the student model, and the teacher context construction unit is configured to provide the target low-resource language query, the high-resource language version query, and the thought chain instruction to the teacher model; the top-K entropy calculation unit of the dual-view safety weighting module is configured to calculate the confidence weight of the teacher model, the deviation calculation unit is configured to calculate the deviation weight of the student model, and the weight normalization unit is configured to calculate the normalization penalty weight; the optimization module includes a KL divergence calculation unit, a weighted loss calculation unit, and a gradient update unit, and the gradient update unit is configured to update only the parameters of the student model according to the weighted loss. Through the refined division of labor within each module's internal units, efficient execution and precise control of multilingual self-distillation safety alignment are achieved.

[0019] Compared with the prior art, the present invention has the following beneficial effects: First, this invention instantiates teacher and student models from the same large language model and utilizes differentiated contextual conditions to achieve self-distillation training. This enables cross-language secure capability transfer without requiring any response data, eliminating the problem of existing methods relying on high-cost response data generation. Experimental data shows that existing supervised fine-tuning methods require generating high-quality response data for each target language, with data generation costs as high as $650, distillation methods cost $450, and preference optimization methods cost $1200. In contrast, the data generation cost of this invention is zero, significantly reducing the implementation cost of multilingual secure alignment.

[0020] Second, the dual-perspective security weighting mechanism proposed in this invention improves the accuracy of security alignment by jointly evaluating teacher confidence and student deviation, adaptively applying higher penalty weights to security-critical tokens and lower penalty weights to non-critical tokens. Experimental data show that on the Qwen-3-8B model, the average weight of security-critical tokens is 0.189, and that of non-critical tokens is 0.112; on the LLaMA-3-8B-Instruct model, the average weight of security-critical tokens is 0.182, and that of non-critical tokens is 0.117. After introducing the dual-perspective security weighting mechanism, the attack success rate of the online strategy decreased from 11.09% to 10.04% in the MultiJail benchmark test of the Qwen-3-8B model, and from 3.33% to 2.03% in the LLaMA-3-8B-Instruct model, demonstrating the effectiveness of the mechanism.

[0021] Third, this invention achieves superior security alignment performance in multiple representative large language models and multilingual jailbreak benchmarks, particularly demonstrating strong generalization ability on low-resource languages ​​not seen in the training set. In the MultiJail benchmark of the LLaMA-3-8B-Instruct model, the offline strategy reduced the attack success rate of Swahili from 69.52% in the original model to 6.03%, and the online strategy further reduced it to 4.13%; the attack success rate of Javanese was reduced from 39.37% to 11.11%, and the online strategy reduced it to 5.08%. In the PKU-SafeRLHF benchmark, the offline strategy reduced the attack success rate of Swahili from 65.20% to 3.20%, and the online strategy reduced it to 2.80%. These experimental results demonstrate that this invention can effectively transfer the security protection capabilities of high-resource languages ​​to low-resource languages ​​not seen in the training set, exhibiting strong cross-language generalization ability.

[0022] Fourth, while improving multilingual security performance, this invention effectively maintains the model's general knowledge and reasoning abilities without significant degradation. In the MMMLU multilingual knowledge understanding benchmark test, the LLaMA-3-8B-Instruct model, after online policy training, achieved an accuracy of 48.16%, essentially consistent with the original model's 48.17%. In the MGSM multilingual mathematical reasoning benchmark test, the accuracy after online policy training was 58.11%, higher than the original model's 56.04%. In comparison, the preference optimization method achieved an accuracy of 41.53% in MMMLU and 53.13% in MGSM on this model. This demonstrates that this invention can achieve multilingual secure alignment without compromising the model's general capabilities.

[0023] Fifth, this invention supports two self-distillation modes: online and offline, which can flexibly adapt to different application scenarios. The offline strategy samples the output sequence from the teacher model, and the frozen parameters of the teacher model provide a stable supervision signal, suitable for scenarios requiring stable training. The online strategy samples the output sequence from the student model, and the teacher model provides real-time supervision of the output generated by the student itself, suitable for scenarios requiring higher security alignment accuracy. Experimental data shows that both strategies outperform existing baseline methods, and the online strategy achieves better security performance in most cases. Attached Figure Description

[0024] Figure 1 A schematic diagram of the overall framework for multilingual secure alignment.

[0025] Figure 2 The diagram shows the process flow of the multilingual self-distillation method, where (a) is the main process flow diagram and (b) is the DPSW-KL objective function and dual-view safety weighting in the multilingual self-distillation method.

[0026] Figure 3 This is a system structure diagram. Detailed Implementation

[0027] The specific embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0028] Example 1 This embodiment provides a multilingual self-distillation method based on an offline strategy, such as... Figure 1 and Figure 2 As shown in (a). Figure 1 The overall framework for multilingual secure alignment is presented. The left side illustrates the security imbalance: the same large language model can correctly reject harmful content for English queries, but produces unsafe responses for Swahili and Javanese queries. The right side demonstrates the mechanism for achieving cross-language secure alignment through self-distillation.

[0029] Figure 2 Figure (a) illustrates the complete process of the multilingual self-distillation method. Specifically, teacher and student models are instantiated from the LLaMA-3-8B-Instruct model, both having the same model structure and initial parameters. The distinction is made solely based on the contextual information of the input. For Swahili queries, the student model only receives the Swahili query as input, along with the harmfulness of the Swahili input; the teacher model additionally receives the English version of the query, the harmfulness of the English input, and thought chain instructions.

[0030] The specific content of the thought chain instruction is: "You are a helpful assistant. User query: <Target low-resource language query>. The query in English is: <English query>. Instructions: 1. Analyze the query in English to determine if it violates safety guidelines. 2. If unsafe, generate a polite refusal. If safe, answer helpfully. Always respond in the same language as the original query." This instruction guides the teacher model to perform safe reasoning in English and generate a final response in the target language.

[0031] Both the teacher and student models are derived from the same large language model. Instantiation is differentiated based on different contextual conditions. The conditional probability distribution of the student model is defined as follows: That is, the student model uses the same parameters as the original large language model. Below, queries are performed only in the target low-resource language. Reasoning based on conditions… (Formula 1). The conditional probability distribution of the teacher model is defined as… That is, the teacher model has the same parameters. Below, additional searches in English. Reasoning is performed using the thought chain instruction C as a condition… (Formula 2). Both have the same model structure and initial parameters during initialization. During training, the parameters of the teacher model are kept frozen, while the parameters of the student model are updated according to the loss function. The two models differ in the contextual information of the input: the student model only receives queries in the target low-resource language, while the teacher model additionally receives queries in English and thought chain instructions, thereby stimulating the model's inherent safe reasoning ability in high-resource languages.

[0032] In offline strategy mode, the output sequence y is sampled from the teacher model (y is the complete output token sequence obtained from the sampling), that is, y follows a distribution. ,in This represents the conditional probability distribution of the teacher model. Targeting low-resource language queries, For English queries, C represents the thought chain instruction. The parameters of the teacher model are frozen during sampling to provide stable supervision signals.

[0033] For the sampled output sequence, a penalty weight for each token is calculated using a dual-view security weighting mechanism, such as... Figure 2 As shown in (b). Specifically, when generating the t-th Token, the confidence weight of the teacher model is first calculated. A top-K token set is extracted from the probability distribution of the teacher model; in this embodiment, K is set to 32. The probabilities within this set are renormalized to obtain a normalized probability distribution. The entropy of this normalized distribution is calculated and divided by... Normalization was performed, and the teacher confidence weights were adjusted. It equals 1 minus the normalized entropy value: ...(Formula 6). For example, for a certain security-critical token, the top-32 normalized entropy of the teacher model is 0.15, then... This indicates that the teacher model has a high degree of confidence in the token.

[0034] Then calculate the deviation weights of the student model. This weight measures the degree of deviation of the student model from the token selected by the teacher model. Let the token selected by the teacher model at step t be... This refers to the token with the highest probability in the teacher probability distribution. Student deviation weight. It equals 1 minus the student model's probability of that token: ...(Formula 7). This indicator captures the risk of student models being biased towards harmful generation. The higher the value, the greater the deviation of the student model from the token selected by the teacher.

[0035] Multiply the teacher confidence weights and student deviation weights to obtain the original weights. In the example above, The original weights of all tokens in the entire output sequence are normalized to obtain normalized penalty weights. ,in The summation range is from the first token in the output sequence to the second token. Each token. During training, a stop-gradient operation is applied to the normalized penalty weights, treating them as constant scalars.

[0036] To unify the representation of online and offline sampling methods, a generalized sampling distribution is introduced. Under the offline strategy, Equivalent to the sampling distribution of the teacher model, i.e. ... (Formula 4); Under the online strategy, Equivalent to the sampling distribution of the student model, i.e. ...(Formula 3). The output sequences generated under both strategies can be uniformly represented as... Based on the generalized sampling distribution, the basic training objective function of MSD without DPSW is defined as the function that performs all target low-resource language queries on all sample sets X. And the nested form of triple expectation on all sampled output sequences y: ... (Formula 5), ​​where Indicates the reverse Kullback-Leibler divergence. This represents the length of the output sequence. With the introduction of DPSW, the complete training objective function is upgraded to: Figure 2 The DPSW-KL objective function shown in (b) is: ... (Formula 8), where Let be the normalized penalty weight for the t-th token. During training... Apply a stop-gradient operation to treat it as a constant scalar, and it will not participate in the gradient calculation.

[0037] Update only the parameters of the student model according to the optimization objective function. Parameters of the teacher model Maintaining a frozen state means not updating gradients throughout the entire training process. This parameter freezing strategy ensures that the teacher model always provides a stable, high-resource language safety supervision signal, avoiding supervision signal drift and degradation of general capabilities caused by parameter updates.

[0038] The training data in this embodiment comes from the XSafety multilingual security query dataset, which covers 14 categories of security issues, including 7 typical security scenarios, 6 command attack scenarios, and 1 common-sense security test. The training data only contains query pairs of the target low-resource language query and its English version, without any response data. For example, for an Arabic query "harmful input in Arabic", the training data only provides the Arabic query and its English version "harmful input in English", without providing any secure or insecure responses. This method achieves security capability transfer from English to Arabic without generating any response data.

[0039] Example 2 This embodiment provides a multilingual self-distillation method based on an online strategy. The difference from Embodiment 1 is that the sampling strategy is changed to sample the output sequence from the student model, i.e., y follows a distribution. ,in Let represent the conditional probability distribution of the student model. Under this strategy, the student model autonomously generates each token in the output sequence, and the teacher model provides real-time supervision of each token generated by the student model. Specifically, at each generation step t, the teacher model, based on the prefixes already generated by the student... Additional information for teachers (English search) (and thought chain instruction C), calculate the probability distribution for this step. .

[0040] Example 3 This embodiment provides a multilingual self-distillation method using Chinese as a high-resource language. This method is suitable for models with a high proportion of pre-trained Chinese data, such as the Qwen-3-8B model. The difference from Embodiment 1 is that the additional information received by the teacher model is changed from English queries and English thought chain instructions to Chinese queries and Chinese thought chain instructions. Specifically, for the target low-resource language query... The teacher model receives the Chinese version of the query. and Chinese mind chain instructions The Chinese thought chain instruction is as follows: "You are a helpful assistant. User query: <Target low-resource language query>. The Chinese version of this query is: <Chinese query>. Instructions: 1. Analyze the query in Chinese to see if it violates security guidelines. 2. If it is unsafe, politely refuse; if it is safe, provide assistance. Always reply in the language of the original query." The teacher model performs secure inference based on Chinese queries and Chinese thought chain instructions, and generates the final response using the target low-resource language. Other technical details, such as the dual-view security weighting mechanism, optimization objective function, and parameter freezing strategy, are the same as in Example 1. This example also achieves cross-language security capability transfer without requiring any response data, making it suitable for large language models with a high proportion of pre-trained Chinese data.

[0041] Example 4 This embodiment provides a multi-model, multi-dataset comparative experiment to verify the effectiveness of the method of the present invention. The test conditions include: four representative large language models (Qwen-2.5-7B-Instruct, Qwen-3-8B, LLaMA-2-7B-Chat, LLaMA-3-8B-Instruct), two multilingual jailbreak benchmark tests (MultiJail and PKU-SafeRLHF), two general ability benchmark tests (MMMLU and MGSM), and ten languages ​​with different resource levels (high resource languages: English, Chinese, Italian, Vietnamese; medium resource languages: Arabic, Korean, Thai; low resource languages: Bengali, Swahili, Javanese).

[0042] The comparison methods include six baseline methods: supervised fine-tuning method SFT; preference optimization methods DPO, rDPO, KTO, ORPO, R-DPO, and SimPO; representation engineering method PolyRefuse; prompt-based method Self-Defense; distillation method SDRRL; and multilingual secure alignment method MPO. Additionally, the raw model was tested as a benchmark. All training methods were trained for one epoch on the XSafety dataset, using the same hyperparameter search space to ensure a fair comparison.

[0043] Table 1 shows a comparison of the safety performance of the MSD method and the baseline method on MultiJail and PKU-SafeRLHF.

[0044] Table 1 As shown in Table 1, both the offline and online MSD strategies outperform the baseline methods on all models and datasets. Particularly on low-resource languages, the MSD method significantly reduces the attack success rate. For example, for the LLaMA-3-8B-Instruct model in the MultiJail benchmark, the original model had an attack success rate of 69.52% on Swahili, which was reduced to 6.03% by the offline MSD strategy and further reduced to 4.13% by the online MSD strategy; the original model had an attack success rate of 39.37% on Javanese, which was reduced to 11.11% by the offline MSD strategy and further reduced to 5.08% by the online MSD strategy. In the PKU-SafeRLHF benchmark, the original model had an attack success rate of 65.20% on Swahili, which was reduced to 3.20% by the offline MSD strategy and further reduced to 2.80% by the online MSD strategy. This demonstrates that the method of this invention can effectively transfer the security protection capabilities of high-resource languages ​​to low-resource languages ​​not seen in the training set, exhibiting strong cross-language generalization ability.

[0045] Table 2 shows the performance comparison of each method on the general capability benchmark test.

[0046] Table 2 As shown in Table 2, both the offline and online MSD strategies effectively maintain the model's general knowledge and reasoning capabilities while improving multilingual security performance. For the LLaMA-3-8B-Instruct model, the DPO method achieves an MMMLU accuracy of 41.53% and an MGSM accuracy of 53.13%, while the online MSD strategy achieves 48.16% and 58.11% respectively, essentially maintaining and slightly exceeding the original model's 48.17% and 56.04%. This demonstrates that the method of this invention achieves multilingual secure alignment without compromising the model's general capabilities.

[0047] Table 3 shows a comparison of the response data generation costs of different methods.

[0048] Table 3 As shown in Table 3, traditional supervised fine-tuning methods require generating high-quality response data for each target language, with data generation costs as high as $650; distillation methods require translating English responses into the target language, costing $450; and preference optimization methods require generating pairs of safe and unsafe responses, costing up to $1200. In contrast, the MSD method of this invention requires no response data at all, with zero data generation cost, significantly reducing the implementation cost of multilingual safe alignment.

[0049] Example 5 This embodiment provides an ablation experiment using the Dual-Perspective Safety Weighting (DPSW) mechanism to verify its impact on multilingual security alignment performance. The test conditions are as follows: offline and online strategies with and without DPSW are tested on the Qwen-3-8B and LLaMA-3-8B-Instruct models, respectively. The attack success rate is evaluated on the MultiJail and PKU-SafeRLHF benchmark tests.

[0050] Table 4 shows the ablation experimental results of the DPSW mechanism.

[0051] Table 4 As shown in Table 4, the introduction of the DPSW mechanism reduced the attack success rate on both models and benchmarks, regardless of whether the strategy was offline or online. For example, for the Qwen-3-8B model, the attack success rate on MultiJail decreased from 11.09% to 10.04% after introducing DPSW on the online strategy, and from 9.10% to 7.62% on PKU-SafeRLHF. For the LLaMA-3-8B-Instruct model, the attack success rate on MultiJail decreased from 3.33% to 2.03% after introducing DPSW on the online strategy, and from 3.02% to 1.78% on PKU-SafeRLHF. This demonstrates that the DPSW mechanism can effectively improve the performance of multilingual security alignment.

[0052] To further verify the design rationality of the DPSW mechanism, 1000 queries were randomly sampled from the XSafety training set, and the first training step of MSD was run on Qwen-3-8B and LLaMA-3-8B-Instruct, recording the token-level weights. GPT-4o was used to segment each completed sequence generated by the student into continuous, non-overlapping token spans, labeled as "safety critical" (directly supporting security judgment, rejection stance, risk assessment) or "non-critical" (stylized, conjunction, general interpretation, filler words).

[0053] Table 5 shows a comparison of the average weights of security-critical tokens and non-critical tokens.

[0054] Table 5 Table 5 shows the average weight of security-critical tokens. Teacher's perspective weight Weight of student perspective Overall, it is higher than non-critical tokens. For example, in the Qwen-3-8B model, the security-critical token... The weight is 0.189 for key tokens and 0.112 for non-key tokens; weight from the teacher's perspective. The weights are 0.886 and 0.740 respectively; student perspective weights The values ​​are 0.253 and 0.199 respectively. In the LLaMA-3-8B-Instruct model, the security-critical token... The score was 0.182 for critical tokens and 0.117 for non-critical tokens. These results demonstrate that the DPSW mechanism can identify and impose higher penalty weights on critical tokens, validating the effectiveness of the joint assessment of teacher confidence and student deviation.

[0055] Example 6 This embodiment provides a sensitivity analysis experiment of the top-K parameter to verify the impact of the top-K parameter used in the teacher perspective weight calculation in the DPSW mechanism on performance. The test conditions are: on two models, Qwen-3-8B and LLaMA-3-8B-Instruct, with online policy MSD and K values ​​of 16, 32 and 64, respectively.

[0056] Table 6 shows a comparison of attack success rates under different K values.

[0057] Table 6 As shown in Table 6, all K values ​​achieved good security performance, with K=32 showing the best overall performance. For example, for the Qwen-3-8B model, the attack success rate on MultiJail was 10.04% with K=32, better than K=16 (10.69%) and K=64 (10.49%); for the LLaMA-3-8B-Instruct model, the attack success rate on MultiJail was 2.03% with K=32, better than K=16 (2.60%) and K=64 (2.28%). This indicates that the DPSW mechanism is robust to the K parameter, and K=32 is the preferred value. This parameter ensures the accuracy of teacher confidence assessment while avoiding the dilution of the full vocabulary entropy calculation by long-tail probabilities.

[0058] Example 7 This embodiment provides an ablation experiment combining the CoT instruction with an English query to verify the effectiveness of each component in the teacher model's additional information. The test conditions are as follows: on the LLaMA-3-8B-Instruct model, with the online policy MSD, four variants are tested: MSD w / GT (providing a true rejection response), MSD w / EN (providing only an English query), MSD w / CoT (providing only the CoT instruction), and MSD w / EN+CoT (the complete MSD, providing both the English query and the CoT instruction).

[0059] Table 7 shows the ablation experiment results of the combination of CoT command and English query.

[0060] Table 7 As shown in Table 7, the complete MSD method (MSD w / EN+CoT) has the lowest attack success rate on MultiJail and PKU-SafeRLHF, at 2.03% and 1.78% respectively, while maintaining the best generality (MMMLU accuracy of 48.16%). Providing only English queries (MSD w / EN) or only CoT instructions (MSD w / CoT) both perform worse than the complete MSD, indicating that both the English query and CoT components are important for stimulating the teacher model's high-resource language-safe reasoning ability. Directly providing true rejection responses (MSD w / GT) severely impairs the model's generality (MMMLU accuracy drops from 48.17% to 18.58%), suggesting that directly providing true rejection responses may lead to shortcut learning and information leakage. The MSD method of this invention avoids this problem through a self-distillation mechanism that does not require response data.

[0061] Example 8 This embodiment provides a comparative experiment on prompting languages ​​to verify the effectiveness of using different high-resource languages ​​(Chinese vs. English) as additional information for the teacher model. The test conditions are as follows: on the Qwen-3-8B model with a high proportion of Chinese pre-training data, the offline strategy and the online strategy MSD using Chinese (w / ZH) and English (w / EN) as high-resource languages ​​are tested respectively.

[0062] Table 8 shows the results of the comparative experiment on prompting language.

[0063] Table 8 As shown in Table 8, on the Qwen-3-8B model, the MSD method using English as the high-resource language outperforms the version using Chinese in terms of security performance. For example, the attack success rate on MultiJail is 10.04% when the online strategy uses English, which is better than 11.79% when using Chinese; the attack success rate on PKU-SafeRLHF is 7.62%, which is better than 9.44% when using Chinese. Meanwhile, the MSD method using Chinese as the high-resource language still significantly outperforms the best baseline method, demonstrating the flexibility of the present invention framework, where the high-resource language can be selected based on the distribution of the pre-training data of the target model.

[0064] Regarding hyperparameters Analysis, Strategies for controlling the update of teacher model parameters. This indicates that the teacher model parameters are completely frozen. This indicates that the teacher model parameters are updated using an exponential moving average (EMA). Experimental results show that... (Completely freezing teacher parameters) achieves the optimal balance between security and versatility, avoiding the degradation of high-resource language security capabilities and loss of versatility due to teacher parameter updates.

[0065] like Figure 3 As shown, this invention also provides a secure alignment system for large language models based on multilingual self-distillation. This system includes four core functional modules: a model instantiation module, a sampling module, a dual-view secure weighting module, and an optimization module. The model instantiation module includes a student context construction unit and a teacher context construction unit. The student context construction unit only provides the target low-resource language query to the student model, while the teacher context construction unit provides the target low-resource language query, high-resource language version query, and thought chain instructions to the teacher model. The sampling module includes a strategy selection unit and a sequence sampling unit. The strategy selection unit selects an online strategy or an offline strategy, and the sequence sampling unit samples the output sequence from the teacher model under the offline strategy, or samples the output sequence from the student model under the online strategy.

[0066] The dual-view safety weighting module includes a top-K entropy calculation unit, a deviation calculation unit, and a weight normalization unit. The top-K entropy calculation unit calculates the confidence weights of the teacher model. The deviation calculation unit calculates the deviation weights of the student model. The weight normalization unit calculates the normalized penalty weight. The optimization module includes a KL divergence calculation unit, a weighted loss calculation unit, and a gradient update unit. The KL divergence calculation unit calculates the KL divergence between the probability distributions of the student model and the teacher model. The weighted loss calculation unit multiplies the KL divergence by the normalized penalty weights to obtain the weighted loss. The gradient update unit updates only the parameters of the student model based on the weighted loss, while the parameters of the teacher model remain frozen.

[0067] Information is exchanged between modules via data connections. The model instantiation module outputs model branches and context information of the teacher and student models to the sampling module via data connections. The sampling module outputs the output sequence and token-level probability distribution to the dual-view safety weighting module via data connections. The dual-view safety weighting module outputs normalized penalty weights to the optimization module via data connections. The optimization module updates the student model parameters based on the weighted loss and updates the student model parameters in the model instantiation module via feedback connections.

[0068] It is understood that the high-resource language in the above embodiments is not limited to English or Chinese, but can also be other languages ​​with a high proportion in the model pre-training data, such as French or Spanish. Only the query language and CoT instruction language in the teacher's additional information need to be adjusted accordingly. Teacher perspective weights in the DPSW mechanism. The confidence metrics are not limited to top-K entropy; other confidence measures such as full vocabulary entropy, maximum probability value, and variance of probability distributions can also be used. KL divergence is not limited to inverse KL divergence; forward KL divergence, JS divergence, and other divergence measures can also be used. Training data is not limited to the XSafety dataset; other multilingual query datasets can be used, as long as they contain the correspondence between target language queries and high-resource language queries. The model architecture is not limited to LLaMA and Qwen series; it can also be applied to other large language models, such as Gemma and Mistral, as long as the model possesses basic safety protection capabilities and context learning capabilities in high-resource languages.

Claims

1. A secure alignment method for large language models based on multilingual self-distillation, the method being executed by a computing device including a processor and a memory, characterized in that, Includes the following steps: Step S1: The processor instantiates a teacher model and a student model from the same large language model. The teacher model and the student model have the same model structure and initial parameters, and are distinguished only by the input context information. The student model only receives queries for the target low-resource language, while the teacher model additionally receives queries for the high-resource language version and thought chain instructions. Step S2: The processor samples the output sequence from the teacher model or the student model according to a sampling strategy, the sampling strategy including an online strategy and an offline strategy; Step S3: Based on the output sequence, the processor calculates the penalty weight for each token using a dual-view security weighting mechanism. This mechanism calculates the confidence weight of the teacher model based on the entropy of the top-K token set of the teacher model, and calculates the deviation weight of the student model based on the probability of the token selected by the teacher model. The two weights are then multiplied to obtain the penalty weight. The dual-view security weighting mechanism applies a higher penalty weight to security-critical tokens and a lower penalty weight to non-critical tokens. Step S4: The processor constructs an optimization objective function based on the penalty weights and the KL divergence between the probability distributions of the teacher model and the student model. The optimization objective function performs a weighted summation of the KL divergence of each token and takes the expectation over all samples. Step S5: The processor updates only the parameters of the student model according to the optimization objective function, while the parameters of the teacher model remain frozen; The training data for the method contains only query pairs of the target low-resource language query and the high-resource language version query, and does not contain any response data.

2. The method according to claim 1, characterized in that, In step S3, the calculation process of the dual-view security weighting mechanism includes: Step S31: Calculate the confidence weight of the teacher model, which is based on the normalized entropy of the top-K token set of the teacher model; Step S32: Calculate the deviation weight of the student model, which is calculated based on the probability of the student model selecting the token by the teacher model; Step S33: Calculate the original weight of the t-th Token, where the original weight is the product of the confidence weight and the deviation weight; Step S34: Normalize the original weights to obtain normalized penalty weights; Step S35: Apply a stop-gradient operation to the normalized penalty weights during training.

3. The method according to claim 2, characterized in that, The K value in the top-K Token set is 16, 32, or 64.

4. The method according to claim 1, characterized in that, In step S2, the sampling process of the offline strategy includes: The output sequence is sampled from the teacher model; The parameters of the teacher model are kept frozen during the sampling process to provide a stable supervision signal, and the offline strategy provides a stable supervision signal by freezing the parameters of the teacher model.

5. The method according to claim 1, characterized in that, In step S2, the sampling process of the online strategy includes: Sample the output sequence from the student model; The teacher model provides real-time supervision of each token generated by the student model, and the online strategy provides real-time supervision of the output sequence generated by the student model itself through the teacher model, thereby improving the accuracy of security alignment.

6. The method according to claim 1, characterized in that, The thought chain instruction is used to guide the teacher model to perform secure reasoning in the high-resource language and generate a final response using the original language queried in the target low-resource language.

7. The method according to claim 1, characterized in that, The high-resource language is English, or for models with a high proportion of Chinese pre-trained data, the high-resource language is Chinese.

8. The method according to claim 1, characterized in that, The training data comes from a multilingual security query dataset, which covers high-resource, medium-resource, and low-resource languages. The training data contains only multilingual query pairs and does not contain any response data.

9. A secure alignment system for large language models based on multilingual self-distillation, characterized in that, include: The model instantiation module is configured to instantiate a teacher model and a student model from the same large language model. The teacher model and the student model have the same model structure and initial parameters, and are distinguished only by the input context information. The student model only receives queries for the target low-resource language, while the teacher model additionally receives queries for the high-resource language version and thought chain instructions. The model instantiation module includes a student context construction unit and a teacher context construction unit. A sampling module is configured to sample output sequences from the teacher model or the student model according to a sampling strategy, the sampling strategy including an online strategy and an offline strategy; A dual-view security weighting module is configured to calculate the penalty weight for each token. The dual-view security weighting module jointly evaluates the confidence weight of the teacher model and the deviation weight of the student model. The dual-view security weighting module includes a top-K entropy calculation unit, a deviation calculation unit, and a weight normalization unit. The optimization module is configured to construct an optimization objective function based on the penalty weights and KL divergence, and update only the parameters of the student model according to the optimization objective function, while keeping the parameters of the teacher model frozen; The model instantiation module is connected to the sampling module via a data connection. The model instantiation module outputs the model branches and context information of the teacher model and the student model to the sampling module via a data connection. The sampling module is connected to the dual-view security weighting module via a data connection. The sampling module outputs the output sequence and token-level probability distribution to the dual-view security weighting module via a data connection. The dual-view security weighting module is connected to the optimization module via a data connection.

10. The system according to claim 9, characterized in that, The model instantiation module includes a student context construction unit and a teacher context construction unit. The student context construction unit is configured to provide the target low-resource language query only to the student model, and the teacher context construction unit is configured to provide the target low-resource language query, the high-resource language version query, and the thought chain instruction to the teacher model. The dual-view safety weighting module includes a top-K entropy calculation unit, a deviation calculation unit, and a weight normalization unit. The top-K entropy calculation unit is configured to calculate the confidence weight of the teacher model, the deviation calculation unit is configured to calculate the deviation weight of the student model, and the weight normalization unit is configured to normalize the product of the two to obtain a normalized penalty weight. The optimization module includes a KL divergence calculation unit, a weighted loss calculation unit, and a gradient update unit. The gradient update unit is configured to update only the parameters of the student model based on the weighted loss, while the parameters of the teacher model remain frozen.

Images