A large model protection training method based on reinforcement learning, and a storage medium and an electronic device

Abstract

The application discloses a large model protection training method based on reinforcement learning, a storage medium and an electronic device. The method comprises the following steps: inputting an attack request into a second large model after fine-tuning training to generate a plurality of output sequences; calculating the comprehensive reward of each output sequence according to an evaluation model; updating an exploration-exploitation coefficient tau according to the policy entropy and the comprehensive reward; calculating the advantage value of the attack request based on the comprehensive reward of the plurality of output sequences under the attack request, and taking the negative logarithm of the selection probability of each output sequence as the information quantity of the output sequence; determining the product of the sine function of the advantage value and the exploration-exploitation coefficient tau as a weighted coefficient, and updating the selection probability of the output sequence with the largest information quantity under the attack request based on the weighted coefficient; and stopping parameter updating and completing training when the preset convergence condition is met in the current round of reinforcement learning training. The application solves the problem that the existing large model has low security protection capability.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a large-scale model protection training method and storage medium and electronic device based on reinforcement learning. Background Technology

[0002] With the widespread application of large language models in scenarios such as dialogue assistants, code generation, and content creation, attackers can use carefully designed "jailbreak prompts" to induce models to bypass security policies and generate illegal, harmful, or sensitive content, posing serious risks to platform security and compliance. Current mainstream protection solutions mainly include pre-filtering and static security alignment. Pre-filtering typically uses sensitive word rules or violation classification models at the user input end to intercept obvious malicious prompts, but these methods struggle to detect and address risks that gradually emerge during the generation process. Static security alignment usually uses supervised fine-tuning or offline reinforcement learning to impose unified security constraints on the final output of the large language model, but it lacks dynamic monitoring of the intermediate inference process, making it prone to failure against complex jailbreak attacks. In other words, existing large language models face the problem of low security protection capabilities.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a reinforcement learning-based large model protection training method, storage medium, and electronic device to at least address the problem of low security protection capabilities faced by large models.

[0005] According to one aspect of the embodiments of this application, a large-scale model defense training method based on reinforcement learning is provided, comprising: constructing an attack dataset containing multiple attack methods; performing multi-round reinforcement learning training on a second large model through the following steps: fine-tuning the trained second large model by randomly selecting attack methods from the attack dataset as attack requests input to generate multiple output sequences; inputting the multiple output sequences into an evaluation model; calculating the thought reward and outcome reward of each output sequence according to the evaluation model, and linearly combining the thought reward and outcome reward of each output sequence according to preset weights, and determining the linear combination result of each output sequence as the comprehensive reward corresponding to the output sequence; updating the exploration utilization coefficient τ according to the policy entropy and the comprehensive reward; calculating the advantage value of the attack request based on the comprehensive reward of multiple output sequences under the attack request, and setting the value of each output sequence as the comprehensive reward of the attack request. The selection probability of a column is taken as the negative logarithm, which is used as the information content of the output sequence. The product of the sine function of the advantage value and the exploration utilization coefficient τ is determined as the weighting coefficient. The selection probability of the output sequence with the largest information content under the attack request is updated based on the weighting coefficient. If the training of this round of reinforcement learning meets the preset convergence condition, the parameter update is stopped and the training is completed. Each output sequence includes a security thought chain text and a security response text. The security thought chain text includes at least the intent analysis of the request, risk assessment, compliance check, and alternative solutions. The security response text is the security compliance text output to the user. The thought reward refers to the difference between the true value evaluation probability and the false value evaluation probability of the evaluation model for the security thought chain text. The result reward refers to the difference between the true value evaluation probability and the false value evaluation probability of the evaluation model for the security response text. The parameters include at least the exploration utilization coefficient τ and the selection probability.

[0006] Optionally, in this embodiment of the application, before constructing the attack dataset containing multiple attack methods, the method further includes: constructing a fine-tuning dataset based on the teacher model; adjusting the generated content of the second major model so that the generated content of the second major model includes security thought chain text and security response text; and fine-tuning the training of the second major model based on the fine-tuning dataset, wherein the teacher model is the first major model, and the fine-tuning dataset includes user requests, security thought chain text corresponding to user requests, and security response text.

[0007] Optionally, in this embodiment of the application, the above-mentioned construction of a fine-tuning dataset based on a teacher model includes: obtaining the user request and the first prompt word; inputting the user request and the first prompt word into the teacher model; the teacher model outputting the security thought chain text and the security response text; storing each user request, the security thought chain text corresponding to the user request, and the security response text as fine-tuning data to generate the fine-tuning dataset, wherein the user request includes normal requests and potentially malicious requests, and the first prompt word is used to prompt the teacher model to output the security thought chain text and the security response text.

[0008] Optionally, in this embodiment of the application, before updating the exploration utilization coefficient τ according to the policy entropy and the total reward, the method further includes: maintaining a value and an upper confidence bound exploration term for each of the multiple output sequences of the attack request; in each round of reinforcement learning training, calculating the selection probability of each output sequence under the attack method using a softmax function based on the sum of the value and the upper confidence bound exploration term of each output sequence of the attack method, wherein the value is used to represent the expected reward of selecting the current output sequence, and the upper confidence bound exploration term is used to represent the exploration reward of selecting other output sequences besides the current output sequence.

[0009] Optionally, in this embodiment, after each round of reinforcement learning training, the value of the output sequence is updated according to the following formula: in, Let r be the first preset weight value, r be the comprehensive reward corresponding to each of the multiple output sequences corresponding to the attack request, Var(r) be the variance of the multiple comprehensive rewards, and Mean(r) be the mean of the multiple comprehensive rewards. The value of the above output sequence in the i-th round of training.

[0010] Optionally, in this embodiment, updating the exploration utilization coefficient τ based on the policy entropy and the comprehensive reward includes: calculating the mean of all policy entropies prior to the current round of reinforcement learning training; determining the lower critical value by subtracting a preset value from the mean, and determining the upper critical value by adding the preset value to the mean; setting the penalty H to a negative value if the policy entropy corresponding to the current round of reinforcement learning training is greater than the upper critical value, and setting the penalty H to a positive value if the policy entropy corresponding to the current round of reinforcement learning training is less than the lower critical value; and updating the exploration utilization coefficient τ according to the following formula: Each round of reinforcement learning training corresponds to one of the aforementioned policy entropies. The second preset weight value, This is the difference between the average reward of this round of reinforcement learning training and the average reward of the previous round of reinforcement learning training. The average reward refers to the average of the sum of all the above-mentioned comprehensive rewards under the above-mentioned number of attack requests in this round of reinforcement learning training.

[0011] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the reinforcement learning-based large model protection training method provided in any of the foregoing embodiments when running.

[0012] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the reinforcement learning-based large model protection training method provided in any of the foregoing embodiments through the computer program.

[0013] In this embodiment, an attack dataset containing multiple attack methods is constructed. The second major model is trained through multiple rounds of reinforcement learning using the following steps: Attack methods randomly selected from the attack dataset are used as attack requests as input to fine-tune the trained second major model to generate multiple output sequences; these multiple output sequences are input into an evaluation model; the thinking reward and outcome reward for each output sequence are calculated based on the evaluation model, and the thinking reward and outcome reward for each output sequence are linearly combined according to preset weights. The linear combination result of each output sequence is determined as the comprehensive reward corresponding to the output sequence; the exploration utilization coefficient τ is updated based on the policy entropy and the comprehensive reward; the advantage value of the attack request is calculated based on the comprehensive reward of multiple output sequences under the attack request, and the negative logarithm of the selection probability of each output sequence is taken as the information content of the output sequence; the product of the sine function of the advantage value and the exploration utilization coefficient τ is determined as the weighting coefficient, and the selection probability of the output sequence with the largest information content under the attack request is updated based on the weighting coefficient; if the current round of reinforcement learning training meets the preset convergence condition, parameter updates are stopped, and training is completed. In other words, this application extends the protection of large models from input filtering to monitoring and constraining the entire generation process through explicit security thought chain text. Furthermore, by designing a comprehensive reward system for security thought chain text and security response text, it enhances security while ensuring useful responses to normal requests, achieving a balance between security and usability. This addresses the problem of low security protection capabilities faced by existing large models. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 This is a schematic diagram of an optional reinforcement learning-based large model protection training method provided according to an embodiment of this application;

[0016] Figure 2 This is a schematic diagram of another optional reinforcement learning-based large model protection training method according to an embodiment of this application;

[0017] Figure 3 This is a schematic diagram of another optional reinforcement learning-based large model protection training method according to an embodiment of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] Optionally, as an alternative implementation, the above-mentioned reinforcement learning-based large model protection training method includes:

[0021] Construct an attack dataset containing multiple attack methods;

[0022] The second model is trained through multiple rounds of reinforcement learning using the following steps:

[0023] The attack methods randomly selected from the aforementioned attack dataset are used as input to attack requests to fine-tune the trained second-largest model, thereby generating multiple output sequences.

[0024] Input multiple of the above output sequences into the evaluation model;

[0025] Based on the above evaluation model, the thinking reward and the result reward for each of the above output sequences are calculated, and the thinking reward and the result reward for each of the above output sequences are linearly combined according to a preset weight. The linear combination result of each of the above output sequences is determined as the comprehensive reward corresponding to the above output sequence.

[0026] The exploration utilization coefficient τ is updated based on the strategy entropy and the aforementioned comprehensive reward.

[0027] The advantage value of the attack request is calculated based on the comprehensive reward of the multiple output sequences under the above attack request, and the negative logarithm of the selection probability of each output sequence is taken as the information content of the output sequence.

[0028] The product of the sine function of the aforementioned advantage value and the aforementioned exploration and utilization coefficient τ is determined as the weighting coefficient. Based on the aforementioned weighting coefficient, the selection probability of the aforementioned output sequence with the largest information content under the aforementioned attack request is updated.

[0029] If the preset convergence condition is met in this round of reinforcement learning training, parameter updates will be stopped and training will be completed.

[0030] Each of the above output sequences includes a security thought chain text and a security response text. The security thought chain text includes at least an analysis of the intent of the request, a risk assessment, a compliance check, and alternative solutions. The security response text is a security compliance text output to the user. The thought reward refers to the difference between the probability of the evaluation model's true value assessment and the probability of its false value assessment for the security thought chain text. The result reward refers to the difference between the probability of the evaluation model's true value assessment and the probability of its false value assessment for the security response text. The parameters include at least the exploration utilization coefficient τ and the selection probability.

[0031] Optionally, in this embodiment, the reinforcement learning-based large-scale model protection training method can be applied to any domain where monitoring of the model's output is required. The evaluation model can be Qianwen Flash. The purpose of reinforcement learning training is to ensure the security of the large-scale model's output while meeting human expectations. The attack dataset includes at least the following attack methods: role-playing, translation bypass, step-by-step induction, context nesting, and format spoofing. During each round of reinforcement learning training, one attack method can be randomly selected from the attack dataset, or multiple attack methods can be randomly selected from the attack dataset, or a combination of various attack methods of different categories in the attack dataset can be used as one attack method. The selected attack method is adjusted to the input format required by the second large-scale model to serve as an attack request. The selection probability refers to the probability value of the output sequence as the final output result, and the policy entropy represents the difference between multiple selection probabilities. For example, a batch of user requests containing normal requests and potentially malicious requests is constructed, and corresponding prompt words are set so that the output of the teacher model contains security thought chain text and security response text; the above user requests, the security thought chain text and security response text corresponding to the user requests are used as training datasets and input into the second large-scale model for fine-tuning training to perform cold start. During each reinforcement learning training session, an attack method is randomly selected from an attack dataset containing multiple attack methods as the input to the finely tuned second-largest model. This second-largest model outputs multiple output sequences corresponding to the attack request. Each output sequence contains a security thought chain text and a security response text. An evaluation model is used to assess the security thought chain text and the security response text to output the thought reward and outcome reward corresponding to each output sequence. These rewards are then linearly combined with preset weights; the result of this linear combination is the comprehensive reward for that output sequence. The exploration utilization coefficient τ is updated based on the policy entropy and the comprehensive reward, and the advantage value of the attack request and the information content of each output sequence are calculated. The selection probability of the output sequence is updated based on the advantage value, the exploration utilization coefficient τ, and the information content. When the comprehensive reward or security evaluation metric reaches a preset convergence condition, parameter updates are stopped, training is complete, and a large-scale model with stable protection capabilities under various jailbreak attack scenarios is output.

[0032] In this embodiment, an attack dataset containing multiple attack methods is constructed. The second major model is trained through multiple rounds of reinforcement learning using the following steps: Attack methods randomly selected from the attack dataset are used as attack requests as input to fine-tune the trained second major model to generate multiple output sequences; these multiple output sequences are input into an evaluation model; the thinking reward and outcome reward for each output sequence are calculated based on the evaluation model, and the thinking reward and outcome reward for each output sequence are linearly combined according to preset weights. The linear combination result of each output sequence is determined as the comprehensive reward corresponding to the output sequence; the exploration utilization coefficient τ is updated based on the policy entropy and the comprehensive reward; the advantage value of the attack request is calculated based on the comprehensive reward of multiple output sequences under the attack request, and the negative logarithm of the selection probability of each output sequence is taken as the information content of the output sequence; the product of the sine function of the advantage value and the exploration utilization coefficient τ is determined as the weighting coefficient, and the selection probability of the output sequence with the largest information content under the attack request is updated based on the weighting coefficient; if the current round of reinforcement learning training meets the preset convergence condition, parameter updates are stopped, and training is completed. In other words, this application extends the protection of large models from input filtering to monitoring and constraining the entire generation process through explicit security thought chain text. Furthermore, by designing a comprehensive reward system for security thought chain text and security response text, it enhances security while ensuring useful responses to normal requests, achieving a balance between security and usability. This addresses the problem of low security protection capabilities faced by existing large models.

[0033] Based on one or more of the above embodiments, this application further improves upon them, such as... Figure 1 As shown, before constructing the attack dataset containing multiple attack methods, the following steps are also included:

[0034] S102, constructing a fine-tuning dataset based on the teacher model;

[0035] S104, Adjust the generated content of the second major model so that the generated content of the second major model includes security thought chain text and security response text;

[0036] S106, fine-tuning the training of the second major model based on the fine-tuning dataset, in which the teacher model is the first major model. The fine-tuning dataset includes user requests, the security thought chain text corresponding to the user requests, and the security response text.

[0037] Optionally, in the embodiments of this application, the teacher model refers to a pre-trained, higher-performing, but often more complex and computationally expensive model in the machine learning paradigm, used to guide the training of another model (called the student model).

[0038] Based on one or more of the above embodiments, this application further improves upon them, such as... Figure 2As shown, the fine-tuning dataset constructed based on the teacher model includes:

[0039] S202, Obtain user request and first prompt word;

[0040] S204, Input the user request and the first prompt word into the teacher model;

[0041] S206, The teacher model outputs a safety mindset text and a safety response text;

[0042] S208, each user request, the corresponding security thought chain text, and the security response text are stored as fine-tuning data to generate a fine-tuning dataset. The user requests include normal requests and potentially malicious requests. The first prompt word is used to prompt the teacher model to output the security thought chain text and the security response text. Based on one or more of the above embodiments, this application further improves upon the following: before updating the exploration utilization coefficient τ according to the policy entropy and total reward, it further includes: maintaining a value and an upper confidence bound exploration term for each of the multiple output sequences of each attack request; in each round of reinforcement learning training, based on the sum of the value and the upper confidence bound exploration term of each output sequence of the attack method, the selection probability of each output sequence under the attack method is calculated using the softmax function, where the value represents the expected reward of selecting the current output sequence, and the upper confidence bound exploration term represents the exploration reward of selecting other output sequences besides the current output sequence. Based on one or more of the above embodiments, this application further improves upon the following: after each round of reinforcement learning training, the value of the output sequence is updated according to the following formula: in, Let r be the first preset weight value, r be the comprehensive reward corresponding to each of the multiple output sequences corresponding to the attack request, Var(r) be the variance of the multiple comprehensive rewards, and Mean(r) be the mean of the multiple comprehensive rewards. The value of the output sequence in the i-th round of training. Based on one or more of the above embodiments, this application further improves upon them, such as... Figure 3 As shown, the exploration utilization coefficient τ is updated based on strategy entropy and comprehensive reward, including:

[0043] S302, calculate the mean of all policy entropies before this round of reinforcement learning training;

[0044] S304, the value obtained by subtracting a preset value from the mean is determined as the lower critical value, and the value obtained by adding a preset value to the mean is determined as the upper critical value;

[0045] S306. If the policy entropy corresponding to this round of reinforcement learning training is greater than the upper threshold, the penalty H is set to a negative value; if the policy entropy corresponding to this round of reinforcement learning training is less than the lower threshold, the penalty H is set to a positive value.

[0046] S308, Update the exploration utilization coefficient τ according to the following formula: Each round of reinforcement learning training corresponds to a policy entropy. The second preset weight value, This is the difference between the draw reward in this round of reinforcement learning training and the average reward in the previous round of reinforcement learning training. The average reward refers to the sum of all comprehensive rewards in this round of reinforcement learning training under the number of attack requests.

[0047] Optionally, in this embodiment, the preset rules for setting penalty H to a negative value and setting penalty H to a positive value according to the preset rules are as follows:

[0048] If the policy entropy is greater than the upper critical value, it indicates that the entropy is too high and a negative penalty needs to be imposed using formula (1):

[0049] H_penalty = (H_now - high_th) / (H_ref + 1e-8) (1)

[0050] When the policy entropy is greater than the upper threshold, it indicates that the entropy is too low and a positive reward needs to be given through formula (2):

[0051] H_penalty = (H_now - low_th) / (H_ref + 1e-8) (2)

[0052] Otherwise, except in the two cases mentioned above, set the value of H_penalty to zero.

[0053] Here, H_penalty represents the penalty H, H_now represents the current policy entropy, high_th represents the upper critical value, low_th represents the lower critical value, and H_ref represents the reference entropy. The reference entropy is set according to the actual distribution characteristics of the policy entropy or task experience and is used to standardize the deviation term of the entropy, so that the magnitude of H_penalty is more controllable (e.g., falling within the range of [-1,1] or [-0.1,0.1]), avoiding the penalty term being too large or too small due to the difference in the absolute value of the entropy.

[0054] An electronic device includes a memory and a processor, the memory storing a computer program, the processor being configured to perform the method described in any of the above embodiments via the computer program.

[0055] A computer-readable storage medium comprising a stored program, wherein the program, when executed by a processor, performs the method described in any of the above embodiments.

[0056] Optionally, as an alternative implementation, the above method is explained and illustrated by the following steps:

[0057] Step 1 includes the design of the output format for the security mindset chain.

[0058] In this step, the output format of the large language model is first defined as " <think> Security Mind Chain Text Content< / think> "Security response text content". Among them, <think> The content within the tag is the security thought chain section, which is used to break down user requests into multiple steps, analyze risk factors step by step, and make security decisions.< / think> The content following the tag is the safety response section, used to provide direct feedback to the user. Through a unified output format design, the thought process and the final result can be evaluated and optimized separately in subsequent reinforcement learning stages.

[0059] Step 2, constructing a fine-tuned dataset containing the security mindset chain.

[0060] In this step, one or more external teacher models are selected, and inference is performed on a pre-constructed fine-tuning dataset based on these teacher models to generate standard answers with secure thought chain text and secure response text. The instruction dataset can contain two main categories: normal user requests and potentially malicious user requests. Each user request corresponds to at least one secure thought chain text and one secure response text. The user requests, their corresponding secure thought chain texts, and the secure response texts are organized into fine-tuning samples according to the format of step 1, providing a data foundation for subsequent supervised fine-tuning of the large language model.

[0061] Step S3: Cold start of supervised fine-tuning training of the large language model.

[0062] In this step, the fine-tuned samples obtained in step 2 are used to train the large language model. Through supervised fine-tuning training, the large language model learns to first generate a security thought chain text when responding to user requests, and then generate a security response text. After this cold start phase, the large language model can output structured security thought chain text in most request scenarios, thus providing a stable and reliable structural premise for subsequent reinforcement learning based on security thought chains.

[0063] Step S4: Sampling and roll-out of jailbreak attack data based on curriculum reinforcement learning.

[0064] In this step, several typical jailbreak attack strategies and specific attack methods are predefined. Attack methods within each attack category include at least role-playing, translation bypass, step-by-step manipulation, contextual nesting, and format spoofing. Each specific attack method is considered an arm in a multi-armed slot machine, and each arm maintains a value representing its attack effectiveness and diversity, along with an upper confidence bound exploration term. In each round of reinforcement learning training, an attack method, based on a large language model, outputs multiple output sequences. Each output sequence includes a secure thought chain text and a secure response text. The sum of the current value and the upper confidence bound exploration term for each output sequence is calculated, and the selection probability of each output sequence is calculated using the softmax function. Then, the selected attack method drives the attacker model (e.g., Qianwen Flash) to construct an attack request. This attack request is input into the large language model, which has undergone supervised fine-tuning, to obtain roll-out data containing the secure thought chain text and the secure response text.

[0065] After each round of training, the value of each output sequence in the attack method is updated according to the comprehensive reward sequence r (obtained in subsequent step 5) corresponding to the multiple output sequences generated by a certain attack method. That is, each output sequence corresponds to a comprehensive reward, and the multiple comprehensive rewards corresponding to multiple output sequences constitute a comprehensive reward sequence r. The update formula (3) is as follows:

[0066] (3)

[0067] Where Var(r) is the variance of the comprehensive reward sequence r, and Mean(r) is the mean of the comprehensive reward sequence r, thereby encouraging the retention of attack methods that can produce large reward fluctuations without being too extreme, so that diverse and effective jailbreak samples can be continuously generated in subsequent training.

[0068] Step S5, Integrated Reward Construction and Exploration – Utilizing Adaptive Adjustment and Policy Optimization.

[0069] In this step, the roll-out data obtained in step S104 is first evaluated using an evaluation model (such as Qianwen Flash). Specifically, the attack request and its corresponding security thought chain text are input into the evaluation model. The evaluation model only outputs the probability corresponding to "true" or "false", and the difference between the probabilities of "true" and "false" is used as the thought reward. ,Right now This is used to measure whether the security mindset text has reasonably broken down the attack request, fully analyzed potential risks, and given an appropriate security judgment. The attack request and its corresponding security response text are input into the evaluation model, which outputs only the probability of "true" or "false," with the difference between the probabilities of "true" and "false" serving as the reward. , This approach aims to prioritize the usefulness and completeness of the security thought process text for benign requests, while focusing on whether the security thought process text strictly avoids outputting harmful information and provides necessary warnings for malicious requests. Then, the thought reward and outcome reward are linearly combined according to a preset weight λ to obtain a comprehensive reward. .

[0070] After obtaining the comprehensive reward, an exploration and utilization coefficient τ is introduced for adaptive adjustment. The exploration and utilization coefficient τ ranges from [-1, 1]. When τ > 0, it promotes an increase in policy entropy in the loss function, thereby increasing exploration; when τ < 0, it promotes a decrease in policy entropy, thereby favoring utilization. An entropy penalty term is constructed by statistically analyzing the mean of the current policy entropy and the historical policy entropy during the training process. When the current entropy is higher than the upper critical value based on the average entropy of several steps in the early training phase, let... It becomes negative when the current entropy is below the lower critical value. The value is positive; at the same time, the difference between the most recent average reward time window and the previous average reward time window is calculated to construct the reward trend term ΔR. When the most recent average reward increases, ΔR is set to positive, and when the most recent average reward decreases, ΔR is set to negative. Here, the average reward refers to the average of the sum of all the comprehensive rewards under the current round of reinforcement learning training under the number of attack requests. Finally, the exploration utilization coefficient τ is updated according to formula (4) so ​​that it dynamically adjusts the exploration intensity according to the training state:

[0071] (4)

[0072] in, H represents punishment. This is the difference between the average reward of this round of reinforcement learning training and the average reward of the previous round of reinforcement learning training. This is the second preset weight value.

[0073] In terms of strategy optimization, the advantage value A of the attack request is calculated based on the comprehensive reward of multiple output sequences under the attack request, and an information weighting mechanism is introduced under the Distributed Advantage Policy Optimization (DAPO) framework. Specifically, for each output sequence, the information content of the output sequence under the current policy is calculated according to formula (5). :

[0074] (5)

[0075] in, Let A represent the selection probability of the output sequence, and t represent the t-th output sequence. The greater the information content, the rarer the action corresponding to the output sequence is under the current policy, and the more it reflects the exploratory nature. Based on the sign of the advantage value A and the updated exploration utilization coefficient τ, the weighting coefficient of the t-th output sequence is calculated using formula (6). :

[0076] (6)

[0077] The policy gradient of the output sequence is weighted using information content and weighting coefficients: when the advantage value is positive and τ>0, the gradient weights of tokens with higher information content are amplified to strengthen the impact of key decisions in successful trajectories; when the advantage value is negative and τ>0, the gradient weights of output sequences with higher information content are suppressed to correct bad explorations; when τ<0, the overall trend shrinks towards exploitation. By redistributing the distributed advantage through information weighting in this way, an improved policy optimization loss function is constructed to update the parameters of the large language model.

[0078] During training, steps 4 and 5 are repeated until the average reward or security assessment metric reaches the preset convergence condition, ultimately obtaining a large language model with stable protection capabilities under various jailbreak attack scenarios. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0079] This application implements all or part of the processes in the above embodiments, which can be accomplished by a computer program instructing related hardware. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / electronic device, recording media, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0080] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0081] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0082] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device controller embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A large model protection training method based on reinforcement learning, characterized in that, include: Construct an attack dataset containing multiple attack methods; The second model is trained through multiple rounds of reinforcement learning using the following steps: The attack method randomly selected from the attack dataset is used as the input of the attack request to fine-tune the trained second model to generate multiple output sequences; Input multiple of the output sequences into the evaluation model; The thinking reward and the result reward for each output sequence are calculated based on the evaluation model, and the thinking reward and the result reward for each output sequence are linearly combined according to a preset weight. The linear combination result of each output sequence is determined as the comprehensive reward corresponding to the output sequence. The exploration utilization coefficient τ is updated based on the policy entropy and the comprehensive reward, specifically including: calculating the mean of all policy entropies before this round of reinforcement learning training; determining the lower critical value by subtracting a preset value from the mean, and determining the upper critical value by adding the preset value to the mean; if the policy entropy corresponding to this round of reinforcement learning training is greater than the upper critical value, setting the penalty H to a negative value according to a preset rule; if the policy entropy corresponding to this round of reinforcement learning training is less than the lower critical value, setting the penalty H to a positive value according to the preset rule; and updating the exploration utilization coefficient τ according to the following formula: wherein each round of reinforcement learning training corresponds to one of the policy entropy, is a second preset weight value, is a difference between an average reward of the current round of reinforcement learning training and the average reward of the previous round of reinforcement learning training, the average reward referring to a mean value of a sum of all the comprehensive rewards under the attack request times in the current round of reinforcement learning training; The advantage value of the attack request is calculated based on the comprehensive reward of multiple output sequences under the attack request, and the negative logarithm of the selection probability of each output sequence is taken as the information content of the output sequence. The product of the sine function of the advantage value and the exploration utilization coefficient τ is determined as the weighting coefficient, and the selection probability of the output sequence with the largest information content under the attack request is updated based on the weighting coefficient; If the preset convergence condition is met in this round of reinforcement learning training, parameter updates will be stopped and training will be completed. Each output sequence includes a security thought chain text and a security response text. The security thought chain text includes at least an analysis of the request's intent, a risk assessment, a compliance check, and alternative solutions. The security response text is a security compliance text output to the user. The thought reward refers to the difference between the probability of the evaluation model evaluating the security thought chain text as true and the probability of it as false. The result reward refers to the difference between the probability of the evaluation model evaluating the security response text as true and the probability of it as false. The parameters include at least the exploration utilization coefficient τ and the selection probability. After each round of reinforcement learning training, the value of the output sequence is updated according to the following formula: in, Let r be the first preset weight value, r be the comprehensive reward corresponding to each of the multiple output sequences corresponding to the attack request, Var(r) be the variance of the multiple comprehensive rewards, and Mean(r) be the mean of the multiple comprehensive rewards. The value of the output sequence in the i-th round of training.

2. The method according to claim 1, characterized in that, Before constructing the attack dataset containing multiple attack methods, the following steps are also included: A fine-tuning dataset was constructed based on a teacher model; Adjust the generated content of the second major model so that the generated content of the second major model includes the security thought chain text and the security response text; The second major model is fine-tuned and trained based on the fine-tuning dataset, wherein the teacher model is the first major model, and the fine-tuning dataset includes user requests, the security thought chain text corresponding to the user requests, and the security response text.

3. The method according to claim 2, characterized in that, The fine-tuning dataset constructed based on the teacher model includes: Obtain the user request and the first prompt word; input the user request and the first prompt word into the teacher model; The teacher model outputs the safety thought chain text and the safety response text; Each user request, the corresponding security thought chain text, and the security response text are stored as fine-tuning data to generate the fine-tuning dataset. The user requests include normal requests and potentially malicious requests. The first prompt word is used to prompt the teacher model to output the security thought chain text and the security response text.

4. The method according to claim 1, characterized in that, Before updating the exploration utilization coefficient τ based on the policy entropy and the comprehensive reward, the method further includes: For each of the multiple output sequences of the attack request, maintain a value and an upper confidence bound exploration term respectively; In each round of reinforcement learning training, the selection probability of each output sequence under the attack method is calculated by the softmax function based on the sum of the value of each output sequence of the attack method and the upper confidence bound exploration term, wherein the value is used to represent the expected reward of selecting the current output sequence, and the upper confidence bound exploration term is used to represent the exploration reward of selecting other output sequences besides the current output sequence.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the method described in any one of claims 1 to 4.

6. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 4 through the computer program.

Images