Large language model security alignment training method and device, electronic equipment and medium
By using gradient adaptive loss functions and harmless data sets in training large language model, the problem of excessively rapid drop in harmful answer probability caused by jailbreak attacks is solved, and the generalization ability and normal task performance of the model are improved.
Patent Information
- Application Number
- CN202410859925.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-15
AI Technical Summary
When existing large language models reduce the probability of harmful answers in order to defend against jailbreak attacks during training, they cause problems in model optimization and affect normal task performance.
By collecting harmful data sets and adding them to the training data, the probability of the model generating harmful problems is statistically calculated, the loss function of gradient adaptation is calculated, the model parameters are optimized, and the general performance of the model is maintained in combination with harmless data sets to avoid overfitting.
It significantly improves the generalization ability of large language models, defends against jailbreak attacks, and maintains the performance of the model on normal tasks.
Smart Images

Figure CN118966299A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a large language model security alignment training method, device, electronic equipment and medium. Background Art
[0002] Large Language Models (LLMs) are machine learning models that can understand and generate natural language and other types of content after being trained with a large amount of data, thereby performing various tasks. In recent years, large language models have achieved impressive results in many fields, such as human-computer dialogue interaction and automatic writing; however, large language models have also exposed many security issues, such as providing guidance for harmful behaviors. It is particularly noteworthy that large language models are vulnerable to jailbreak attacks. Jailbreak attacks use carefully designed input prompts to make large language models answer harmful questions that they would not otherwise answer, allowing large language models to output harmful answers.
[0003] At present, the existing large language models mainly use the gradient ascent method to reduce the probability of harmful answers during training, such as the supervised fine-tuning baseline method (Supervised Fine-Tuning, SFT for short); but the problem with this method is that the probability of harmful answers drops too quickly, causing problems in model optimization, which in turn affects the performance of the model on normal tasks, and therefore continues to be improved. Summary of the invention
[0004] The present invention provides a large language model security alignment training method, device, electronic device and medium, which are used to solve the defect in the prior art that the answers to harmful questions are reduced during model training to defend against jailbreak attacks, thereby affecting normal business, and realize the defense against jailbreak attacks using a small amount of security-related fine-tuning data, and can well maintain the general performance of the model.
[0005] According to a first aspect of the present invention, the present invention provides a large language model security alignment training method, the method comprising: Collecting a harmful data set and adding it to the training data, each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question; Counting the probability of generating corresponding harmful answers for harmful questions in the harmful data set before the target large language model is trained, to obtain a first probability; Iteratively training the target large language model using the training data; After each round of iteration, the probability of the current target large language model generating harmful questions and corresponding harmful answers in the harmful data set is counted to obtain a second probability; calculating a first loss based on the first probability and the second probability; Optimize the parameters of the target large language model using the gradient of the first loss, and return to the step of iteratively training the target large language model using the training data until the iterative training is completed.
[0006] According to the large language model security alignment training method of the present invention, each element in the harmful data set also includes a rejection-free harmless answer corresponding to the harmful question. After each round of iteration, the method further includes: Assign the probability of harmful questions and corresponding harmful answers that decrease in this iteration compared to the previous iteration to the rejection-based harmless answers; The probability that the target large language model generates a rejection-innocuous answer in the harmful data set is counted to obtain a third probability; and the second loss is calculated based on the third probability.
[0007] According to the large language model security alignment training method of the present invention, before the step of iteratively training the target large language model using the training data, the method further includes: Collecting a harmless data set and adding it to the training data, wherein each element in the harmless data set includes a normal question and a normal answer corresponding to the normal question; After each round of iteration is completed, the method further includes: Counting the probability that the target large language model generates normal questions and corresponding normal answers in the harmless data set to obtain a fourth probability; A third loss is calculated based on the fourth probability.
[0008] According to the large language model security alignment training method of the present invention, the calculating of the first loss based on the first probability and the second probability includes: Substituting the first probability and the second probability into the following formula 1 to obtain a first loss; Among them, L h represents the first loss, D f represents a harmful dataset, |D f | represents the data set D f The total number of elements in , M represents the target large language model, M ref represents the target large language model before training, Indicates D f The harmful answer in the i-th element of Indicates D f The harmful problem in the i-th element of , σ represents the sigmoid function, β represents the hyperparameter, Represents model M refGenerate the first probability of harmful questions and corresponding harmful answers in the i-th element, represents the second probability of the harmful question and the corresponding harmful answer in the i-th element generated by model M; The gradient of the first loss is calculated according to the following formula 2 and formula 3; in, represents the gradient of the first loss, D f represents a harmful dataset, Indicates that D f expectations, represents the gradient of model M, P M (r h |q h ) represents the probability that model M generates corresponding harmful answers based on harmful questions, W M (r h ,q h ) represents the intermediate parameter, β represents the hyperparameter, represents the generation probability of model M, Represents model M ref The probability of generation.
[0009] According to the large language model security alignment training method of the present invention, the calculating the second loss based on the third probability includes: Substituting the third probability into the following formula 4 to obtain the second loss; Among them, L r represents the second loss, D f represents a harmful dataset, |D f | represents the data set D f The total number of elements in , Indicates D f The rejection-innocuous answer in the i-th element of , M represents the target large language model, It represents the third probability that model M generates the corresponding i-th rejection harmless answer based on the i-th harmful question.
[0010] According to the large language model security alignment training method of the present invention, the calculating of the third loss based on the fourth probability includes: Substituting the fourth probability into the following formula 5 to obtain the third loss; Among them, L g represents the third loss, D r represents a harmless dataset, represents the normal problem in the i-th element in the dataset, represents the normal answer in the i-th element in the data set, |D r | represents the total number of elements in the dataset Dr, M represents the target large language model, It represents the fourth probability that model M generates the corresponding i-th normal answer based on the i-th normal question.
[0011] According to the large language model security alignment training method of the present invention, before returning to the step of iteratively training the target large language model using the training data each time, the method further includes: Get the number of iterations in this round and compare it with the preset number of iterations; If the number of iterations in this round exceeds the preset number of iterations, it is confirmed that the iterative training is completed; If the number of iterations in this round does not exceed the preset number of iterations, calculating the overall loss based on the first loss, the second loss and the third loss; comparing the overall loss with a preset value; If the overall loss exceeds the preset value, it is confirmed that the iteration needs to be continued; If the overall loss does not exceed the preset value, it is confirmed that the iterative training is completed; The calculating of the overall loss based on the first loss, the second loss and the third loss includes: Substituting the first loss, the second loss and the third loss into the following formula 6 to obtain the second loss; L=γL g +θL r +αL h Formula 6; Among them, L represents the overall loss, L g represents the third loss, γ represents the weighted coefficient of the third loss, L r represents the second loss, θ represents the weighted coefficient of the second loss, L h represents the first loss, and α represents the weighting coefficient of the first loss.
[0012] According to a second aspect of the present invention, the present invention provides a large language model security alignment training device, the device comprising: A collection module, used for collecting harmful data sets and adding them to training data, wherein each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question; A first statistical module is used to calculate the probability of generating corresponding harmful answers from harmful questions in the harmful data set before the target large language model is trained, so as to obtain a first probability; An iteration module, used for iteratively training the target large language model using the training data; A second statistical module is used to count the probability of the current target large language model generating harmful questions and corresponding harmful answers in the harmful data set after each round of iteration to obtain a second probability; A calculation module, configured to calculate a first loss based on the first probability and the second probability; An optimization module is used to optimize the parameters of the target large language model using the gradient of the first loss, and return to the iteration module until the iterative training is completed.
[0013] According to the third aspect of the present invention, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the large language model security alignment training method as described above is implemented.
[0014] According to a fourth aspect of the present invention, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the large language model secure alignment training methods described above.
[0015] The present invention provides a large language model security alignment training method, which collects harmful data sets and adds them to training data, firstly counts the first probability of generating corresponding harmful answers from harmful questions in the harmful data sets before the model is trained, then uses the training data to iteratively train the target large language model, counts the second probability of the current target large language model generating data in the harmful data sets after each round of training, then uses the first probability and the second probability to calculate the first loss, which has a gradient adaptive effect, so that the target large language model can fully forget the harmful knowledge included in the harmful answers, and at the same time makes the training process more stable, significantly improving the generalization ability of the large language model.
[0016] In addition, a large language model security alignment training device, an electronic device and a non-transitory computer-readable storage medium provided by the present invention can also achieve the above-mentioned technical effects, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1It is a flowchart of the large language model security alignment training method provided by the present invention.
[0019] Figure 2 It is a schematic diagram of the implementation principle of forgetting harmful knowledge provided by the present invention.
[0020] Figure 3 This is a comparison chart of the results of training a large language model using the two learning methods provided by the present invention.
[0021] Figure 4 It is a schematic diagram of a large language model security alignment training device provided by the present invention.
[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] Combine the following Figures 1 to 5 The present invention describes a large language model security alignment training method, a large language model security alignment training device, an electronic device and a non-transitory computer-readable storage medium.
[0025] Figure 1 This is a flow chart of the large language model security alignment training method provided by the embodiment of the present invention. Please refer to Figure 1 As shown, this embodiment provides a large language model security alignment training method, which can be implemented by steps S101 to S106. Each step will be described in detail below: Step S101, collect a harmful data set and add it to the training data, each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question.
[0026] In this embodiment, the harmful data set can be obtained from the jailbreak attack records of the large language model, and the harmful data set can also be obtained in a customized way. The harmful questions and harmful answers in the harmful data set are paired. The harmful questions refer to questions that provide guidance for harmful behaviors. Exemplary harmful behaviors can be theft of private information, endangering personal freedom, etc. The harmful answers refer to the ways to achieve harmful behaviors that are searched from the database. Preferably, the total number of elements in the harmful data set is between 15 and 25.
[0027] Step S102, statistically analyzing the probability of harmful questions in the harmful data set generating corresponding harmful answers before the target large language model is trained, to obtain a first probability.
[0028] In this embodiment, the target large language model is the large language model to be trained. The large language model can be a model for performing any task. Exemplary tasks can be human-computer dialogue tasks, automatic writing, etc.
[0029] Step S103, iteratively training the target large language model using the training data.
[0030] Iterative training using training data means that the target large language model learns the training data in multiple rounds to continuously optimize performance. Usually, iterative training sets specific iteration stop conditions, and training can be terminated when the iteration stop conditions are met.
[0031] Step S104, after each round of iterations, the probability of the target large language model generating harmful questions and corresponding harmful answers in the harmful data set is counted to obtain a second probability.
[0032] Step S105: Calculate a first loss based on the first probability and the second probability.
[0033] Step S106, optimizing the parameters of the target large language model using the gradient of the first loss, and returning to step S103 to iteratively train the target large language model using the training data until the iterative training is completed.
[0034] The large language model security alignment training method of this embodiment collects harmful data sets and adds them to training data, first statistically generates a first probability of generating corresponding harmful answers for harmful questions in the harmful data sets before the model is trained, and then uses the training data to iteratively train the target large language model. After each round of training, the second probability of the current target large language model generating data in the harmful data sets is statistically generated, and then the first probability and the second probability are used to calculate a first loss. The loss has a gradient adaptive effect, so that the target large language model can fully forget the harmful knowledge included in the harmful answers, while making the training process smoother and significantly improving the generalization ability of the large language model.
[0035] In some possible implementations, each element in the harmful data set further includes a rejection-free harmless answer corresponding to the harmful question, and after each round of iteration is completed, the method further includes: Assign the probability of harmful questions and corresponding harmful answers that decrease in this iteration compared to the previous iteration to the rejection-based harmless answers; The probability that the target large language model generates a rejection-innocuous answer in the harmful data set is counted to obtain a third probability; and the second loss is calculated based on the third probability.
[0036] The large language model security alignment training method of this embodiment enables the target large language model to learn to reject harmful questions by adding rejection-based harmless answers corresponding to harmful questions to the harmful data set and allocating the probability of harmful replies to the rejection-based harmless answers, thereby allowing the model to smoothly reject harmful questions.
[0037] In some possible implementations, before the step of iteratively training the target large language model using the training data, the method further includes: Collecting a harmless data set and adding it to the training data, wherein each element in the harmless data set includes a normal question and a normal answer corresponding to the normal question; After each round of iteration is completed, the method further includes: Counting the probability that the target large language model generates normal questions and corresponding normal answers in the harmless data set to obtain a fourth probability; A third loss is calculated based on the fourth probability.
[0038] The large language model security alignment training method of this embodiment maintains the performance of the target large language model on general tasks by adding harmless data sets to the training data, avoids overfitting of the model, and enables the model to answer normal questions normally.
[0039] In some possible implementations, calculating the first loss based on the first probability and the second probability includes: Substituting the first probability and the second probability into the following formula 1 to obtain a first loss; Among them, L h represents the first loss, D f represents a harmful dataset, |D f | represents the data set D f The total number of elements in , M represents the target large language model, M ref represents the target large language model before training, Indicates D f The harmful answer in the i-th element of Indicates D f The harmful problem in the i-th element of , σ represents the sigmoid function, β represents the hyperparameter, Represents model M ref Generate the first probability of harmful questions and corresponding harmful answers in the i-th element, represents the second probability of the harmful question and the corresponding harmful answer in the i-th element generated by model M; The gradient of the first loss is calculated according to the following formula 2 and formula 3; in, represents the gradient of the first loss, Df represents the harmful dataset, represents the expectation on Df, represents the gradient of model M, P M (r h |q h ) represents the probability that model M generates corresponding harmful answers based on harmful questions, W M (r h ,q h ) represents the intermediate parameter, β represents the hyperparameter, represents the generation probability of model M, Represents model M ref Generate probability.
[0040] The large language model security alignment training method of this embodiment proposes a loss function with a gradient adaptive effect, which can effectively adaptively adjust the optimization speed and enable the model to fully forget the harmful knowledge included in the harmful answers.
[0041] In some possible implementations, calculating the second loss based on the third probability includes: Substituting the third probability into the following formula 4 to obtain the second loss; Among them, L r represents the second loss, D f represents a harmful dataset, |D f | represents the data set D f The total number of elements in , Indicates D f The rejection-innocuous answer in the i-th element of , M represents the target large language model, It represents the third probability that model M generates the corresponding i-th rejection harmless answer based on the i-th harmful question.
[0042] The large language model security alignment training method of this embodiment avoids the model from generating repeated and meaningless responses when facing harmful questions by allocating the probability of the model reducing the answers to harmful questions to rejection-based harmless responses.
[0043] In some possible implementations, calculating the third loss based on the fourth probability includes: Substituting the fourth probability into the following formula 5 to obtain the third loss; Among them, L g represents the third loss, D r represents a harmless data set, represents the normal problem in the i-th element in the dataset, represents the normal answer in the i-th element in the data set, |D r | represents the data set D r The total number of elements in , M represents the target large language model, It indicates that model M generates the fourth probability corresponding to the i-th normal answer based on the i-th normal question.
[0044] The large language model secure alignment training method of this embodiment defines a specific loss for the harmless dataset, and uses the loss to avoid overfitting of the model to the training data of the harmful dataset, thereby ensuring the model's performance on general tasks.
[0045] In some possible implementations, before returning to the step of iteratively training the target large language model using the training data each time, the method further includes: Get the number of iterations in this round and compare it with the preset number of iterations; If the number of iterations in this round exceeds the preset number of iterations, it is confirmed that the iterative training is completed; If the number of iterations in this round does not exceed the preset number of iterations, calculating the overall loss based on the first loss, the second loss and the third loss; comparing the overall loss with a preset value; If the overall loss exceeds the preset value, it is confirmed that the iteration needs to be continued; If the overall loss does not exceed the preset value, it is confirmed that the iterative training is completed; The calculating of the overall loss based on the first loss, the second loss and the third loss includes: Substituting the first loss, the second loss and the third loss into the following formula 6 to obtain the second loss; L=γL g +θL r +αL h Formula 6; Among them, L represents the overall loss, L g represents the third loss, γ represents the weighted coefficient of the third loss, L r represents the second loss, θ represents the weighted coefficient of the second loss, L hrepresents the first loss, and α represents the weighting coefficient of the first loss. Preferably, the weighting coefficient of the third loss is 1.
[0046] The large language model secure alignment training method of this embodiment can further avoid overfitting of the target large language model during the training process by pre-judging whether the number of iterations exceeds the preset number of iterations before returning to the next round of iteration each time, thereby ensuring the performance and efficiency of the model training; by judging whether the overall loss exceeds the set value, the performance of the target large language model in each round of training is measured by the overall loss, thereby realizing automatic judgment on whether the iterative training is completed.
[0047] In some possible implementations, in order to facilitate understanding of the solution of the present invention, it is useful to take a harmful data set consisting of 20 simple harmful questions that do not contain any jailbreak attack prompts as an example. This embodiment provides another large language model training method, and proposes SafeUnlearnin including forgetting harmful knowledge, learning to refuse to answer, and maintaining general performance. g For learning strategies, please refer to Figure 2 As shown, the implementation principle of the present invention is as follows: a forgetting loss function with a gradient adaptive effect, namely the first loss, is used to enable the model to fully forget the harmful knowledge included in the harmful answers. Two types of auxiliary loss functions, namely the second loss and the third loss, are also designed to achieve learning to refuse to answer harmful questions and learning normal answers to normal questions, thereby being able to maintain the general performance of the model well.
[0048] The implementation method for forgetting harmful knowledge is as follows: In order to make the model forget harmful knowledge, it is necessary to collect a harmful dataset Each element in the harmful dataset consists of a triplet, represents the harmful problem in the i-th triple in the dataset, r i h represents the harmful answer in the i-th triple in the dataset, Represents the rejection of harmless answers in the i-th triplet in the data set. Compared with the method of reducing the probability of harmful answers by gradient ascent, the problem with this method is that the probability of harmful answers decreases too quickly, causing problems in model optimization, thereby affecting the performance of the model on normal tasks. Therefore, the present invention uses a loss function that can adaptively adjust the gradient scale, which can slow down the speed at which the probability of harmful answers further decreases when the probability of harmful answers is already low. The loss function is expressed as: Where M is the large language model to be trained, and Mef is the large language model before training.
[0049] The gradient of this loss function is expressed as: It can be found that WM (r h ,q h ) will increase with the probability of harmful response generation P M (r h |q) is reduced, which can effectively play the role of adaptively adjusting the optimization speed, thereby making the training process more stable.
[0050] The implementation method for learning to refuse to answer is as follows: Considering that only the probability of harmful replies is reduced, the model will not know where to allocate this part of the reduced probability, which will cause the model to produce repeated and meaningless replies when facing harmful questions. Therefore, it is necessary to allocate this part of the reduced probability to the rejection of safe answers, so that the model can smoothly reject harmful questions. The corresponding loss function is expressed as:
[0051] The implementation method for maintaining general performance is as follows: Considering that training only on harmful questions by forgetting harmful knowledge and learning to refuse to answer will cause the model to overfit to this part of the data, thereby rejecting all inquiries (including normal and harmless inquiries), it is necessary to mix a part of the data containing harmless questions for training, so as to maintain the performance of the model on general tasks. Specifically, a harmless dataset that maintains general performance was collected. Each element in the harmless dataset consists of a pair of tuples. represents the normal question in the i-th bigram in the dataset, represents the normal answer in the i-th binary in the dataset. The loss function for training with this dataset is expressed as: Finally, the model training loss function is the weighted superposition of the above three loss functions, and the overall loss is expressed as L = L g +θL r +αL h , where θ and α are hyperparameters.
[0052] Please refer to Figure 3 As shown in the figure, taking the supervised fine-tuning baseline method for training a large language model as a reference, the model obtained by the supervised fine-tuning baseline method represented by the dotted arrow cannot be generalized to harmful problems that do not appear in the model, and cannot resist jailbreak attacks; while the method of the present invention uses a small amount of security-related fine-tuning data to effectively defend against jailbreak attacks, and can generalize to harmful problems and jailbreak attack problems that have not been seen in training. By superimposing the three loss functions of forgetting harmful knowledge, learning to refuse to answer, and maintaining general performance, it can better defend against various jailbreak attacks than the baseline method and can maintain general performance well.
[0053] The large language model security alignment training device provided by the present invention is described below. The large language model security alignment training device described below and the large language model security alignment training method described above can be referenced to each other.
[0054] Please refer to Figure 4 As shown, this embodiment provides a large language model security alignment training device, specifically, the device includes: a collection module 401, a first statistical module 402, an iteration module 403, a second statistical module 404, a calculation module 405 and an optimization module 406, and the above modules are described in detail below: A collection module 401 is used to collect a harmful data set and add it to the training data, wherein each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question; A first statistical module 402 is used to calculate the probability of generating corresponding harmful answers from harmful questions in the harmful data set before the target large language model is trained, to obtain a first probability; Iteration module 403, used for iteratively training the target large language model using the training data; A second statistical module 404 is used to count the probability of the current target large language model generating harmful questions and corresponding harmful answers in the harmful data set after each round of iteration to obtain a second probability; A calculation module 405, configured to calculate a first loss based on the first probability and the second probability; The optimization module 406 is used to optimize the parameters of the target large language model using the gradient of the first loss, and return to the iteration module 403 until the iterative training is completed.
[0055] The large language model security alignment training device of this embodiment collects harmful data sets and adds them to training data, first statistically generates a first probability of generating corresponding harmful answers for harmful questions in the harmful data sets before the model is trained, and then uses the training data to iteratively train the target large language model. After each round of training, statistically generates a second probability of the current target large language model generating data in the harmful data sets, and then uses the first probability and the second probability to calculate a first loss. The loss has a gradient adaptive effect, so that the target large language model can fully forget the harmful knowledge included in the harmful answers, while making the training process smoother and significantly improving the generalization ability of the large language model.
[0056] It should be noted that each module in the above-mentioned large language model security alignment training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0057] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the large language model security alignment training method, which includes: collecting a harmful data set and adding it to the training data, wherein each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question; counting the probability of generating a harmful answer corresponding to the harmful question in the harmful data set before the target large language model is trained, and obtaining a first probability; iteratively training the target large language model using the training data; after each round of iteration, counting the probability of the current target large language model generating a harmful question and a corresponding harmful answer in the harmful data set, and obtaining a second probability; calculating a first loss based on the first probability and the second probability; optimizing the parameters of the target large language model using the gradient of the first loss, and returning to the step of iteratively training the target large language model using the training data until the iterative training is completed.
[0058] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0059] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large language model security alignment training method provided by the above methods, which method includes: collecting a harmful data set and adding it to the training data, each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question; counting the probability of generating a harmful answer corresponding to the harmful question in the harmful data set before the target large language model is trained, and obtaining a first probability; iteratively training the target large language model using the training data; after each round of iteration, counting the probability of the current target large language model generating a harmful question and a corresponding harmful answer in the harmful data set, and obtaining a second probability; calculating a first loss based on the first probability and the second probability; optimizing the parameters of the target large language model using the gradient of the first loss, and returning to the step of iteratively training the target large language model using the training data until the iterative training is completed.
[0060] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the large language model security alignment training method provided by the above-mentioned methods, the method comprising: collecting a harmful data set and adding it to the training data, wherein each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question; counting the probability of generating a corresponding harmful answer from a harmful question in the harmful data set before the target large language model is trained, to obtain a first probability; iteratively training the target large language model using the training data; after each round of iteration, counting the probability of the current target large language model generating a harmful question and a corresponding harmful answer in the harmful data set, to obtain a second probability; calculating a first loss based on the first probability and the second probability; optimizing the parameters of the target large language model using the gradient of the first loss, and returning to the step of iteratively training the target large language model using the training data until the iterative training is completed.
[0061] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A large language model security alignment training method, characterized in that: The method comprises: Collecting a harmful data set and adding it to the training data, each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question; Counting the probability of generating corresponding harmful answers for harmful questions in the harmful data set before the target large language model is trained, to obtain a first probability; Iteratively training the target large language model using the training data; After each round of iteration, the probability of the current target large language model generating harmful questions and corresponding harmful answers in the harmful data set is counted to obtain a second probability; calculating a first loss based on the first probability and the second probability; Optimize the parameters of the target large language model using the gradient of the first loss, and return to the step of iteratively training the target large language model using the training data until the iterative training is completed.
2. The large language model security alignment training method according to claim 1, characterized in that: Each element in the harmful data set also includes a rejection harmless answer corresponding to the harmful question. After each round of iteration, the method further includes: allocating the probability of the harmful question and the corresponding harmful answer being reduced in the current round of iteration compared to the previous round of iteration to the rejection harmless answer; Counting the probability that the target large language model generates a rejection-innocuous answer in the harmful data set to obtain a third probability; A second loss is calculated based on the third probability.
3. The large language model security alignment training method according to claim 2, characterized in that: Before the step of iteratively training the target large language model using the training data, the method further includes: Collecting a harmless data set and adding it to the training data, wherein each element in the harmless data set includes a normal question and a normal answer corresponding to the normal question; After each round of iteration is completed, the method further includes: Counting the probability that the target large language model generates normal questions and corresponding normal answers in the harmless data set to obtain a fourth probability; A third loss is calculated based on the fourth probability.
4. The large language model security alignment training method according to claim 1, characterized in that: The calculating a first loss based on the first probability and the second probability includes: Substituting the first probability and the second probability into the following formula 1 to obtain a first loss; Among them, L h represents the first loss, D f represents a harmful dataset, |D f | represents the data set D f The total number of elements in , M represents the target large language model, M ref represents the target large language model before training, Indicates D f The harmful answer in the i-th element of Indicates D f The harmful problem in the i-th element of , σ represents the sigmoid function, β represents the hyperparameter, Represents model M ref Generate the first probability of harmful questions and corresponding harmful answers in the i-th element, represents the second probability of the harmful question and the corresponding harmful answer in the i-th element generated by model M; The gradient of the first loss is calculated according to the following formula 2 and formula 3; in, represents the gradient of the first loss, D f represents a harmful dataset, Indicates that D f expectations, represents the gradient of model M, P M (r h |q h ) represents the probability that model M generates corresponding harmful answers based on harmful questions, W M (r h ,q h ) represents the intermediate parameter, β represents the hyperparameter, represents the generation probability of model M, Represents model M ref The probability of generation.
5. The large language model security alignment training method according to claim 2, characterized in that: The calculating the second loss based on the third probability comprises: Substituting the third probability into the following formula 4 to obtain the second loss; Among them, L r represents the second loss, D f represents a harmful dataset, |D f | represents the data set D f The total number of elements in , Indicates D f The rejection-innocuous answer in the i-th element of , M represents the target large language model, It represents the third probability that model M generates the corresponding i-th rejection harmless answer based on the i-th harmful question.
6. The large language model security alignment training method according to claim 3, characterized in that: The calculating the third loss based on the fourth probability comprises: Substituting the fourth probability into the following formula 5 to obtain the third loss; Among them, L g represents the third loss, D r represents a harmless dataset, represents the normal problem in the i-th element in the dataset, represents the normal answer in the i-th element in the data set, |D r | represents the data set D r The total number of elements in , M represents the target large language model, It represents the fourth probability that model M generates the corresponding i-th normal answer based on the i-th normal question.
7. The large language model security alignment training method according to claim 3, characterized in that: Before returning to the step of iteratively training the target large language model using the training data each time, the method further includes: Get the number of iterations in this round and compare it with the preset number of iterations; If the number of iterations in this round exceeds the preset number of iterations, it is confirmed that the iterative training is completed; If the number of iterations in this round does not exceed the preset number of iterations, calculating the overall loss based on the first loss, the second loss and the third loss; comparing the overall loss with a preset value; If the overall loss exceeds the preset value, it is confirmed that the iteration needs to be continued; If the overall loss does not exceed the preset value, it is confirmed that the iterative training is completed; The calculating of the overall loss based on the first loss, the second loss and the third loss includes: Substituting the first loss, the second loss and the third loss into the following formula 6 to obtain the second loss; L=γL g +θL r +αL h Formula 6; Among them, L represents the overall loss, L g represents the third loss, γ represents the weighted coefficient of the third loss, L r represents the second loss, θ represents the weighted coefficient of the second loss, L h represents the first loss, and α represents the weighting coefficient of the first loss.
8. A large language model security alignment training device, characterized in that: The device comprises: A collection module, used for collecting harmful data sets and adding them to training data, wherein each element in the harmful data set includes a harmful question and a harmful answer corresponding to the harmful question; A first statistical module is used to calculate the probability of generating corresponding harmful answers from harmful questions in the harmful data set before the target large language model is trained, so as to obtain a first probability; An iteration module, used for iteratively training the target large language model using the training data; A second statistical module is used to count the probability of the current target large language model generating harmful questions and corresponding harmful answers in the harmful data set after each round of iteration to obtain a second probability; A calculation module, configured to calculate a first loss based on the first probability and the second probability; An optimization module is used to optimize the parameters of the target large language model using the gradient of the first loss, and return to the iteration module until the iterative training is completed.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the large language model security alignment training method as described in any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large language model security alignment training method as described in any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Machine forgetting learning method based on information retrieval and large language model driving
CN119848214A
A machine forgetting learning method based on information retrieval and large language model
CN119848214B
Model training method and device, electronic equipment and storage medium
CN120068983A
Multi-modal large language model question and answer method based on answer refusal
CN120705257A
