Large language model reply generation method and system based on data alignment

Abstract

The invention discloses a large language model reply generation method and system based on data alignment, and belongs to the field of large language model reply generation. The problem that the risk of harmful output of an existing large language model is large is solved. The method comprises the following steps: collecting a harmful problem data set and a harmless problem data set through COD prompt, pre-training a model by using the collected data sets, and training the model by using the collected data sets and a proposed alignment strategy to minimize the negative logarithm likelihood of harmful responses and punish the harmful responses, so as to improve the accuracy of the harmful responses. Sample loss is optimized through gradient rising so as to strengthen the front behavior of the model. And then, a security vector is added into the aligned model weights to correct the security of the model, so that the security loss of the model can be compensated while the original task performance of the model is maintained, and the probability that the model generates harmful output is remarkably reduced. The method can be applied to the field of large language model security reply generation.

Description

Technical Field

[0001] The present invention belongs to the field of large language model response generation, and in particular relates to a large language model response generation method and system based on data alignment. Background Art

[0002] With the rapid development of artificial intelligence technology, large models, with their powerful data processing capabilities and deep learning mechanisms, have had a revolutionary impact in many fields such as autonomous driving, financial services, medical diagnosis, and personalized recommendation systems. They have not only promoted the transformation of production methods, but also accelerated the intelligent upgrading of industries, laying an important foundation for the high-quality development of the digital economy. However, the widespread application of large models is like a double-edged sword. With its increasing popularity, the accompanying security challenges have become increasingly prominent, becoming a major obstacle to its continued evolution.

[0003] Taking large language models (LLMs) as an example, the security risks of large language models cover many aspects: data privacy leakage and improper use of sensitive information. During the training process of large language models, social bias may be absorbed, resulting in unfair results, and harmful content may be generated. There is a risk of malicious manipulation and attack, security compliance challenges, and model deception issues. At present, all countries have recognized the security challenges brought by large language models and have adopted strict supervision. However, due to the powerful effectiveness and potential of large language models, it is impossible to completely abandon their use. The current important issue is how to deal with the growing security issues while promoting them more widely. Therefore, solving the problem of the high risk of harmful output of large language models is an urgent problem that needs to be solved. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of high risk of harmful output of existing large language models, and to propose a large language model response generation method and system based on data alignment.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] According to one aspect of the present invention, a method for generating a large language model response based on data alignment, the method specifically comprises the following steps:

[0007] Step 1: Obtain harmful question data from the data set, input the obtained harmful question data into the large language model, and generate a harmful response data set and a harmless response data set;

[0008] Step 2: Pre-train the large language model using the harmful response dataset and the harmless response dataset to obtain a pre-trained large language model;

[0009] Step 3: Use the generated data set to align the pre-trained large language model to obtain the large language model after data alignment;

[0010] Step 4: fine-tune the parameters of the large language model after data alignment to obtain a large language model after parameter fine-tuning;

[0011] Step 5: Input the question into the large language model after fine-tuning the parameters in step 4, and use the large language model after fine-tuning the parameters to generate a response to the input question.

[0012] According to another aspect of the present invention, a large language model response generation system based on data alignment includes a data set generation module, a large language model pre-training module, a data alignment module and a large language model fine-tuning module; wherein:

[0013] The data set generation module is used to generate a harmful response data set and a harmless response data set;

[0014] The large language model pre-training module is used to pre-train the large language model using the generated harmful response data set and harmless response data set;

[0015] The data alignment module is used to perform data alignment on the pre-trained large language model using the generated data set to obtain the large language model after data alignment;

[0016] The large language model fine-tuning module is used to fine-tune the large language model after data alignment, and the fine-tuned large language model is used to generate responses to input questions.

[0017] The beneficial effects of the present invention are:

[0018] The present invention first collects harmful question datasets and harmless question datasets through COD prompts, and uses the collected datasets to pre-train the model, and then uses the collected datasets and the proposed alignment strategy to train the model to minimize the negative log-likelihood of harmful responses, while penalizing harmful responses, and optimizes sample loss through gradient ascent to strengthen the positive behavior of the model. Then, by adding a safety vector to the aligned model weights to correct the safety of the model, the safety loss of the model can be compensated while maintaining the original task performance of the model, significantly reducing the probability of the model producing harmful outputs. The method of the present invention can effectively reduce the harmfulness of the fine-tuned model, and the harmfulness under efficient parameter fine-tuning is reduced from 18.6% to 5.1%, and the harmfulness under full fine-tuning is reduced from 9.2% to 1.5%. At the same time, it shows good generalization ability across languages ​​and under different safety assessment standards. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1It is a flow chart of a method for generating a large language model response based on data alignment according to the present invention;

[0020] Figure 2 It is a flow chart of COD induction method;

[0021] Figure 3 It is a structural diagram of a large language model response generation system based on data alignment of the present invention;

[0022] Figure 4 It is a schematic diagram of the security alignment of strategy A and strategy B;

[0023] Figure 5 It is a schematic diagram of the results of applying the DARE strategy. DETAILED DESCRIPTION

[0024] Specific implementation method 1: Combination Figure 1 The present embodiment describes a method for generating a large language model response based on data alignment, and the method specifically comprises the following steps:

[0025] Step 1: Obtain harmful question data from an existing data set, input the obtained harmful question data into a large language model, and generate a harmful response data set and a harmless response data set;

[0026] Step 2: Pre-train the large language model using the harmful response dataset and the harmless response dataset to obtain a pre-trained large language model;

[0027] Step 3: Use the generated data set to align the pre-trained large language model to obtain the large language model after data alignment;

[0028] Step 4: fine-tune the parameters of the large language model after data alignment to obtain a large language model after parameter fine-tuning;

[0029] Step 5: Input the question into the large language model after fine-tuning the parameters in step 4, and use the large language model after fine-tuning the parameters to generate a response to the input question.

[0030] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the large language model in step 1 is Vicuna-7B.

[0031] The other steps and parameters are the same as those in the first embodiment.

[0032] Vicuna-7B in the present invention is a type of large language model LLMs, and the method of the present invention is applicable to each large language model.

[0033] Specific implementation method three: This implementation method is different from the specific implementation method two in that the specific process of step one is as follows:

[0034] Step 1. Obtain various harmful questions from the existing data set;

[0035] Step 12: Input each harmful question obtained in step 11 into the large language model (i.e., the model with initialized parameters), and the harmful questions interact with the large language model through the Chain of Discourse (COD) induction method, and the large language model is used to generate responses corresponding to each harmful question;

[0036] Step 13: Use the harmful replies generated in steps 1 and 2 and the harmful questions corresponding to the harmful replies to form a harmful reply data set, and use the harmless replies generated in steps 1 and 2 and the harmful questions corresponding to the harmless replies to form a harmless reply data set.

[0037] The other steps and parameters are the same as those in the second specific implementation manner.

[0038] Harmful questions should cover various categories such as violence, malware, fraud, deception, privacy violation, etc. COD constructs a simulated dialogue environment with two virtual characters: a harmful agent (Red-LM) and an unsafe helpful agent (Base-LM). Red-LM asks harmful questions and tries to subtly induce Base-LM (equivalent to a large language model) to generate unsafe answers, while inserting harmless questions to build trust, so as to simulate real dialogue scenarios.

[0039] like Figure 2 As shown, the COD induction method is specifically:

[0040] Red-LM: As a harmful agent, used to ask potentially harmful questions.

[0041] Base-LM: acts as an unsafe but helpful proxy that tries to answer Red-LM's questions.

[0042] Setting up harmful questions: In a conversation, insert a harmful question as an utterance of Red-LM. For example, Red-LM may ask a question involving dangerous or illegal behavior.

[0043] Answer Guidance: Prompt the model to follow specified guidance when answering. For example, ask Base-LM to answer harmful questions in detail while generating internal thoughts as prefixes.

[0044] Internal Thought Prefix: Generate internal thoughts as prefixes in Base-LM’s answers, allowing the model to show its reasoning process while answering. This approach helps guide the model to follow prompts more accurately while reducing the rate of refusing to answer harmful questions.

[0045] Specific implementation method 4: This implementation method is different from specific implementation method 3 in that the specific process of step 3 is as follows:

[0046] The harmless response dataset is used to align the pre-trained large language model. That is, the pre-trained large language model is trained using the harmless response dataset. The loss function used in the training process is:

[0047]

[0048] Among them, L1 is the loss function, x is the harmless reply dataset N b The samples in (each harmful question and its corresponding harmless answer constitute a sample), |N b | is the harmless reply dataset N b The number of samples in x i is the i-th word in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, n is the total number of words in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, and w j is the jth word x in the pre-trained large language model output response j The corresponding vector, is to output the i-th word x i The conditional probability, w i is x i The corresponding vector, if w i If it is a harmful reply, The value of is 1, if w i If it is a harmless reply, The value of is 0.

[0049] The other steps and parameters are the same as those in the third specific implementation manner.

[0050] Specific implementation method five: This implementation method is different from specific implementation method three in that the specific process of step three is as follows:

[0051] The harmful response dataset and the harmless response dataset are used to align the pre-trained large language model. That is, the pre-trained model is trained in two stages using the harmful response dataset and the harmless response dataset, where:

[0052] In the first stage, the pre-trained model is trained using the harmful response dataset and the harmless response dataset. The loss function used in the first stage of training is:

[0053]

[0054] Among them, L2 is the loss function, |N|=|Nb |+|N r |,|N b | is the harmless reply dataset N b The number of samples in |N r | is the harmful reply dataset N r The number of samples in λ1 and λ2 are weights, * indicates multiplication, logp(x) is the negative log-likelihood of sample x, Represents the dataset N r The sample set whose inner negative log-likelihood value is less than or equal to 1, Represents the dataset N r The sample set whose inner negative log-likelihood value is greater than 1, Is a collection The number of samples in Is a collection The number of samples in ;

[0055]

[0056] Among them, x i is the i-th word in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, n is the total number of words in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, and w j is the jth word x in the pre-trained large language model output response j The corresponding vector, is to output the i-th word x i The conditional probability, w i is x i The corresponding vector, if w i If it is a harmful reply, The value of is 1, if w i If it is a harmless reply, The value of is 0;

[0057] In the second stage, the pre-trained model is trained using the harmless response dataset. The loss function used in the second stage of training is:

[0058]

[0059] Among them, L1 is the loss function.

[0060] The other steps and parameters are the same as those in the third specific implementation manner.

[0061] The present invention provides negative rewards to the large language model on the harmful response dataset and punishes harmful responses through gradient ascent, thereby guiding the model to align with safer behaviors, which can produce stronger protection measures and guide the large language model away from harmful responses. By applying two safety alignment strategies to the large language model, it can ensure that the large language model can make safe and beneficial responses when facing potentially harmful inputs, prompting the model to prioritize beneficial rather than harmful when generating responses.

[0062] Specific implementation method 6: This implementation method is different from specific implementation methods 4 or 5 in that the specific process of step 4 is as follows:

[0063] Step 41: Calculate intermediate variables

[0064]

[0065] Among them, δ SFT is the ideal offset, which is a known quantity; λ∈R + , R + is a positive real number; the safety vector θ base are the pre-trained model parameters, are the model parameters after data alignment;

[0066] Step 42: According to calculate (i.e. DARE strategy), the specific process is:

[0067] Step 421: Generate a random variable m, and the random variable m obeys the Bernoulli distribution with probability p;

[0068] m~Bernoulli(p)

[0069] Step 422: Use random variable m to To perform a discard operation:

[0070]

[0071] in, yes The corresponding discard operation results, ⊙ represents the dot product operation;

[0072] Step 423, follow The ratio of Zoom in and get the zoom result

[0073]

[0074] Step 4.3: Use the zoom result Model parameters after data alignment Fine-tuning (ie RESTA strategy):

[0075]

[0076] in, are the fine-tuned model parameters.

[0077] The other steps and parameters are the same as those in the fourth or fifth embodiment.

[0078] The present invention uses security vectors to fine-tune the parameters of large language models to compensate for the loss of security. DARE technology can identify and eliminate unnecessary parameter redundancy in the multi-task learning process. Doing so can improve the efficiency and performance of the model because it reduces the computational burden and improves the generalization ability of the model when processing various tasks, such as Figure 5 As shown, the DARE strategy can effectively eliminate 90% or even 99% of redundant parameters.

[0079] Specifically, DARE technology uses advanced algorithms to identify parameter redundancy in the multi-task learning process. This means that it can accurately determine which parameters are redundant for the model and can be safely removed without affecting the performance of the model. This ability to identify and eliminate redundancy not only improves the operating efficiency of the model, but also helps reduce the size of the model. By reducing unnecessary parameters, DARE technology significantly reduces the computational requirements of the model. This optimization makes the model more lightweight, faster, and consumes less energy. This is especially important for applications that need to run on mobile devices or edge computing devices. The simplicity and effectiveness of this approach make the model more suitable for deployment in resource-limited environments. In the field of multi-task learning and model optimization, DARE technology provides new possibilities for improving model performance and efficiency. DARE technology helps the model generalize better to new tasks by eliminating unnecessary parameters. This is because it reduces the risk of overfitting the model and makes the model more focused on those features that are really important for the task. Because DARE technology reduces the complexity of the model, it makes it easier to scale the model to larger data sets and more complex tasks. This scalability is critical for deploying machine learning models in the real world. In general, the application of DARE technology in the RESTA strategy shows how to improve the efficiency and effectiveness of multi-task learning by intelligently optimizing model parameters.

[0080] Specific implementation method seven: Combination Figure 3The present embodiment is described in a data alignment-based large language model response generation system, the system comprising a data set generation module, a large language model pre-training module, a data alignment module and a large language model fine-tuning module; wherein:

[0081] The data set generation module is used to generate a harmful response data set and a harmless response data set;

[0082] The large language model pre-training module is used to pre-train the large language model using the generated harmful response data set and harmless response data set;

[0083] The data alignment module is used to perform data alignment on the pre-trained large language model using the generated data set to obtain the large language model after data alignment;

[0084] The large language model fine-tuning module is used to fine-tune the large language model after data alignment, and the fine-tuned large language model is used to generate responses to input questions.

[0085] Specific implementation eight: This implementation is different from specific implementation seven in that the working process of the data set generation module is as follows:

[0086] Step 1. Obtain various harmful questions from the existing data set;

[0087] Step 12: Input each harmful question obtained in step 11 into the large language model (i.e., the model with initialized parameters), and the harmful questions interact with the large language model through the Chain of Discourse (COD) induction method, and the large language model is used to generate responses corresponding to each harmful question;

[0088] Step 13: Use the harmful replies generated in steps 1 and 2 and the harmful questions corresponding to the harmful replies to form a harmful reply data set, and use the harmless replies generated in steps 1 and 2 and the harmful questions corresponding to the harmless replies to form a harmless reply data set.

[0089] The other steps and parameters are the same as those in the seventh embodiment.

[0090] Specific implementation method 9: This implementation method is different from specific implementation method 8 in that the data alignment module uses strategy A or strategy B to perform data alignment; wherein:

[0091] (1) The specific process of strategy A is as follows:

[0092] The harmless response dataset is used to align the pre-trained large language model. That is, the pre-trained large language model is trained using the harmless response dataset. The loss function used in the training process is:

[0093]

[0094]

[0095] Among them, L1 is the loss function, N b is the harmless reply dataset, |N b | is the harmless reply dataset N b The number of samples in x i After the harmful question in sample x (each harmful question and its corresponding harmless reply constitute a sample) is input into the pre-trained large language model, the pre-trained large language model outputs the i-th word in the reply, w j is the jth word x in the pre-trained large language model output response j The corresponding vector, is to output the i-th word x i The conditional probability, w i is x i The corresponding vector, if w i If it is a harmful reply, The value of is 1, if w i If it is a harmless reply, The value of is 0;

[0096] (2) The specific process of strategy B is as follows:

[0097] First, the harmful response dataset and the harmless response dataset are used to perform the first data alignment on the pre-trained large language model, that is, the harmful response dataset and the harmless response dataset are used to perform the first stage of training on the pre-trained large language model. The loss function used in the first stage of training is:

[0098]

[0099] Among them, L2 is the loss function, |N|=|N b |+|N r |,|N b | is the harmless reply dataset N b The number of samples in |N r | is the harmful reply dataset N r The number of samples in λ1 and λ2 are weights, * indicates multiplication, logp(x) is the negative log-likelihood of sample x, Represents the dataset N r The sample set whose inner negative log-likelihood value is less than or equal to 1, Represents the dataset N r The sample set whose inner negative log-likelihood value is greater than 1, Is a collection The number of samples in Is a collection The number of samples in ;

[0100]

[0101] Among them, x i is the i-th word in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, n is the total number of words in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, and w j is the jth word x in the pre-trained large language model output response j The corresponding vector, is to output the i-th word x i The conditional probability, w i is x i The corresponding vector, if w i If it is a harmful reply, The value of is 1, if w i If it is a harmless reply, The value of is 0;

[0102] The harmless reply dataset is then used to perform a second data alignment on the model after the first data alignment, that is, the harmless reply dataset is used to perform a second stage of training on the model after the first training. The loss function used in the second stage of training is:

[0103]

[0104] Among them, L1 is the loss function.

[0105] The other steps and parameters are the same as those in the eighth embodiment.

[0106] Specific implementation method ten: This implementation method is different from specific implementation method nine in that the working process of the large language model fine-tuning module is as follows:

[0107] Step 1: Calculate intermediate variables

[0108] Among them, δ SFT is the ideal offset, which is a known quantity; λ∈R + , R + is a positive real number; the safety vector θ base are the pre-trained model parameters, are the model parameters after data alignment;

[0109] Step 2: According to calculate (i.e. DARE strategy), the specific process is:

[0110] Step 21: Generate a random variable m, and the random variable m obeys the Bernoulli distribution with probability p;

[0111] m~Bernoulli(p)

[0112] Step 22: Use random variable m to To perform a discard operation:

[0113]

[0114] in, yes The corresponding discard operation results, ⊙ represents the dot product operation;

[0115] Step 23: Follow The ratio of Zoom in and get the zoom result

[0116] Step 3: Use the scaling results Model parameters after data alignment Fine-tuning (ie RESTA strategy):

[0117]

[0118] in, are the fine-tuned model parameters.

[0119] The other steps and parameters are the same as those in the ninth embodiment.

[0120] The method of the present invention specifically comprises the following parts:

[0121] 1. Generation of the SECURE-RECOVERY partial training dataset: Blue data (harmless answers) and red data (harmful answers) are generated by interacting with a large language model. Harmful questions covering a wide range of topics are collected through COD dialogues. These data cover multiple topics and subtopics, ensuring the breadth and diversity of harmful questions, thereby providing the model with rich harmful scenario learning materials. This process not only collects harmful inputs, but also includes safe and unsafe responses generated by the model. The diversity and authenticity of these data are crucial for the model to understand and identify harmful content, because they provide examples of various challenging scenarios that the model may encounter in the real world.

[0122] SAFE-ALIGN model alignment: Using the generated training dataset, the model is guided towards safe alignment by minimizing the negative log-likelihood of harmful answers and penalizing harmful answers, using gradient ascent to optimize the sample loss. This ensures that the model prioritizes providing helpful and safe responses rather than harmful outputs when answering questions.

[0123] like Figure 4 As shown, the idea behind the safe alignment of blue (Strategy-A) and blue-red (Strategy-B) is that Strategy-A adjusts the base model M0 on the blue data (represented by dark blue circles) that mainly contains harmless responses to obtain M1. This is represented as trajectory P. (Strategy-B) first moves the model away from harmful responses in the red data (red circles) to obtain a more conservative intermediate model state M2, as shown in trajectory N. * As shown, then Strategy-A is trained on the harmless response to obtain M3 (in green), through the trajectory P * . Note that training on the red data shifts the model's initial state from M0 to M2. M3 will be safer and equally helpful relative to M1. The blue and green snowflake symbols represent equally helpful models obtained by training the blue data on different starting states. The closer the snowflake is to the red cluster, the more likely it is to generate harmful outputs.

[0124] In this phase, the model weights are optimized by gradient ascent to make the model more inclined to generate helpful responses. The goal of this phase is to adjust the model's behavior so that it can respond safely when faced with harmful inputs. In this way, the model can not only avoid generating harmful content, but also provide helpful information when possible. This targeted training method enhances the model's resistance to harmful information by simulating real conversation scenarios, thereby improving the safety of the model in practical applications. Through role-playing, the SECURE-RECOVERY framework simulates real conversation scenarios. This simulation not only enables the model to learn in a safe environment, but also enables researchers to better understand the model's behavior in various complex situations. This in-depth understanding is key to improving model security because it allows researchers to identify and solve problems that the model may encounter when dealing with harmful content. In addition, the SECURE-RECOVERY framework also emphasizes transparency and explainability in the model training process. By recording each step of the model's training process in detail, it is easier to track changes in the model's behavior and understand the reasons behind these changes. This transparency is crucial to building user trust in the model because it allows users to see how the model is trained to avoid harmful behavior. In general, the SECURE-RECOVERY framework of this invention provides a new direction for the security training of language models. It not only proposes a systematic method to improve the security of the model, but also enhances the model's resistance to harmful information by simulating real dialogue scenarios and role-playing. With the further development and application of this framework, language models will make greater progress in security and effectiveness.

[0125] The present invention challenges currently widely deployed models such as GPT-4 and ChatGPT through a series of carefully designed Chain of Discourse (COD) prompts, revealing that these models will produce unethical responses in more than 65% and 73% of harmful queries. This shows that even the most advanced models have significant security risks. The method of the present invention achieves safe alignment of the model through a two-stage process. First, a dataset of harmful questions covering a wide range of topics is collected through COD prompts; secondly, the collected dataset is used to train the model through an alignment strategy to minimize the negative log-likelihood of harmful responses and penalize harmful responses.

[0126] 2. RESTA method fine-tuning

[0127] The RESTA (REstoring Safety through Task Arithmetic) method improves the safety of fine-tuned language models (LLMs) by adding a pre-computed safety vector to the model's parameters. The core advantage of this method is that it can effectively restore the safety guarantees that may be lost during the fine-tuning process. In particular, when language models undergo supervised fine-tuning (SFT), although their performance in specific domains is improved, non-ideal task vectors are often introduced, causing the model to produce harmful outputs.

[0128] Test environment setup. Hardware environment: High-performance computing servers were used, equipped with sufficient GPU resources to support the training and evaluation of large-scale language models. Development tools: Pycharm Visual Studio Code, used for writing and debugging code. Operating environment: python3.11.

[0129] Test data generation: The present invention can use three test methods, standard, cot (Chain of Thoughts) and COD (Chain of Discourse), to evaluate the security of large language models (LLMs). The test principle is to evaluate the response ability of large language models (LLMs) when facing challenges that may lead to unsafe outputs through specific prompt templates. These test methods detect and improve the security of the model by simulating the behavior of attackers.

[0130] Standard: This test method uses standard prompts to evaluate the model's response. It is the most direct test method and does not involve any additional thinking steps or discourse chains.

[0131] CoT (Chain of Thoughts): This testing method guides the model to generate deeper and more coherent answers through a series of thinking steps.

[0132] COD (Chain of Discourse): This testing method tests the model's response through a series of utterances. It simulates a conversation scenario where one character (Red-LM) tries to induce another character (Base-LM) to generate unsafe responses. The purpose of this method is to test the model's ability to respond to challenges that may induce unsafe responses, and how it handles these challenges without violating safety guidelines.

[0133] The present invention uses COD prompt technology to construct a question-and-answer dataset containing 1,900 harmful questions, and collects 9,500 safe and 7,300 harmful dialogue examples. The data covers multiple fields such as literature, philosophy, social sciences, and medicine, ensuring the comprehensiveness of the test. At the same time, the method of the present invention is tested in multiple minority languages. By supporting multiple minority language inputs, it is not limited to the English environment, but also demonstrates its applicability in multilingual environments, including Chinese, Vietnamese, etc. By constructing a multilingual benchmark dataset containing 550 harmful questions covering 11 categories and 55 subcategories, the effectiveness and generalization ability of the method of the present invention in a global language range are verified to avoid security vulnerabilities caused by language differences.

[0134] Result evaluation: Define the attack success rate (ASR) as a metric to calculate the proportion of harmful responses given by the model in harmful queries. For the open source model Vicuna-7B, ASR can directly reflect the number of harmful responses.

[0135] Moreover, the test results show that the performance of the model after security alignment exceeds that of the model without security alignment, proving the effectiveness of the SECURE-RECOVERY part. While maintaining most of the practical performance of the model, the model after security alignment significantly reduces the harmful response rate. In both PEFT mode and Full-FT mode, the RESTA method can effectively reduce the insecurity of the model, but the RESTA method combined with the DARE strategy performs better in the PEFT mode, while in the Full-FT mode, the RESTA method alone is effective enough. The test results show that by fine-tuning the model by adding security vectors, the harmful response rate can be reduced from 18.6% to 5.1%, while keeping most of the model performance unaffected. In addition, the security vector has good generalization ability, which is not only effective across categories, but also can be transferred between multiple languages.

[0136] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A large language model response generation method based on data alignment, characterized in that: The method specifically comprises the following steps: Step 1: Obtain harmful question data from the data set, input the obtained harmful question data into the large language model, and generate a harmful response data set and a harmless response data set; Step 2: Pre-train the large language model using the harmful response dataset and the harmless response dataset to obtain a pre-trained large language model; Step 3: Use the generated data set to align the pre-trained large language model to obtain the large language model after data alignment; Step 4: fine-tune the parameters of the large language model after data alignment to obtain a large language model after parameter fine-tuning; Step 5: Input the question into the large language model after fine-tuning the parameters in step 4, and use the large language model after fine-tuning the parameters to generate a response to the input question.

2. The method for generating a large language model response based on data alignment according to claim 1, characterized in that: The large language model in step 1 is Vicuna-7B.

3. The method for generating a large language model response based on data alignment according to claim 2, characterized in that: The specific process of step one is: Step 1. Obtain each harmful problem from the existing data set; Step 12: Input each harmful question obtained in step 11 into the large language model respectively. The harmful questions interact with the large language model through discourse chain induction, and the large language model is used to generate responses corresponding to each harmful question. Step 13: Use the harmful replies generated in steps 1 and 2 and the harmful questions corresponding to the harmful replies to form a harmful reply data set, and use the harmless replies generated in steps 1 and 2 and the harmful questions corresponding to the harmless replies to form a harmless reply data set.

4. The method for generating a large language model response based on data alignment according to claim 3, characterized in that: The specific process of step three is: The harmless response dataset is used to align the pre-trained large language model. That is, the pre-trained large language model is trained using the harmless response dataset. The loss function used in the training process is: Among them, L1 is the loss function, x is the harmless reply dataset N b The samples in |N b | is the harmless reply dataset N b The number of samples in x i is the i-th word in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, n is the total number of words in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, and w j is the jth word x in the pre-trained large language model output response j The corresponding vector, is to output the i-th word x i The conditional probability, w i is x i The corresponding vector, if w i If it is a harmful reply, The value of is 1, if w i If it is a harmless reply, The value of is 0.

5. The method for generating a large language model response based on data alignment according to claim 3, characterized in that: The specific process of step three is: The harmful response dataset and the harmless response dataset are used to align the pre-trained large language model. That is, the pre-trained model is trained in two stages using the harmful response dataset and the harmless response dataset, where: In the first stage, the pre-trained model is trained using the harmful response dataset and the harmless response dataset. The loss function used in the first stage of training is: Among them, L2 is the loss function, |N|=|N b |+|N r |,|N b | is the harmless reply dataset N b The number of samples in |N r | is the harmful reply dataset N r The number of samples in λ1 and λ2 are weights, * indicates multiplication, logp(x) is the negative log-likelihood of sample x, Represents the dataset N r The sample set whose inner negative log-likelihood value is less than or equal to 1, Represents the dataset N r The sample set whose inner negative log-likelihood value is greater than 1, Is a collection The number of samples in Is a collection The number of samples in ; Among them, x i is the i-th word in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, n is the total number of words in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, and w j is the jth word x in the pre-trained large language model output response j The corresponding vector, is to output the i-th word x i The conditional probability, w i is x i The corresponding vector, if w i If it is a harmful reply, The value of is 1, if w i If it is a harmless reply, The value of is 0; In the second stage, the pre-trained model is trained using the harmless response dataset. The loss function used in the second stage of training is: Among them, L1 is the loss function.

6. The method for generating a large language model response based on data alignment according to claim 4 or 5, characterized in that: The specific process of step 4 is as follows: Step 41: Calculate intermediate variables Among them, δ SFT is the ideal offset; λ∈R + , R + is a positive real number; the safety vector θ base are the pre-trained model parameters, are the model parameters after data alignment; Step 42: According to calculate The specific process is: Step 421: Generate a random variable m, and the random variable m obeys the Bernoulli distribution with probability p; m~Bernoulli(p) Step 422: Use random variable m to To perform a discard operation: in, yes The corresponding discard operation results, ⊙ represents the dot product operation; Step 423, follow The ratio of Zoom in and get the zoom result Step 4.3: Use the zoom result Model parameters after data alignment To make fine adjustments: in, are the fine-tuned model parameters.

7. A large language model response generation system based on data alignment, characterized in that: The system includes a data set generation module, a large language model pre-training module, a data alignment module and a large language model fine-tuning module; wherein: The data set generation module is used to generate a harmful response data set and a harmless response data set; The large language model pre-training module is used to pre-train the large language model using the generated harmful response data set and harmless response data set; The data alignment module is used to perform data alignment on the pre-trained large language model using the generated data set to obtain the large language model after data alignment; The large language model fine-tuning module is used to fine-tune the large language model after data alignment, and the fine-tuned large language model is used to generate responses to input questions.

8. The large language model response generation system based on data alignment according to claim 7, characterized in that: The working process of the data set generation module is as follows: Step 1. Obtain each harmful problem from the existing data set; Step 12: Input each harmful question obtained in step 11 into the large language model respectively. The harmful questions interact with the large language model through discourse chain induction, and the large language model is used to generate responses corresponding to each harmful question. Step 13: Use the harmful replies generated in steps 1 and 2 and the harmful questions corresponding to the harmful replies to form a harmful reply data set, and use the harmless replies generated in steps 1 and 2 and the harmful questions corresponding to the harmless replies to form a harmless reply data set.

9. The large language model response generation system based on data alignment according to claim 8, characterized in that: The data alignment module adopts strategy A or strategy B to perform data alignment; wherein: (1) The specific process of strategy A is as follows: The harmless response dataset is used to align the pre-trained large language model. That is, the pre-trained large language model is trained using the harmless response dataset. The loss function used in the training process is: Among them, L1 is the loss function, N b is the harmless reply dataset, |N b | is the harmless reply dataset N b The number of samples in x i After the harmful question in sample x is input into the pre-trained large language model, the pre-trained large language model outputs the i-th word in the reply, w j is the jth word x in the pre-trained large language model output response j The corresponding vector, is to output the i-th word x i The conditional probability, w i is x i The corresponding vector, if w i If it is a harmful reply, The value of is 1, if w i If it is a harmless reply, The value of is 0; (2) The specific process of strategy B is as follows: First, the harmful response dataset and the harmless response dataset are used to perform the first data alignment on the pre-trained large language model, that is, the harmful response dataset and the harmless response dataset are used to perform the first stage of training on the pre-trained large language model. The loss function used in the first stage of training is: Among them, L2 is the loss function, |N|=|N b |+|N r |,|N b | is the harmless reply dataset N b The number of samples in |N r | is the harmful reply dataset N r The number of samples in λ1 and λ2 are weights, * indicates multiplication, logp(x) is the negative log-likelihood of sample x, Represents the dataset N r The sample set whose inner negative log-likelihood value is less than or equal to 1, Represents the dataset N r The sample set whose inner negative log-likelihood value is greater than 1, Is a collection The number of samples in Is a collection The number of samples in ; Among them, x i is the i-th word in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, n is the total number of words in the response output by the pre-trained large language model after the harmful question in sample x is input into the pre-trained large language model, and w j is the jth word x in the pre-trained large language model output response j The corresponding vector, is to output the i-th word x i The conditional probability, w i is x i The corresponding vector, if w i If it is a harmful reply, The value of is 1, if w i If it is a harmless reply, The value of is 0; The harmless reply dataset is then used to perform a second data alignment on the model after the first data alignment, that is, the harmless reply dataset is used to perform a second stage of training on the model after the first training. The loss function used in the second stage of training is: Among them, L1 is the loss function.

10. The large language model response generation system based on data alignment according to claim 9, characterized in that: The working process of the large language model fine-tuning module is as follows: Step 1: Calculate intermediate variables Among them, δ SFT is the ideal offset; λ∈R + , R + is a positive real number; the safety vector θ base are the pre-trained model parameters, are the model parameters after data alignment; Step 2: According to calculate The specific process is: Step 21: Generate a random variable m, and the random variable m obeys the Bernoulli distribution with probability p; m~Bernoulli(p) Step 22: Use random variable m to To perform a discard operation: in, yes The corresponding discard operation results, ⊙ represents the dot product operation; Step 23: Follow The ratio of Zoom in and get the zoom result Step 3: Use the scaling results Model parameters after data alignment To make fine adjustments: in, are the fine-tuned model parameters.

Images