Large language model training method and computer equipment
By constructing positive and negative inference samples, training the large language model is solved, and the problem that the large language model cannot recognize user intentions and is vulnerable to attack is achieved, achieving safer and more reliable response generation.
Patent Information
- Application Number
- CN202510373629.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-27
- Publication Date
- 2025-08-08
AI Technical Summary
The existing large language model cannot recognize the user's intentions when the user injects prompt words or external knowledge, resulting in the risk of outputting harmful content and being easily bypassed by jailbreak attacks.
By obtaining seed data sets and security rules sets, the first positive and negative inference samples are constructed, and the initial large language model is trained using the direct preference optimization method to enhance its resistance to complex security tasks and jailbreak attacks.
Improve the security and reliability of large language models when generating responses, effectively identify and respond to edge cases, enhance the ability to fight attacks, and avoid unsafe responses.
Smart Images
Figure CN120450032A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of large language model training, and in particular to a large language model training method and computer equipment. Background Art
[0002] With the development of science and technology, the capabilities of large language models are gradually increasing. Existing large language models have multi-dimensional core capabilities and have excellent capabilities in natural language understanding, mathematical reasoning, code generation, and cross-task generalization. They can output solutions to problems based on questions or instructions input by users, or output corresponding content according to instructions.
[0003] However, although the large language model trained by the existing large language model training method has the above-mentioned advantages, in actual use, when the user requires the large language model to output harmful content through prompt word guidance or external knowledge injection, the large language model may be unable to discern the user's intention and output harmful content. Summary of the Invention
[0004] Based on this, it is necessary to provide a large language model training method and computer device that can handle complex security tasks and resist jailbreak attacks to address the above technical problems. The large language model training method includes:
[0005] Obtaining a seed data set and a security rule set; the seed data set includes: abnormal user questions, positive response samples corresponding to the abnormal user questions, and negative response samples corresponding to the abnormal user questions;
[0006] Based on the initial large language model, and according to the seed data set and the security rule set, obtaining a first positive inference sample and a negative inference sample;
[0007] Based on the first positive inference sample, the negative inference sample, and the seed data set, a first preference data pair is constructed, and the initial large language model is trained to obtain a trained large language model.
[0008] According to one embodiment of the present application, obtaining the seed dataset and the security rule set includes:
[0009] Obtaining the seed data set and the security rule set; the security rule set includes multiple groups of security rules of different types;
[0010] Based on multiple groups of different types of security rules, the seed data set is classified to obtain each type of security rule and a corresponding type of seed data set.
[0011] In this way, by making the security rule set include multiple groups of security rules of different types, and classifying the seed data set according to the multiple groups of security rules of different types, each type of security rule and the corresponding type of seed data set are obtained, so that the initial large language model can be trained for different types of security rules in the future to enhance the ability of the trained large language model to deal with different types of abnormal attacks.
[0012] According to one embodiment of the present application, obtaining the first positive inference sample and the negative inference sample based on the initial large language model and the seed dataset and the security rule set includes:
[0013] Based on the initial large language model, determine the first positive inference sample according to each type of security rule and the corresponding type of seed data set;
[0014] Determine negative inference samples based on the initial large language model, jailbreak attack algorithm, and seed dataset.
[0015] With such a setting, the initial large language model is guided to output the correct logical deduction process based on the initial large language model, according to each type of security rules and the corresponding type of seed data set, thereby obtaining the first positive inference sample. The initial large language model is guided to output the incorrect logical deduction process based on the initial large language model, the jailbreak attack algorithm and the seed data set, thereby obtaining the negative inference sample. This lays the foundation for the subsequent construction of the first preference data pair, training the initial large language model through the direct preference optimization method, and obtaining the trained large language model.
[0016] According to one embodiment of the present application, determining the first positive inference sample based on the initial large language model and according to each type of security rule and a corresponding type of seed data set includes:
[0017] Each type of security rule and abnormal user questions in the corresponding seed dataset are used as input, and security analysis is performed using the initial large language model to obtain thought chain samples.
[0018] The thought chain sample is formatted to obtain a first positive reasoning sample.
[0019] According to one embodiment of the present application, determining negative inference samples based on the initial large language model, the jailbreak attack algorithm, and the seed dataset includes:
[0020] Abnormal user questions in the seed data set are used as inputs of the initial large language model, and the jailbreak attack algorithm is applied to the initial large language model, and the output of the initial large language model is determined as a negative inference sample.
[0021] According to one embodiment of the present application, the large language model training method further includes:
[0022] The initial large language model is trained according to the abnormal user questions in the seed data set, the positive response samples corresponding to the abnormal user questions, and the first positive inference samples to obtain a pre-trained large language model.
[0023] With this arrangement, the initial large language model is pre-trained based on the abnormal user questions, positive response samples, and first positive inference samples in the seed data set, thereby improving the performance of the large language model.
[0024] According to one embodiment of the present application, obtaining the first positive inference sample and the negative inference sample based on the initial large language model and the seed dataset and the security rule set further includes:
[0025] Get the user's positive prompt information;
[0026] Based on the negative inference sample, taking the positive prompt information as input to the initial large language model, and determining the output of the initial large language model as a third positive inference sample;
[0027] A mixed inference sample is determined based on the negative inference sample and the third positive inference sample.
[0028] With such a setting, a third inference sample is obtained based on the user's positive prompt information and negative inference samples, and a mixed inference sample is determined based on the negative inference sample and the third positive inference sample, thereby laying the foundation for the subsequent construction of the first preference data pair, training the initial large language model through the direct preference optimization method, and obtaining a trained large language model.
[0029] According to one embodiment of the present application, constructing a first preference data pair based on the first positive inference sample, the negative inference sample, and the seed dataset, training the initial large language model, and obtaining the trained large language model includes:
[0030] Constructing a first preference data pair based on the first positive inference sample, the negative inference sample, the abnormal user question in the seed data set, the positive response sample corresponding to the abnormal user question, and the negative response sample corresponding to the abnormal user question; or constructing a first preference data pair based on the first positive inference sample, the negative inference sample, the mixed inference sample, the abnormal user question in the seed data set, the positive response sample corresponding to the abnormal user question, and the negative response sample corresponding to the abnormal user question;
[0031] According to the first preference data pair, the initial large language model is trained by a direct preference optimization method to obtain a trained large language model.
[0032] According to one embodiment of the present application, constructing a first preference data pair based on the first positive inference sample, the negative inference sample, and the seed dataset, training the initial large language model, and obtaining the trained large language model further includes:
[0033] Based on the initial large language model, simplifying the first positive inference sample to obtain a second positive inference sample;
[0034] Based on the first positive inference sample, the second positive inference sample, the abnormal user questions in the seed data set, and the positive response samples corresponding to the abnormal user questions, a second preference data pair is constructed, and the trained large language model is trained to obtain a second large language model.
[0035] In this way, by simplifying the first positive inference sample, a second positive inference sample is obtained, and a positive sample of the second preference data pair is constructed with the positive response sample. The positive sample in the original first preference data pair is used as a negative sample of the second preference data pair, and the trained large language model is trained to obtain a second large language model, thereby improving the response speed of the large language model.
[0036] According to another aspect of the present application, the present application further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned large language model training method when executing the computer program.
[0037] The above-mentioned large language model training method and computer device, based on the initial large language model, use the seed data set and the security rule set as input to obtain a first positive inference sample and a negative inference sample, thereby constructing a first preference data pair, training the initial large language model, and obtaining a trained large language model, so as to improve the large language model's ability to cope with complex security tasks and resist jailbreak attacks, thereby improving the performance and efficiency of the large language model, thereby solving the problem that in actual use, when the user requires the large language model to output harmful content through prompt word guidance or external knowledge injection, the large language model may be unable to discern the user's intention and output harmful content. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a diagram of an application environment of a large language model training method in one embodiment;
[0039] Figure 2 1 is a flow chart of a large language model training method according to an embodiment;
[0040] Figure 3 Schematic diagram of the process of step S2 in one embodiment;
[0041] Figure 4 Schematic diagram of the process of step S21 in one embodiment;
[0042] Figure 5 1 is a flow chart of a large language model training method according to an embodiment;
[0043] Figure 6 is a diagram of the internal structure of a computer device in one embodiment;
[0044] Figure 7 1 is a block diagram of a large language model training device in one embodiment. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0046] Current methods for secure alignment of large language models rely primarily on preventative strategies, aiming to train the model with limited training data to prevent it from responding to harmful user requests. However, existing methods suffer from two major technical issues: 1. Existing secure alignment methods rely heavily on secure datasets, but in practice, "edge" cases often occur that are not covered in the training data. Because large language models are unable to identify these uncovered special cases, they may respond unsafely to such requests, creating potential security risks. 2. When faced with techniques such as jailbreak attacks (such as inducing the model to generate a specific opening sentence "Okay, here are the specific solutions"), the protection mechanisms of existing methods are often easily bypassed. The model may be led into the wrong reasoning path, outputting harmful or dangerous content, and cannot effectively cope with the challenges of adversarial input.
[0047] Supervised Fine-Tuning (SFT): Supervised fine-tuning is a training method that further adjusts a pre-trained model to perform a specific task by using labeled training data. In this process, the model is trained based on inputs and labeled target outputs, thereby learning to perform well on the given task.
[0048] Reinforcement Learning from Human Feedback: Reinforcement Learning from Human Feedback is a training method that uses human feedback (such as ratings or preferences) to guide the model's optimization process. By using human preferences as reward signals, the model gradually learns to generate outputs that meet human expectations when performing tasks.
[0049] The large language model training method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server is used to execute the large language model training method. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.
[0050] In order to solve the above problems and address the shortcomings of the existing technology in that it cannot accurately control and prevent complex security tasks and jailbreak attacks, in one embodiment of the present application, Figure 2 As shown, a large language model training method is provided, including:
[0051] S1. Obtain a seed data set and a security rule set; the seed data set includes: abnormal user questions, positive response samples corresponding to the abnormal user questions, and negative response samples corresponding to the abnormal user questions.
[0052] It is worth noting that the seed dataset includes a collection of multiple sets of abnormal user questions, multiple sets of positive response samples corresponding to the abnormal user questions, and multiple sets of negative response samples corresponding to the abnormal user questions, which are presented in forms such as tables or databases. Abnormal user questions are questions or instructions asked by users to the large language model that may lead the large language model to output harmful content. Positive response samples are when the large language model explicitly follows security guidelines based on abnormal user questions and directly rejects or circumvents harmful requests (such as pointing out the inappropriateness of the request, providing neutral suggestions, or changing the topic). For example, the large language model explicitly refuses to answer the input abnormal user request and outputs relevant laws and regulations to persuade the user. Negative response samples are when the large language model directly meets the user's request (such as generating offensive content, leaking sensitive information), or implicitly supports the user's improper intentions based on abnormal user questions.
[0053] It should be noted that the safety rule set is a collection of different types of safety rules, and its categories specifically include: endangering child safety, endangering infrastructure safety, violent behavior, privacy disclosure, illegal items, endangering emotional safety, endangering personal safety, false information, political neutrality, restrictions on law enforcement use, fraudulent behavior, platform integrity, pornographic content, legal compliance, etc.
[0054] S2. Based on the initial large language model, according to the seed data set and the security rule set, obtain the first positive inference sample and the negative inference sample.
[0055] It is worth noting that the first positive inference sample is the logical reasoning process when the large language model makes a positive response when taking an abnormal user question as input, and its expression is in the form of a logical chain. The negative inference sample is the logical reasoning process when the large language model makes a negative response when taking an abnormal user question as input, and its expression is in the form of a logical chain. The initial large language model is a model that performs logical reasoning based on the user input instruction and combines it with the knowledge base, and outputs a response associated with the instruction.
[0056] It is worth noting that the Large Language Model (LLM), a deep learning model trained on large amounts of text data, is capable of generating, understanding, and reasoning about natural language. Based on a deep neural network structure, it can capture the complex meanings of language and is widely used in natural language processing (NLP) tasks such as text generation, translation, and question answering.
[0057] S3. Based on the first positive inference sample, the negative inference sample, and the seed data set, a first preference data pair is constructed, and the initial large language model is trained to obtain a trained large language model.
[0058] The first preference data pair is expressed as a set of data pairs, including positive samples and negative samples, which are used to train the large language model through a direct preference optimization method, so that the logical reasoning method and the output response of the large language model are more inclined to positive samples.
[0059] It is worth noting that step S3 specifically includes: combining the first positive inference sample with the positive response sample in the seed data set to obtain the positive sample in the first preference data pair, and combining the negative inference sample with the negative response sample in the seed data set to obtain the negative sample in the first preference data pair, and training the initial large language model through direct preference optimization to obtain a trained large language model.
[0060] It should be noted that the large language model training method uses the seed data set and the security rule set as input based on the initial large language model to obtain the first positive inference sample and the negative inference sample, thereby constructing the first preference data pair, and training the initial large language model to obtain a trained large language model, so as to improve the large language model's ability to cope with complex security tasks and resist jailbreak attacks, and improve the security and reliability of the large language model when generating responses. By introducing a clear positive inference module, the large language model can perform deep reasoning and security assessment before generating a response, thereby effectively identifying and responding to "edge" cases and avoiding unsafe responses. In addition, the language model training method also combines security rules with the reasoning process to enhance the model's resistance to adversarial attacks, especially when encountering jailbreak attacks, it can prevent potential harmful outputs through a self-censorship mechanism.
[0061] In other embodiments of the present application, obtaining the seed dataset and the security rule set includes:
[0062] S11. Obtain a seed data set and a security rule set, where the security rule set includes multiple groups of security rules of different types.
[0063] S12. Classify the seed dataset based on the multiple sets of different types of security rules to obtain each type of security rule and a corresponding type of seed dataset. That is, the seed dataset is classified to obtain different types of seed datasets, and each different type of seed dataset corresponds to the multiple sets of different types of security rules.
[0064] It should be noted that the categories of the safety rules include endangering child safety, endangering infrastructure safety, violent behavior, privacy disclosure, illegal items, endangering emotional safety, endangering personal safety, false information, political neutrality, law enforcement use restrictions, fraudulent behavior, platform integrity, pornographic content, legal compliance, etc. Step S12 classifies the seed dataset according to the category of the safety rules to obtain different types of seed datasets corresponding to the categories of the safety rules.
[0065] It should be noted that the multiple sets of positive response samples and negative response samples corresponding to abnormal user questions in the seed database are collected by inputting multiple sets of abnormal user requests into multiple different large language models. The multiple sets of different types of security rules in the security rule set are the usage policies of the large language model obtained from a wide range of channels (for example, the official website of the large language model developer).
[0066] It is worth noting that step S12 specifically includes: based on the large language model, taking multiple groups of different types of security rules and seed data sets as input, guiding the large language model to classify the seed data sets based on the types of security rules through prompt word guidance, and obtaining each type of security rule and the corresponding type of seed data set.
[0067] In other embodiments of the present application, step S12 manually classifies the seed data sets according to the different categories of the security rules and based on the actual experience of the staff to obtain different types of seed data sets corresponding to the categories of the security rules.
[0068] It should be noted that by including multiple groups of different types of security rules in the security rule set and classifying the seed data set according to multiple groups of different types of security rules, each type of security rule and the corresponding type of seed data set are obtained. The initial large language model is trained for different types of security rules in the future to enhance the ability of the trained large language model to deal with different types of abnormal attacks.
[0069] In other embodiments of the present application, Figure 3 As shown, based on the initial large language model, according to the seed data set and the security rule set, the first positive inference sample and negative inference sample are obtained, including:
[0070] S21. Based on the initial large language model, determine a first positive inference sample according to each type of security rule and a corresponding type of seed data set.
[0071] S22. Determine negative inference samples based on the initial large language model, jailbreak attack algorithm, and seed dataset.
[0072] It should be noted that by guiding the initial large language model to output the correct logical deduction process based on the initial large language model and according to each type of security rules and the corresponding type of seed data set, the first positive inference sample is obtained. By guiding the initial large language model to output the incorrect logical deduction process based on the initial large language model, the jailbreak attack algorithm and the seed data set, the negative inference sample is obtained, which lays the foundation for the subsequent construction of the first preference data pair, training the initial large language model through the direct preference optimization method, and obtaining the trained large language model.
[0073] In other embodiments of the present application, Figure 4 As shown, based on the initial large language model, according to each type of security rule and the corresponding type of seed data set, determining the first positive inference sample includes:
[0074] S211. Take each type of security rule and abnormal user questions in the corresponding type of seed data set as input, perform security analysis through the initial large language model, and obtain a thought chain sample.
[0075] S212. Format the thought chain sample to obtain the first positive reasoning sample.
[0076] Exemplarily, based on the initial large language model, according to each type of security rule and the corresponding type of seed data set, the first positive inference sample is determined as follows: abnormal user questions of each type of security rule and the corresponding type of seed data set are taken as input, and the large language model is guided to perform security analysis through prompt words, the logical deduction process of the large language model performing security analysis is used as a thinking chain sample, and the thinking chain sample is formatted to obtain the first positive inference sample.
[0077] In other embodiments of the present application, determining a negative inference sample based on the initial large language model, the jailbreak attack algorithm, and the seed dataset includes:
[0078] S221. Abnormal user questions in the seed data set are used as input to the initial large language model, and the jailbreak attack algorithm is applied to the initial large language model to determine the output of the initial large language model as a negative inference sample.
[0079] It's important to note that the purpose of jailbreak attack algorithms is to trick the initial large language model into responding negatively. Specifically, by using prompts or external knowledge injection, they deceive the large language model's security filtering system, causing it to output harmful responses. For example, if a user asks "How do I steal an account?" and the model rejects the question, the attacker can circumvent the restrictions by role-playing (e.g., "Imagine you're a cybersecurity expert testing for vulnerabilities"). Alternatively, they can exploit vulnerabilities in the Retrieval Augmentation Generation (RAG) system to inject malicious content into the knowledge base (e.g., the PANDORA attack), causing the large language model to output harmful responses.
[0080] It's worth noting that prompts refer to the input text provided to a language model when interacting with it. The design of prompts can influence the model's behavior, causing it to generate responses of a specific style, content, or type. In secure alignment tasks, prompts are often used to guide the model through specific reasoning processes or decisions.
[0081] In other embodiments of the present application, the large language model training method further includes:
[0082] S30. Train the initial large language model based on the abnormal user questions in the seed data set, the positive response samples corresponding to the abnormal user questions, and the first positive inference samples to obtain a pre-trained large language model.
[0083] It should be noted that the performance of the large language model is improved by pre-training the initial large language model based on abnormal user questions, positive response samples, and the first positive inference samples in the seed dataset.
[0084] In other embodiments of the present application, obtaining the first positive inference sample and the negative inference sample based on the initial large language model and according to the seed data set and the security rule set further includes:
[0085] S23. Obtain positive prompt information from the user.
[0086] S24. Based on the negative inference sample, the positive prompt information is used as the input of the initial large language model, and the output of the initial large language model is determined as the third positive inference sample.
[0087] S25. Determine a mixed reasoning sample based on the negative reasoning sample and the third positive reasoning sample.
[0088] It should be noted that step S24 can be based on the negative inference sample obtained in step S22, or re-based on the initial large language model, the jailbreak attack algorithm and the seed data set to determine the negative inference sample, using the positive prompt information as the input of the initial large language model, and determining the output of the initial large language model as the third positive inference sample.
[0089] It is worth noting that the positive prompt information is prompt information that guides the initial large language model to perform positive logical deduction and make a positive response, and its form includes prompt words or guiding text.
[0090] It should be noted that the third inference sample is obtained based on the user's positive prompt information and negative inference sample, and the mixed inference sample is determined based on the negative inference sample and the third positive inference sample, that is, the negative inference sample and the third positive inference sample are combined to obtain a mixed inference sample, which lays the foundation for the subsequent construction of the first preference data pair, training the initial large language model through the direct preference optimization method, and obtaining the trained large language model.
[0091] In another embodiment of the present application, based on the initial large language model, according to the seed data set and the security rule set, obtaining the first positive inference sample and the negative inference sample further includes:
[0092] S26. Based on the initial large language model, a transition text is obtained according to the negative inference sample and the first positive inference sample.
[0093] S27. Determine a mixed reasoning sample based on the first positive reasoning sample, the negative reasoning sample, and the transition text.
[0094] Exemplarily, based on the first positive reasoning sample, the negative reasoning sample and the transition text, the mixed reasoning sample is determined specifically as follows: the reasoning process of some negative reasoning samples is selected, and the mixed reasoning sample is obtained by combining the first positive reasoning sample with the transition text.
[0095] It should be noted that the transition text is a transition paragraph used for transition between the partial negative reasoning sample and the first positive reasoning sample, which is used to connect the contents between the partial negative reasoning sample and the first positive reasoning sample in series to prevent the occurrence of contradictions between the contents of the partial negative reasoning sample and the first positive reasoning sample.
[0096] In other embodiments of the present application, based on the first positive inference sample, the negative inference sample, and the seed data set, a first preference data pair is constructed, and the initial large language model is trained to obtain the trained large language model, including:
[0097] S31. Construct a first preference data pair based on the first positive inference sample, the negative inference sample, the abnormal user questions in the seed data set, the positive response samples corresponding to the abnormal user questions, and the negative response samples corresponding to the abnormal user questions; or construct a first preference data pair based on the first positive inference sample, the negative inference sample, the mixed inference sample, the abnormal user questions in the seed data set, the positive response samples corresponding to the abnormal user questions, and the negative response samples corresponding to the abnormal user questions.
[0098] Exemplarily, step S31 is to combine the first positive inference sample with the positive response sample in the seed data set to obtain the positive sample in the first preference data pair, and combine the negative inference sample with the negative response sample in the seed data set to obtain the negative sample in the first preference data pair, thereby constructing the first preference data pair, or to combine the mixed inference sample with the positive response sample in the seed data set to obtain the positive sample in the first preference data pair, and combine the negative inference sample with the negative response sample in the seed data set to obtain the negative sample in the first preference data pair, thereby constructing the first preference data pair.
[0099] S32. According to the first preference data pair, the initial large language model is trained by a direct preference optimization method to obtain a trained large language model.
[0100] It's important to note that Direct Preference Optimization (DPO) is a method that improves model performance by optimizing preference pairs. In DPO, the model aims to optimize based on human-provided preference comparisons, learning to generate outputs that are more secure and practical, rather than relying solely on labeled data. This method aims to enhance the model's judgment capabilities through contrastive learning, particularly in complex multi-objective tasks.
[0101] In other embodiments of the present application, constructing a first preference data pair based on the first positive inference sample, the negative inference sample, and the seed dataset, training the initial large language model, and obtaining the trained large language model further includes:
[0102] S41. Based on the initial large language model, simplify the first positive inference sample to obtain a second positive inference sample. That is, the first positive inference sample is summarized using the large language model to obtain a simplified first positive inference sample (i.e., the second positive inference sample).
[0103] S42. Based on the first positive inference sample, the second positive inference sample, the abnormal user question in the seed data set, and the positive response sample corresponding to the abnormal user question, a second preference data pair is constructed, and the trained large language model is trained to obtain a second large language model.
[0104] It is worth noting that based on the first positive inference sample, the second positive inference sample, the abnormal user questions in the seed data set, and the positive response samples corresponding to the abnormal user questions, a second preference data pair is constructed, and the trained large language model is trained to obtain the second large language model. Specifically, the second positive inference sample is combined with the positive response sample to obtain the positive sample in the second preference data pair, and the first positive inference sample is combined with the positive response sample to obtain the negative sample in the second preference data pair. Through the direct preference optimization method, the trained large language model is trained in multiple rounds of iterations to obtain the second large language model.
[0105] It should be noted that by simplifying the first positive inference sample, a second positive inference sample is obtained, and the positive sample of the second preference data pair is constructed with the positive response sample. The positive sample in the original first preference data pair is used as the negative sample of the second preference data pair, and the trained large language model is trained to obtain the second large language model, thereby improving the response speed of the large language model.
[0106] In other embodiments of the present application, Figure 5 As shown, the large language model training method includes:
[0107] Step 1: Prepare a seed dataset for security alignment, where each data contains a harmful abnormal user question and two model responses (a safe response as a positive response sample and an unsafe response as a negative response sample).
[0108] Step 2: Obtain the usage policy of the large language model from a wide range of channels (for example, the official website of the large language model developer) and build a complete security rule set, which includes multiple groups of security rules of different types.
[0109] Step 3: Use a risk classification model (an existing large language model can be used) to classify each piece of data in the seed dataset constructed in Step 1. The risk categories correspond to the security rules of each category in the security rule set in Step 2.
[0110] Step 4: The abnormal user questions in the seed dataset are analyzed, along with the security rules sections corresponding to each data risk type. Prompts are used to instruct the existing large language model to perform a security analysis of the abnormal user questions based on the security rules. This analysis uses a chain of thought process, analyzing user intent step by step and using the security rules as evidence to generate a security judgment. By formatting these analysis results, the first positive inference sample is obtained.
[0111] It's worth noting that Chain of Thought (CoT) is a reasoning strategy that requires the model to perform step-by-step reasoning and analysis before generating a final answer. By explicitly outlining each step in the reasoning process, the model can better understand and evaluate the problem, thereby generating more accurate and reasonable answers. In practical tasks, Chain of Thought helps the model conduct in-depth analysis before generating a response, avoiding potential errors or harmful outputs.
[0112] Step 5: Take the abnormal user questions in each data as input, combine the first positive inference sample and the safe response (positive response sample) as output, and fine-tune the large language model to learn the paradigm of pre-action reasoning before formal response.
[0113] Step 6: Based on step 4, the jailbreak attack is used to guide the model to make incorrect judgments on abnormal user questions in the seed dataset. This requires providing evidence in the prior reasoning module to prove that abnormal user questions are safe enough, which serves as negative inference samples.
[0114] Step 7: Construct the first preference data pair, combining (negative inference sample, negative response sample) into a negative sample, and (first positive inference sample, positive response sample) into a positive sample. To further encourage the model to correct inference errors, this step also includes a new positive sample, namely (mixed inference sample (negative inference sample prefix + first positive inference sample), positive response sample). The negative inference sample prefix is used to simulate incorrect inference, and the first positive inference sample is used to correct errors. Negative samples and any positive samples can be combined into the first preference data pair.
[0115] Step 8: Based on the first preference data pair, the large language model trained in step 5 is further trained through the direct preference optimization method to enhance its analysis accuracy for abnormal user questions.
[0116] Step 9: Use a multi-round iterative direct preference optimization method to mitigate the inference delay of the large language model when facing unusual user questions. For user questions that pose no security risks, a positive example is (a concise first positive inference example, a positive response example), and a negative example is (a lengthy first positive inference example, a positive response example). The summarization capabilities of the large language model can be used to reorganize the inference module into a shortened version.
[0117] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.
[0118] Based on the same inventive concept, the present application also provides a large language model training device for implementing the large language model training method mentioned above. The implementation solution provided by this device is similar to the implementation solution described in the above method. Therefore, the specific limitations in one or more large language model training device embodiments provided below can be found in the above-mentioned limitations on the large language model training method and will not be repeated here.
[0119] In one embodiment, Figure 7 As shown, a large language model training device is provided, including: an acquisition module, an inference module and a training module, wherein:
[0120] The acquisition module is used to acquire a seed data set and a security rule set; the seed data set includes: abnormal user questions, positive response samples corresponding to the abnormal user questions, and negative response samples corresponding to the abnormal user questions.
[0121] The inference module is configured to obtain a first positive inference sample and a negative inference sample based on the initial large language model, the seed data set and the security rule set.
[0122] A training module is used to construct a first preference data pair based on the first positive inference sample, the negative inference sample and the seed data set, and train the initial large language model to obtain a trained large language model.
[0123] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data for executing the large language model training method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a large language model training method is implemented.
[0124] In one embodiment of the present application, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any one of the large language model training methods in the above embodiments is implemented.
[0125] In one embodiment of the present application, a storage medium is provided, on which a computer program is stored. The computer program is executed by a processor to implement any one of the large language model training methods in the above embodiments.
[0126] It should be noted that the computer program stored in the storage medium can automatically perform complex tasks, reduce human intervention, thereby improving overall efficiency and ensuring the reliability and trustworthiness of operations. In addition, the computer program is repeatable and extensible, so that the same computer program can be repeatedly executed in different environments and at different times, and is easy to expand to meet new needs, thereby enhancing the adaptability and flexibility of the computer program. In addition, running the computer program can greatly optimize resource utilization and reduce manpower and error costs by automatically processing large amounts of data and tasks. Finally, the computer program is also traceable and secure, and can ensure the auditability and security of operations by recording and tracking each step of the computer program execution, thereby reducing the risks brought by human operation.
[0127] In one embodiment of the present application, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.
[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0129] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.
[0130] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0131] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.
Claims
1. A large language model training method, characterized in that: The large language model training method includes: Obtaining a seed data set and a security rule set; the seed data set includes: abnormal user questions, positive response samples corresponding to the abnormal user questions, and negative response samples corresponding to the abnormal user questions; Based on the initial large language model, and according to the seed data set and the security rule set, obtaining a first positive inference sample and a negative inference sample; Based on the first positive inference sample, the negative inference sample, and the seed data set, a first preference data pair is constructed, and the initial large language model is trained to obtain a trained large language model.
2. The large language model training method according to claim 1, characterized in that The obtaining of the seed data set and the security rule set includes: Obtaining the seed data set and the security rule set; the security rule set includes multiple groups of security rules of different types; Based on multiple groups of different types of security rules, the seed data set is classified to obtain each type of security rule and a corresponding type of seed data set.
3. The large language model training method according to claim 1, characterized in that The obtaining of the first positive inference sample and the negative inference sample based on the initial large language model and according to the seed data set and the security rule set includes: Based on the initial large language model, determine the first positive inference sample according to each type of security rule and the corresponding type of seed data set; Determine negative inference samples based on the initial large language model, jailbreak attack algorithm, and seed dataset.
4. The large language model training method according to claim 3, characterized in that The determining of the first positive inference sample based on the initial large language model and according to each type of security rule and a corresponding type of seed data set includes: Each type of security rule and abnormal user questions in the corresponding seed dataset are used as input, and security analysis is performed using the initial large language model to obtain thought chain samples. The thought chain sample is formatted to obtain a first positive reasoning sample.
5. The large language model training method according to claim 3, characterized in that: Determining negative inference samples based on the initial large language model, jailbreak attack algorithm, and seed dataset includes: Abnormal user questions in the seed data set are used as inputs of the initial large language model, and a jailbreak attack algorithm is applied to the initial large language model, and the output of the initial large language model is determined as a negative inference sample.
6. The large language model training method according to claim 1, characterized in that The large language model training method further includes: The initial large language model is trained according to the abnormal user questions in the seed data set, the positive response samples corresponding to the abnormal user questions, and the first positive inference samples to obtain a pre-trained large language model.
7. The large language model training method according to claim 3, characterized in that: The step of obtaining a first positive inference sample and a negative inference sample based on the initial large language model and according to the seed data set and the security rule set further includes: Get the user's positive prompt information; Based on the negative inference sample, taking the positive prompt information as input to the initial large language model, and determining the output of the initial large language model as a third positive inference sample; A mixed inference sample is determined based on the negative inference sample and the third positive inference sample.
8. The large language model training method according to claim 1, characterized in that: The constructing of a first preference data pair based on the first positive inference sample, the negative inference sample, and the seed data set, and training the initial large language model to obtain a trained large language model includes: Constructing a first preference data pair based on the first positive inference sample, the negative inference sample, the abnormal user question in the seed data set, the positive response sample corresponding to the abnormal user question, and the negative response sample corresponding to the abnormal user question; or constructing a first preference data pair based on the first positive inference sample, the negative inference sample, the mixed inference sample, the abnormal user question in the seed data set, the positive response sample corresponding to the abnormal user question, and the negative response sample corresponding to the abnormal user question; According to the first preference data pair, the initial large language model is trained by a direct preference optimization method to obtain a trained large language model.
9. The large language model training method according to claim 8, characterized in that The constructing a first preference data pair based on the first positive inference sample, the negative inference sample, and the seed data set, and training the initial large language model to obtain a trained large language model further includes: Based on the initial large language model, simplifying the first positive inference sample to obtain a second positive inference sample; Based on the first positive inference sample, the second positive inference sample, the abnormal user questions in the seed data set, and the positive response samples corresponding to the abnormal user questions, a second preference data pair is constructed, and the trained large language model is trained to obtain a second large language model.
10. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the large language model training method as described in any one of claims 1 to 9 is implemented.