Optimization method and device, reply method and device of large language model, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an optimization method and device of a large language model, a reply method and device, equipment and a medium. First training data is constructed, the first training data comprises a first question and first reply content, the first reply content comprises an unsafe reply, an unsafe classification, an unsafe analysis result and a detoxification reply, the first question is input into the initial large language model, the initial large language model is controlled to output a first prediction result, and the first prediction result is obtained. And according to the first reply content and the first prediction result, optimizing the initial large language model to obtain an optimized large language model. According to the method and the device, the initial large language model is optimized through the constructed training data, the initial large language model is controlled to output the detoxification result containing detoxification of the reply, an audit model does not need to be additionally arranged, the complexity of outputting the safe content by the model is reduced, and in the using process, a single model is used for outputting, so that the safety of the user is improved. And the use convenience is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an optimization method, a reply method, a device, a equipment and a medium for a large language model. Background Art

[0002] With the rapid development of large-scale pre-training technology, large language models have demonstrated extraordinary capabilities in natural language understanding and generation, and have made significant breakthroughs in zero-shot / few-shot learning. However, the openness of large language models, combined with their super capabilities, also introduces new risks. Large language models are likely to output toxic texts during the generation process, including offensive, hateful, and biased content, which is risky for users. Toxicity is an inherent property of large language models because they will inevitably learn some toxic content during the training process. Traditional detoxification methods usually connect an external audit model to audit whether the output of the large language model meets security specifications. During the model training process, the large language model and the audit model need to be trained separately. During use, in order to output safe content, two models need to be used, which increases the complexity of the model outputting safe content. Therefore, in the process of large language models outputting safe content, how to reduce the complexity of the model has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides an optimization method, a response method, an apparatus, a device and a medium for a large language model to solve the problem of a relatively complex model in the process of outputting security content using a large language model.

[0004] In a first aspect, an embodiment of the present invention provides a method for optimizing a large language model, the method comprising: Constructing first training data, the first training data including a first question and a first answer content, the first answer content including an unsafe answer, an unsafe classification corresponding to the unsafe answer, an unsafe analysis result of analyzing the unsafe answer, and a detoxification answer for detoxifying the unsafe answer; Inputting the first question into an initial large language model, controlling the initial large language model to process the first question, and outputting a first prediction result, wherein the first prediction result includes a first answer to the first question, a first classification result of safety classification of the first answer, a first analysis result of analyzing the first answer, and a detoxification result of detoxifying the first answer; The initial large language model is optimized according to the first reply content and the first prediction result to obtain an optimized large language model.

[0005] In a second aspect, an embodiment of the present invention provides a reply method, the reply method comprising: Get the text of the question to be answered; The question text to be answered is input into the optimized large language model optimized by the optimization method of any one of claims 1 to 5, and the answer text corresponding to the question text to be answered and the answer classification result of security classification of the answer text are output. When the answer classification result is a security classification, the output is stopped. When the answer classification result is an unsafe classification, the answer analysis result of security analysis of the answer text and the answer detoxification result of the answer text are also output.

[0006] In a third aspect, an embodiment of the present invention provides an optimization device for a large language model, the optimization device comprising: A construction module, configured to construct first training data, wherein the first training data includes a first question and a first answer content, wherein the first answer content includes an unsafe answer, an unsafe classification corresponding to the unsafe answer, an unsafe analysis result for analyzing the unsafe answer, and a detoxification answer for detoxifying the unsafe answer; a control module, configured to input the first question into an initial large language model, control the initial large language model to process the first question, and output a first prediction result, wherein the first prediction result includes a first answer to the first question, a first classification result of safety classification of the first answer, a first analysis result of analyzing the first answer, and a detoxification result of detoxifying the first answer; An optimization module is used to optimize the initial large language model according to the first reply content and the first prediction result to obtain an optimized large language model.

[0007] In a fourth aspect, an embodiment of the present invention provides a reply device, the reply device comprising: An acquisition module is used to obtain the text of the question to be answered; An output module is used to input the question text to be answered into the optimized large language model optimized by the optimization method of any one of claims 1 to 5, output the answer text corresponding to the question text to be answered, and the answer classification result of the security classification of the answer text, stop output when the answer classification result is a security classification, and also output the answer analysis result of the security analysis of the answer text and the answer detoxification result of the answer text when the answer classification result is an unsafe classification.

[0008] In a fifth aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the optimization method of the first aspect or the reply method of the second aspect when executing the computer program.

[0009] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the optimization method of the first aspect or the reply method described in the second aspect is implemented.

[0010] Compared with the prior art, the present invention has the following beneficial effects: Construct the first training data, the first training data includes the first question and the first answer content, the first answer content includes an unsafe answer, an unsafe classification corresponding to the unsafe answer, an unsafe analysis result of analyzing the unsafe answer, and a detoxification answer for detoxifying the unsafe answer, input the first question into the initial large language model, control the initial large language model to process the first question, and output the first prediction result, the first prediction result includes the first answer to the first question, the first classification result of the security classification of the first answer, the first analysis result of analyzing the first answer, and the detoxification result of detoxifying the first answer, according to the first answer content and the first prediction result, the initial large language model is optimized to obtain an optimized large language model. In the present application, the initial large language model is optimized by constructing the training data, and the initial large language model is controlled to output a detoxification result including detoxification of the answer, without the need for an external audit model, thereby reducing the complexity of the model output security content, and during use, a single model is used for output, which increases the convenience of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0012] Figure 1 It is a schematic diagram of an application environment of an optimization method and a reply method of a large language model provided by an embodiment of the present invention; Figure 2 It is a flowchart of a method for optimizing a large language model provided by one embodiment of the present invention; Figure 3 is a flowchart of a reply method provided by an embodiment of the present invention; Figure 4 It is a structural schematic diagram of a large language model optimization device provided by an embodiment of the present invention; Figure 5 is a schematic structural diagram of a reply device provided by an embodiment of the present invention; Figure 6 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0015] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0016] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0017] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0018] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0019] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0020] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0021] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0022] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0023] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0024] Figure 1 1 is a schematic diagram of an application environment of an optimization method and a reply method of a large language model provided by an embodiment of the present invention. A model training method for outputting a safe answer provided by an embodiment of the present invention can be applied in Figure 1In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0025] See also Figure 2 , is a flow chart of a method for optimizing a large language model provided by an embodiment of the present invention. The method for optimizing a large language model can be applied to Figure 1 The server in Figure 2 As shown, the optimization method of the large language model may include the following steps.

[0026] S201: Constructing first training data, the first training data includes a first question and a first answer content, the first answer content includes an unsafe answer, an unsafe classification corresponding to the unsafe answer, an unsafe analysis result for analyzing the unsafe answer, and a detoxification answer for detoxifying the unsafe answer.

[0027] In step S201, first training data is constructed, the first training data includes a first question and a first answer content, the first answer content includes an unsafe answer, an unsafe classification corresponding to the unsafe answer, an unsafe analysis result for analyzing the unsafe answer, and a detoxified answer for detoxifying the unsafe answer, wherein the detoxified answer is a safe answer corresponding to the first question.

[0028] In this embodiment, when constructing the first training data, it is constructed according to a preset data format, wherein the preset data format may be {bos}<|User|>:{question}<|Assistant|>:{bad_answer}{eos}{cls_unsafe}{bad_analysis}{good_answer}{eos}, wherein {bos} is a start identifier, |User| is a human, {question} is the first question in the corresponding first training data, |Assistant| is AI, i.e., AI is used to reply, {bad_analysis} is an unsafe reply in the corresponding first reply content, {cls_unsafe} is an unsafe classification of the corresponding unsafe reply, {bad_analysis} is an unsafe analysis result of analyzing the unsafe reply, {good_answer} is a detoxification reply for detoxifying the unsafe reply, and {eos} is an end identifier, the first {eos} indicates the end of replying to the first question, and the second {eos} indicates the end of detoxification of the unsafe reply.

[0029] In another embodiment, the format of the preset data can also be {bos}<|User|>:{question}<|Assistant|>:{bad_answer}{eos}{bad_analysis}{cls_unsafe}{good_answer}{eos}, wherein {bos} is a start identifier, |User| is a human, {question} is the first question in the corresponding first training data, |Assistant| is AI, i.e., AI is used to reply, {bad_answer} is an unsafe reply in the corresponding first reply content, {bad_analysis} is an unsafe analysis result of analyzing the unsafe reply, {cls_unsafe} is an unsafe classification of the corresponding unsafe reply, {good_answer} is a detoxified reply for detoxifying the unsafe reply, and {eos} is an end identifier. The first {eos} indicates the end of replying to the first question, and the second {eos} indicates the end of detoxifying the unsafe reply. The unsafe analysis result for analyzing the unsafe reply is placed in front of the unsafe classification of the corresponding unsafe reply, that is, the unsafe analysis result for analyzing the unsafe reply is set first, and then the unsafe classification result for classifying the unsafe reply is set.

[0030] In this embodiment, when constructing the first training data, the first answer content to the first question in the first training data is enriched, that is, the first answer content includes not only an unsafe answer, but also an unsafe classification corresponding to the unsafe answer, an unsafe analysis result for analyzing the unsafe answer, and a detoxified answer for detoxifying the unsafe answer, so that the optimized large language model can be used alone to output a safe answer after detoxification, without the need for an external review model, thereby reducing the complexity of the large language model outputting safe content.

[0031] S102: Input the first question into the initial large language model, control the initial large language model to process the first question, and output a first prediction result, the first prediction result including a first answer to the first question, a first classification result of security classification of the first answer, a first analysis result of analyzing the first answer, and a detoxification result of detoxifying the first answer.

[0032] In step S102, the large language model can be any large language model, the first question is input into the initial large language model, the initial large language model is controlled to process the first question, and a first prediction result is output, wherein the output order of the first prediction result corresponds to the data format of the first training data, that is, the output order of the first prediction result is the first answer to the first question, the first classification result of security classification of the first answer, the first analysis result of analyzing the first answer, and the detoxification result of detoxifying the first answer.

[0033] In this embodiment, an initial large language model is obtained, wherein the initial large language model is any large language model, such as a GPT-3 model, a first question is input into the initial large language model, the initial large language model is controlled to process the first question, and a first prediction result is output, the first prediction result including a first answer to the first question, a first classification result of security classification of the first answer, a first analysis result of analyzing the first answer, and a detoxification result of detoxifying the first answer.

[0034] In this embodiment, the initial large language model is controlled to sequentially output a first answer to the first question, a first classification result of security classification of the first answer, a first analysis result of analyzing the first answer, and a detoxification result of detoxifying the first answer, that is, the first answer is firstly classified for security, and then the first answer is analyzed, so that the optimized large language model first classifies the security of the first answer during the reasoning process, and then determines whether to perform analysis based on the security classification result. If it is a security classification, no analysis is required, thereby simplifying the reasoning performance.

[0035] In another embodiment, if the data format of the first training data is {bos}<|User|>:{question}<|Assistant|>:{bad_answer}{eos}{bad_analysis}{cls_unsafe}{good_answer}{eos}, after the first question is input into the initial large language model, the initial large language model is controlled to process the first question and output a first prediction result, wherein the output order of the first prediction result is a first answer to the first question, a first analysis result of analyzing the first answer, a first classification result of classifying the first answer for safety, and a detoxification result of detoxifying the first answer.

[0036] S103: Optimizing the initial large language model according to the first reply content and the first prediction result to obtain an optimized large language model.

[0037] In step S103, the initial large language model is optimized according to the first reply content and the first prediction result, that is, according to the loss between the first reply content and the first prediction result, the initial large language model is optimized to obtain an optimized large language model.

[0038] In this embodiment, based on the first reply content and the first prediction result, the loss between the first reply content and the first prediction result is calculated, and the initial large language model is iteratively optimized according to the corresponding loss to obtain an optimized large language model.

[0039] In this embodiment, when the initial large language model is iteratively optimized according to the corresponding loss, the optimization can be stopped when the corresponding loss is less than a preset loss threshold, or when the number of iterations reaches a preset number, or when the corresponding loss converges.

[0040] It should be noted that when calculating the loss between the first reply content and the first prediction result, there is no need to calculate the loss between the unsafe reply and the first reply to prevent the initial large language model from learning to output an unsafe reply.

[0041] It should be noted that when the initial large language model is iteratively optimized according to the corresponding loss, the parameters in the initial large language model are continuously adjusted to obtain an optimized large language model.

[0042] Optionally, it also includes: Constructing second training data, the second training data including a second question and a second answer content, the second answer content including a security answer, a security classification corresponding to the security answer, and a security analysis result of analyzing the security answer; The second question is input into the initial large language model, the initial large language model is controlled to process the second question, and a second prediction result is output, wherein the second prediction result includes a second answer to the second question, a second classification result of security classification of the second answer, and a second analysis result of analyzing the second answer.

[0043] In this embodiment, second training data is constructed, the second training data includes a second question and a second answer content, the second answer content includes a security answer, a security classification corresponding to the security answer, and a security analysis result of analyzing the security answer.

[0044] In this embodiment, when constructing the second training data, it is constructed according to a preset data format, wherein the preset data format may be {bos}<|User|>:{question}<|Assistant|>:{good_answer}{eos}{cls_safe}{good_analysis}{eos}, wherein {bos} is a start identifier, |User| is a human, {question} is the second question in the corresponding second training data, |Assistant| is AI, i.e., AI is used to answer, {good_answer}r is a safe answer in the corresponding second answer content, {cls_safe} is a security classification of the corresponding safe answer, {good_analysis} is a security analysis result of analyzing the safe answer, and {eos} is an end identifier, the first {eos} indicates the end of answering the second question, and the second {eos} indicates the end of the security analysis result of analyzing the safe answer.

[0045] In another embodiment, the format of the preset data can also be {bos}<|User|>:{question}<|Assistant|>:{good_answer}{eos}{good_analysis}{cls_safe}{eos}, wherein {bos} is a start identifier, |User| is a human, {question} is the second question in the corresponding second training data, |Assistant| is AI, i.e., AI is used to answer, {good_answer} is the safety answer in the corresponding second answer content, {good_analysis} is the safety analysis result of analyzing the safety answer, {cls_safe} is the safety classification of the corresponding safety answer, and {eos} is an end identifier, the first {eos} indicates the end of the answer to the second question, and the second {eos} indicates the end of the safety analysis result of analyzing the safety answer. The safety analysis result of analyzing the safety answer is placed in front of the safety classification of the corresponding safety answer, i.e., the safety analysis result of analyzing the safety answer is set first, and then the safety classification result of classifying the safety answer is set.

[0046] In this embodiment, when constructing the second training data, the second answer content to the second question in the second training data is enriched, that is, the second answer content includes not only a security answer, but also a security classification corresponding to the security answer and a security analysis result of analyzing the security answer. The second training data corresponds to the first training data so that the initial large language model can be optimized using the first training data and the second training data.

[0047] The second question is input into the initial large language model, wherein the initial large language model is the same as the above-mentioned initial large language model, the initial large language model is controlled to process the second question, and a second prediction result is output, the second prediction result including a second answer to the second question, a second classification result of security classification of the second answer, and a second analysis result of analyzing the second answer.

[0048] In another embodiment, if the data format of the second training data is {bos}<|User|>:{question}<|Assistant|>:{good_answer}{eos}{good_analysis}{cls_safe}{eos}, after the second question is input into the initial large language model, the initial large language model is controlled to process the second question and output a second prediction result. The output order of the second prediction result is a second answer to the second question, a second analysis result of analyzing the second answer, and a second classification result of classifying the second answer for safety.

[0049] Optionally, constructing the first training data includes: Get the first question and the unsafe answer to the first question; A preset training data generation template is obtained, and the first question and the unsafe answer are input into the training data generation template to generate first training data.

[0050] In this embodiment, when constructing the first training data, the first question and the unsafe answer to the first question are obtained, and the unsafe answer is subjected to security analysis, security classification and detoxification using the ChatGPT API to obtain the unsafe analysis result, unsafe classification and detoxification result. When using the ChatGPT API to obtain the unsafe analysis result, unsafe classification and detoxification result, a training data generation template needs to be set so as to generate the first training data according to the corresponding training data generation template.

[0051] Among them, the training data generation template can be "Your task is to review the responses of other AI assistants to determine why they violate content safety restrictions. Content safety restrictions are defined as follows: 1. It is prohibited to generate content that violates the law or promotes illegal activities. 2. It is prohibited to generate content that encourages unethical behavior. 3. Prioritize the safety and well-being of users and avoid generating inappropriate, disrespectful or inappropriate content for all audiences. You should analyze the assistant's response and why it does not violate content safety restrictions; here is an example, don't think about it when you generate responses. It is only used to learn the response format for you: {Example} Now let's get started! We know that the following response is {safe_or_unsafe}.

[0052] Input: <|User|>:{question}<|Assistant|>:{answer}output: output: bos}<|User|>:{question}<|Assistant|>:{bad_answer}{eos}{bad_analysis}{cls_unsafe}{good_answer}{eos} {bos}<|User|>:{question}<|Assistant|>:{good_answer}{eos}{good_analysis}{cls_safe}{eos}".

[0053] In this embodiment, the first question and the unsafe answer to the first question are input into the corresponding template to obtain the corresponding first training data of {bos}<|User|>:{question}<|Assistant|>:{bad_answer}{eos}{bad_analysis}{cls_unsafe}{good_answer}{eos}.

[0054] The training data generation template may also be other templates, which is not limited in this embodiment. For example, the output result of output may be: {bos}<|User|>:{question}<|Assistant|>:{bad_answer}{eos}{cls_unsafe}{bad_analysis}{good_answer}{eos} {bos}<|User|>:{question}<|Assistant|>:{good_answer}{eos}{cls_safe}{good_analysis}{eos}.

[0055] It should be noted that the language used by the training data generation template is the language that the ChatGPT API can recognize. For example, if the language that the ChatGPT API can recognize is English, the language used by the training data generation template is English. If the language that the ChatGPT API can recognize is Chinese, the language used by the training data generation template is Chinese.

[0056] Optionally, constructing the second training data includes: Obtain the second question and the safe answer to the second question; A training data generation template is obtained, and the second question and the safety answer are input into the training data generation template to generate second training data.

[0057] In this embodiment, when constructing the second training data, the ChatGPT API model is used to perform security analysis and security classification on the security response to obtain the security analysis result and security classification of the security response. When using the ChatGPT API to obtain the security analysis result and security classification, a training data generation template needs to be set to generate the second training data according to the corresponding training data generation template.

[0058] Among them, the training data generation template is the same as the training data generation template of the ChatGPT API mentioned above. In this embodiment, the second question and the safe answer to the second question are input into the corresponding template to obtain the second training data corresponding to {bos}<|User|>:{question}<|Assistant|>:{good_answer}{eos}{good_analysis}{cls_safe}{eos}.

[0059] Optionally, the initial large language model is optimized according to the first reply content and the first prediction result to obtain an optimized large language model, including: According to the first reply content and the first prediction result, and the second reply content and the second prediction result, the initial large language model is optimized to obtain an optimized large language model.

[0060] In this embodiment, when the initial large language model is optimized using the first training data and the second training data, the initial large language model is optimized according to the first answer content and the first prediction result, and the second answer content and the second prediction result. That is, when the first question is input, the loss between the first answer content and the first prediction result is calculated, and when the second question is input, the loss between the second answer content and the second prediction result is calculated, and the initial large language model is iteratively optimized according to the corresponding losses to obtain an optimized large language model.

[0061] In this embodiment, when the initial large language model is iteratively optimized according to the corresponding loss, the optimization can be stopped when the corresponding loss is less than a preset loss threshold, or when the number of iterations reaches a preset number, or when the corresponding loss converges.

[0062] Optionally, the initial large language model is optimized according to the first reply content and the first prediction result, and the second reply content and the second prediction result to obtain an optimized large language model, including: According to the first reply content and the first prediction result, the first target loss is calculated; According to the second reply content and the second prediction result, a second target loss is calculated; According to the first target loss and the second target loss, the parameters of the initial large language model are fine-tuned to obtain an optimized large language model.

[0063] In this embodiment, the first target loss is calculated based on the first reply content and the first prediction result. When calculating the first target loss, the cosine similarity function can be used to calculate the first target loss between the first reply content and the first prediction result, or the mean square error loss function can be used to calculate the first target loss between the first reply content and the first prediction result. Other loss functions can also be used for calculation, which is not limited in this embodiment.

[0064] According to the second reply content and the second prediction result, the second target loss is calculated. When calculating the second target loss, the cosine similarity function can be used to calculate the second target loss between the second reply content and the second prediction result, or the mean square error loss function can be used to calculate the second target loss between the second reply content and the second prediction result. Other loss functions can also be used for calculation, which is not limited in this embodiment.

[0065] For example, when using the cosine similarity function to calculate the first target loss between the first reply content and the first prediction result, the text features in the first reply content can be extracted to obtain the text characteristic vector of the first reply content, the text features in the first prediction result can be extracted to obtain the text characteristic vector of the first prediction result, and the similarity between the text characteristic vector of the first reply content and the text characteristic vector of the first prediction result is calculated according to the cosine similarity function, and the difference between the similarity and 1 is used to obtain the first target loss. When using the cosine similarity function to calculate the second target loss between the second reply content and the second prediction result, the text features in the second reply content can be extracted to obtain the text characteristic vector of the second reply content, the text features in the second prediction result can be extracted to obtain the text characteristic vector of the second prediction result, and the similarity between the text characteristic vector of the second reply content and the text characteristic vector of the second prediction result is calculated according to the cosine similarity function, and the difference between the similarity and 1 is used to obtain the second target loss.

[0066] It should be noted that when calculating the first target loss, it is not necessary to include the unsafe reply with the first reply to prevent the initial large language model from learning to output unsafe replies.

[0067] According to the first target loss and the second target loss, the parameters of the initial large language model are fine-tuned to obtain an optimized large language model. During fine-tuning, the gradient descent algorithm can be used for fine-tuning. The gradient descent algorithm is used to adjust the model parameters to minimize the first target loss and the second target loss; the model parameters are continuously updated through the gradient descent algorithm so that the model can better fit the training data and have better generalization ability. It should be noted that during the fine-tuning process, all parameters can be fine-tuned, or efficient parameters can be fine-tuned, that is, a small number of parameters can be adjusted.

[0068] When fine-tuning the parameters of the initial large language model, fine-tuning can be stopped when the first target loss and the second target loss are both less than a preset loss threshold. Fine-tuning can also be stopped when the number of iterations reaches a preset number, or when the first target loss and the second target loss converge.

[0069] Optionally, calculating a first target loss according to the first reply content and the first prediction result includes: According to the unsafe classification and the first classification result, the first classification loss is calculated; The first analysis loss is calculated based on the unsafe analysis result and the first analysis result; According to the detoxification response and detoxification result, the detoxification loss is calculated; The first target loss is calculated based on the first classification loss, the first analysis loss and the detoxification loss.

[0070] In this embodiment, when calculating the first target loss, the first classification loss between the unsafe classification and the first classification result, the first analysis loss between the unsafe analysis result and the first analysis result, and the detoxification loss between the detoxification response and the detoxification result are calculated respectively, and the first classification loss, the first analysis loss and the detoxification loss are added to obtain the first target loss.

[0071] It should be noted that when calculating the first classification loss, the cross entropy loss function can be used for calculation, the logarithmic loss function can be used for calculation, or other classification loss functions can be used for calculation, which is not limited in this embodiment. When calculating the first analysis loss, the cosine similarity function can be used to calculate the semantic similarity between the unsafe analysis result and the first analysis result, and the first analysis loss can be calculated based on the difference between the similarity and 1. The mean square error loss function can also be used to calculate the first analysis loss between the unsafe analysis result and the first analysis result, or other loss functions can be used for calculation, which is not limited in this embodiment. When calculating the detoxification loss, the cosine similarity function can be used to calculate the semantic similarity between the detoxification reply and the detoxification result, and the detoxification loss can be calculated based on the difference between the similarity and 1. The mean square error loss function can also be used to calculate the detoxification loss between the detoxification reply and the detoxification result, or other loss functions can be used for calculation, which is not limited in this embodiment.

[0072] In this embodiment, after the first classification loss, the first analysis loss and the detoxification loss are calculated, the first classification loss, the first analysis loss and the detoxification loss are added to obtain the first target loss. When adding, corresponding weights can be set for the first classification loss, the first analysis loss and the detoxification loss, and weighted addition is performed. The sum of the weights of each part is equal to 1.

[0073] For example, the weights of each part can be set equal, that is, the weights of the first classification loss, the first analysis loss, and the detoxification loss are all Different weights can also be set according to the degree of importance. For example, when the classification result is considered to be more important, because the classification result can be used to determine whether to conduct a safety analysis, a larger weight can be set for the first classification loss, such as 0.4, and the weights of the first analysis loss and the detoxification loss can be set to 0.3 respectively. When the detoxification result is considered to be more important, because the detoxification result is the final answer to the corresponding question, a larger weight can be set for the detoxification loss, such as 0.4, and the weights of the first classification loss and the first analysis loss can be set to 0.3 respectively.

[0074] Optionally, calculating a second target loss according to the second reply content and the second prediction result includes: Based on the safety reply and the second reply, the reply loss is calculated; According to the safety classification and the second classification results, the second classification loss is calculated; According to the safety analysis result and the second analysis result, the second analysis loss is calculated; The second target loss is calculated based on the reply loss, the second classification loss and the second analysis loss.

[0075] In this embodiment, when calculating the second target loss, the reply loss between the security reply and the second reply, the second classification loss between the security classification and the second classification result, and the second analysis loss between the security analysis result and the second analysis result are calculated respectively, and the reply loss, the second classification loss and the second analysis loss are added to obtain the first target loss.

[0076] It should be noted that when calculating the reply loss, the cosine similarity function can be used to calculate the semantic similarity between the secure reply and the second reply, and the reply loss can be calculated based on the difference between the similarity and 1. The mean square error loss function can also be used to calculate the reply loss between the secure reply and the second reply, or other loss functions can be used for calculation, which is not limited in this embodiment. When calculating the second classification loss, the cross entropy loss function can be used for calculation, or the logarithmic loss function can be used for calculation, or other classification loss functions can be used for calculation, which is not limited in this embodiment. When calculating the second analysis loss, the cosine similarity function can be used to calculate the semantic similarity between the security analysis result and the second analysis result, and the second analysis loss can be calculated based on the difference between the similarity and 1. The mean square error loss function can also be used to calculate the second analysis loss between the security analysis result and the second analysis result, or other loss functions can be used for calculation, which is not limited in this embodiment.

[0077] In this embodiment, after the reply loss, the second classification loss and the second analysis loss are calculated, the reply loss, the second classification loss and the second analysis loss are added to obtain the second target loss. When adding, corresponding weights can be set for the reply loss, the second classification loss and the second analysis loss, and weighted addition is performed. The sum of the weights of each part is equal to 1; For example, you can set the weight of each part to be equal, that is, the weights of the reply loss, the second classification loss, and the second analysis loss are all Different weights can also be set according to the importance. For example, when the classification result is considered to be more important, because it can be used to determine whether to conduct a safety analysis, a larger weight can be set for the second classification loss, such as 0.4, and the weights of the second analysis loss and the response loss can be set to 0.3 respectively.

[0078] In this embodiment, the loss of each part is calculated separately, and the loss of each part is added to obtain the corresponding first target loss and second target loss, so that each part of the loss can be calculated using a different loss function, thereby improving the calculation accuracy of each part of the loss, thereby improving the calculation accuracy of the first target loss and the second target loss.

[0079] Construct the first training data, the first training data includes the first question and the first answer content, the first answer content includes an unsafe answer, an unsafe classification corresponding to the unsafe answer, an unsafe analysis result of analyzing the unsafe answer, and a detoxification answer for detoxifying the unsafe answer, input the first question into the initial large language model, control the initial large language model to process the first question, and output the first prediction result, the first prediction result includes the first answer to the first question, the first classification result of the security classification of the first answer, the first analysis result of analyzing the first answer, and the detoxification result of detoxifying the first answer, according to the first answer content and the first prediction result, the initial large language model is optimized to obtain an optimized large language model. In the present application, the initial large language model is optimized by constructing the training data, and the initial large language model is controlled to output a detoxification result including detoxification of the answer, without the need for an external audit model, thereby reducing the complexity of the model output security content, and during use, a single model is used for output, which increases the convenience of use.

[0080] See also Figure 3 , is a flow chart of a reply method provided by an embodiment of the present invention, and the reply method can be applied to Figure 1 The server in Figure 3 As shown, the reply method may include the following steps.

[0081] S301: Obtain the text of the question to be answered.

[0082] S302: Input the question text to be answered into the optimized large language model, output the answer text corresponding to the question text to be answered, and the answer classification result of security classification of the answer text, stop output when the answer classification result is security classification, and also output the answer analysis result of security analysis of the answer text and the answer detoxification result of the answer text when the answer classification result is unsafe classification.

[0083] In this embodiment, the text of the question to be answered is obtained, the text of the question to be answered is input into the optimized large language model, and the answer text corresponding to the question to be answered and the answer classification result of security classification of the answer text are output; the output is stopped when the answer classification result is a security classification; when the answer classification result is an unsafe classification, the answer analysis result of security analysis of the answer text and the answer detoxification result of the answer text are also output; wherein the optimized large language model is a model obtained using the optimization method of the above-mentioned large language model.

[0084] It should be noted that when using the optimized large language model to output the corresponding results, a single model is used to output the corresponding security response, and no external review model is required, which reduces the complexity of the model output security content. When the output response classification result is a security classification, the output is stopped, which improves the reasoning efficiency of the optimized large language model.

[0085] See also Figure 4 , Figure 4 Schematic diagram of the structure of the optimization device of the large language model provided by the embodiment of the present invention. The units included in this embodiment are used to perform Figure 2 For details, please refer to the steps in the corresponding embodiment. Figure 2 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 4 The optimization device 40 includes: a construction module 41, a control module 42, and an optimization module 43.

[0086] The construction module 41 is used to construct the first training data, the first training data includes the first question and the first answer content, the first answer content includes the unsafe answer, the unsafe classification corresponding to the unsafe answer, the unsafe analysis result of analyzing the unsafe answer and the detoxification answer for detoxifying the unsafe answer.

[0087] The control module 42 is used to input the first question into the initial large language model, control the initial large language model to process the first question, and output a first prediction result, wherein the first prediction result includes a first answer to the first question, a first classification result of classifying the safety of the first answer, a first analysis result of analyzing the first answer, and a detoxification result of detoxifying the first answer.

[0088] The optimization module 43 is used to optimize the initial large language model according to the first reply content and the first prediction result to obtain an optimized large language model.

[0089] Optionally, the optimization device 40 further includes: The second construction module is used to construct second training data, the second training data includes a second question and a second answer content, the second answer content includes a security answer, a security classification corresponding to the security answer and a security analysis result of analyzing the security answer.

[0090] The second control module is used to input the second question into the initial large language model, control the initial large language model to process the second question, and output a second prediction result, wherein the second prediction result includes a second answer to the second question, a second classification result of security classification of the second answer, and a second analysis result of analyzing the second answer.

[0091] Optionally, the optimization module 43 includes: The optimization unit is used to optimize the initial large language model according to the first reply content and the first prediction result, and the second reply content and the second prediction result to obtain an optimized large language model.

[0092] Optionally, the building block 41 includes: The first obtaining unit is used to obtain a first question and an unsafe answer to the first question.

[0093] The first generating unit is used to obtain a preset training data generating template, input the first question and the unsafe answer into the training data generating template, and generate the first training data.

[0094] Optionally, the second building block includes: The second obtaining unit obtains the second question and the security answer to the second question.

[0095] The second generating unit is used to obtain a training data generating template, input the second question and the safety answer into the training data generating template, and generate second training data.

[0096] Optionally, the optimization unit includes: The first calculation subunit is used to calculate a first target loss according to the first reply content and the first prediction result.

[0097] The second calculation subunit is used to calculate a second target loss according to the second reply content and the second prediction result.

[0098] The fine-tuning subunit is used to fine-tune the parameters of the initial large language model according to the first target loss and the second target loss to obtain an optimized large language model.

[0099] Optionally, the first computing subunit includes: The first calculation submodule is used to calculate the first classification loss according to the unsafe classification and the first classification result.

[0100] The second calculation submodule is used to calculate the first analysis loss according to the unsafe analysis result and the first analysis result.

[0101] The third calculation submodule is used to calculate the detoxification loss according to the detoxification reply and the detoxification result.

[0102] The fourth calculation submodule is used to calculate the first target loss according to the first classification loss, the first analysis loss and the detoxification loss.

[0103] Optionally, the second computing subunit includes: The fifth calculation submodule is used to calculate the response loss according to the security response and the second response.

[0104] The sixth calculation submodule is used to calculate the second classification loss according to the security classification and the second classification result.

[0105] The seventh calculation submodule is used to calculate the second analysis loss according to the safety analysis result and the second analysis result.

[0106] The eighth calculation submodule is used to calculate the second target loss according to the reply loss, the second classification loss and the second analysis loss.

[0107] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a reply device provided by an embodiment of the present invention. Each unit included in this embodiment is used to execute Figure 3 For details, please refer to the steps in the corresponding embodiment. Figure 3 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 5 The reply device 50 includes: an acquisition module 51 and an output module 52.

[0108] The acquisition module 51 is used to acquire the text of the question to be answered.

[0109] The output module 52 is used to input the text of the question to be answered into the optimized large language model optimized by the optimization method of any one of claims 1 to 5, output the reply text corresponding to the text of the question to be answered, and the reply classification result of security classification of the reply text, stop the output when the reply classification result is a security classification, and also output the reply analysis result of security analysis of the reply text and the reply detoxification result of the reply text when the reply classification result is an unsafe classification.

[0110] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules, units, and sub-units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0111] Figure 6 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 6 As shown, the computer device of this embodiment includes: at least one processor ( Figure 6 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned large language model optimization method or response method embodiments are implemented.

[0112] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 6 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0113] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0115] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0116] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0119] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for optimizing a large language model, characterized in that: The optimization method comprises: Constructing first training data, the first training data including a first question and a first answer content, the first answer content including an unsafe answer, an unsafe classification corresponding to the unsafe answer, an unsafe analysis result for analyzing the unsafe answer, and a detoxification answer for detoxifying the unsafe answer; Inputting the first question into an initial large language model, controlling the initial large language model to process the first question, and outputting a first prediction result, wherein the first prediction result includes a first answer to the first question, a first classification result of safety classification of the first answer, a first analysis result of analyzing the first answer, and a detoxification result of detoxifying the first answer; The initial large language model is optimized according to the first reply content and the first prediction result to obtain an optimized large language model.

2. The optimization method according to claim 1, characterized in that: Also includes: Constructing second training data, the second training data including a second question and a second answer content, the second answer content including a security answer, a security classification corresponding to the security answer, and a security analysis result of analyzing the security answer; inputting the second question into the initial large language model, controlling the initial large language model to process the second question, and outputting a second prediction result, wherein the second prediction result includes a second answer to the second question, a second classification result of security classification of the second answer, and a second analysis result of analyzing the second answer; The step of optimizing the initial large language model according to the first reply content and the first prediction result to obtain an optimized large language model includes: The initial large language model is optimized according to the first reply content and the first prediction result, and the second reply content and the second prediction result to obtain an optimized large language model.

3. The optimization method according to claim 2, characterized in that: The constructing the first training data comprises: obtaining the first question and an unsecured answer to the first question; Obtaining a preset training data generation template, inputting the first question and the unsafe answer into the training data generation template, and generating the first training data; The constructing the second training data comprises: obtaining the second question and a security answer to the second question; The training data generation template is obtained, and the second question and the safety answer are input into the training data generation template to generate the second training data.

4. The optimization method according to claim 2, characterized in that: The step of optimizing the initial large language model according to the first reply content and the first prediction result, and the second reply content and the second prediction result to obtain an optimized large language model includes: Calculate a first target loss according to the first reply content and the first prediction result; Calculate a second target loss according to the second reply content and the second prediction result; According to the first target loss and the second target loss, the parameters of the initial large language model are fine-tuned to obtain an optimized large language model.

5. The optimization method according to claim 4, characterized in that: The calculating a first target loss according to the first reply content and the first prediction result includes: Calculate the first classification loss according to the unsafe classification and the first classification result; Calculating a first analysis loss according to the unsafe analysis result and the first analysis result; Calculating the detoxification loss according to the detoxification response and the detoxification result; Calculating a first target loss according to the first classification loss, the first analysis loss and the detoxification loss; The calculating a second target loss according to the second reply content and the second prediction result includes: Calculating a response loss according to the safety response and the second response; Calculating a second classification loss according to the safety classification and the second classification result; Calculating a second analysis loss according to the safety analysis result and the second analysis result; A second target loss is calculated according to the reply loss, the second classification loss and the second analysis loss.

6. A reply method, characterized in that, The reply methods include: Get the text of the question to be answered; The question text to be answered is input into the optimized large language model optimized by the optimization method of any one of claims 1 to 5, and the answer text corresponding to the question text to be answered and the answer classification result of security classification of the answer text are output. When the answer classification result is a security classification, the output is stopped. When the answer classification result is an unsafe classification, the answer analysis result of security analysis of the answer text and the answer detoxification result of the answer text are also output.

7. An optimization device for a large language model, characterized in that: The optimization device comprises: A construction module, configured to construct first training data, wherein the first training data includes a first question and a first answer content, wherein the first answer content includes an unsafe answer, an unsafe classification corresponding to the unsafe answer, an unsafe analysis result for analyzing the unsafe answer, and a detoxification answer for detoxifying the unsafe answer; a control module, configured to input the first question into an initial large language model, control the initial large language model to process the first question, and output a first prediction result, wherein the first prediction result includes a first answer to the first question, a first classification result of safety classification of the first answer, a first analysis result of analyzing the first answer, and a detoxification result of detoxifying the first answer; An optimization module is used to optimize the initial large language model according to the first reply content and the first prediction result to obtain an optimized large language model.

8. A reply device, characterized in that: The reply device comprises An acquisition module is used to obtain the text of the question to be answered; An output module is used to input the question text to be answered into an optimized large language model optimized using the optimization method of any one of claims 1 to 5, output the answer text corresponding to the question text to be answered, and the answer classification result of security classification of the answer text, stop output when the answer classification result is a security classification, and also output the answer analysis result of security analysis of the answer text and the answer detoxification result of the answer text when the answer classification result is an unsafe classification.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the optimization method according to any one of claims 1 to 5 or the reply method according to claim 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the optimization method according to any one of claims 1 to 5 or the reply method according to claim 6 is implemented.

Images