Model fine tuning training method and device, answer output method and device and electronic equipment

Abstract

The invention provides a model fine tuning training method and device, an answer output method and device and electronic equipment, and aims to solve the problem of low safety of a large language model. The method comprises the following steps: acquiring a plurality of question texts to be processed; for each question text in the plurality of question texts, acquiring a principle keyword corresponding to the question text, and generating a prompt text corresponding to the question text according to the question text and the principle keyword corresponding to the question text; inputting a prompt text corresponding to the question text into a large language model to enable the large language model to output a plurality of answer texts; determining the answer text with the highest security from the plurality of answer texts; and taking the prompt text corresponding to each question text as training data, taking the answer text with the highest security as a training label, and performing fine tuning training on the large language model to obtain the large language model after fine tuning training.

Description

Technical Field

[0001] The present application relates to the technical field of large language models and artificial intelligence, and more specifically, to a model fine-tuning training method, an answer output method, a device, and an electronic device. Background Art

[0002] At present, most large language models (LLMs) are developed and trained based on Transformer. Open source LLMs such as chatGLM and MOSS-SFT are found in actual practice that the content generated by these LLMs is sometimes not legal and compliant, and sometimes does not conform to social moral values. Therefore, for a large language model for commercial applications, the current large language model has low security. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a model fine-tuning training method, an answer output method, a device and an electronic device, which are used to improve the problem of low security of large language models.

[0004] The embodiment of the present application provides a model fine-tuning training method, including: obtaining multiple question texts to be processed; for each question text in the multiple question texts, obtaining the principle keywords corresponding to the question text, and generating the prompt text corresponding to the question text according to the question text and the principle keywords corresponding to the question text; inputting the prompt text corresponding to the question text into a large language model so that the large language model outputs multiple answer texts; determining the answer text with the highest security from the multiple answer texts; using the prompt text corresponding to each question text as training data and the answer text with the highest security as training labels, fine-tuning the large language model to obtain the large language model after fine-tuning training. In the implementation process of the above scheme, the large language model is fine-tuned specifically in the security field through the prompt text generated by the principle keywords corresponding to the question text and the answer text with the highest security, so that the large language model can not only better understand the context of security-related issues, but also more accurately identify and generate the answer text with the highest security, thereby promoting the self-optimization of the large language model to gradually adapt to the language patterns and logic unique to the security field, discover and learn some implicit knowledge of security experts, and significantly improve the security and reliability of the large language model compared to the general model without fine-tuning training.

[0005] Optionally, in an embodiment of the present application, the prompt text corresponding to the question text is input into the large language model, including: repeatedly inputting the prompt text corresponding to the question text into the large language model multiple times, so that the large language model outputs multiple answer texts. In the implementation process of the above scheme, although the input question text is the same, the large language model may generate texts slightly different in expression each time due to internal randomness (such as dropout mechanism) or different generation paths, which helps to explore different interpretation angles of the problem and the diversity of solutions. The output of multiple answer texts effectively reduces the output error caused by the randomness of the large language model and improves the stability of determining the answer text with the highest security.

[0006] Optionally, in an embodiment of the present application, before determining the answer text with the highest security from multiple answer texts, it also includes: inputting the question text into the large language model so that the large language model outputs a first answer text; adding the first answer text to the multiple answer texts. In the implementation process of the above scheme, by adding the first answer text output by the large language model to the multiple answer texts, and the answer text with the highest security among the multiple answer texts is used to fine-tune the training of the large language model, the quality of the training data is ensured, bad information is filtered out, and the model can be guided to generate more positive, beneficial and socially ethical responses, effectively reducing the risk of the model learning harmful, misleading or inappropriate content during the training process.

[0007] Optionally, in an embodiment of the present application, before determining the answer text with the highest security from multiple answer texts, it also includes: obtaining the security category of the question text and the principle keywords corresponding to the security category; determining a second answer text based on the question text, the security category of the question text and the principle keywords corresponding to the security category; adding the second answer text to the multiple answer texts. In the implementation process of the above scheme, by introducing principle keywords based on security categories to generate answers, the answer set can be prompted to include more diverse perspectives and more comprehensive information coverage, and even more appropriate responses can be provided for issues in specific sensitive or professional fields, thereby enhancing the diversity and comprehensiveness of the answer text, allowing the model to learn not only directly related answer content, but also how to generate answers based on security classifications and principles, which promotes the model's understanding of new situations and adaptive learning performance.

[0008] Optionally, in an embodiment of the present application, the second answer text is determined based on the question text, the security category of the question text, and the principle keywords corresponding to the security category, including: generating a prompt text corresponding to the question text based on the question text, the security category of the question text, and the principle keywords corresponding to the security category; inputting the prompt text corresponding to the question text into the large language model so that the large language model outputs the second answer text. In the implementation process of the above scheme, by adding prompts based on security categories and principle keywords, the large language model will receive clear guidance when generating answers, which helps to ensure that the output content is not only accurate and relevant, but also meets specific safety specifications and ethical standards, reduces the risk of generating misleading or inappropriate content, and effectively enhances the standardization and security of the answer text. Furthermore, through customized prompt text, the large language model can better focus on the core points and specific situations of the problem, so that the generated answers are closer to the real needs of users, improving the pertinence and practicality of the answers.

[0009] Optionally, in an embodiment of the present application, determining the answer text with the highest security from multiple answer texts includes: using a large language model to select multiple answer texts to obtain a selection result; and determining the answer text with the highest security based on the selection result. In the implementation of the above scheme, by using the comprehensive analysis ability of the model to make selections, the influence of subjective bias can be reduced. With the iteration of the selection process, the large language model can continuously learn and optimize its evaluation criteria, gradually adapt to the security requirements in different scenarios, and ultimately improve the adaptive learning ability of the large language model, so that the answer with the highest security finally determined is more objective and reliable, effectively improving the objectivity and reliability of the decision to determine the answer text with the highest security.

[0010] Optionally, in an embodiment of the present application, the selection result is the number of selections of the answer text; determining the answer text with the highest security based on the selection result includes: judging whether the number of selections of the answer text exceeds a preset threshold; if so, determining the answer text as the answer text with the highest security. In the implementation process of the above scheme, by setting a threshold to screen answers, content generation can be encouraged to develop in the direction of high quality and high security. This mechanism can filter out marginalized or high-risk answers, prompting the large language model to pay more attention to the security compliance of the content, making the final answer with the highest security more objective and reliable, and effectively improving the objectivity and reliability of the decision to determine the answer text with the highest security. Furthermore, the preset threshold can be flexibly adjusted according to actual conditions and policy requirements, which provides flexibility in responding to different security level requirements, especially in important sensitive periods or for questions and answers in specific fields, the threshold can be adjusted higher to strengthen risk management.

[0011] Optionally, in an embodiment of the present application, after obtaining the fine-tuned large language model, it also includes: inputting the question text to be processed into the fine-tuned large language model, so that the fine-tuned large language model outputs the answer text. In the implementation process of the above scheme, the large language model after fine-tuning is trained by using the prompt text generated by the principle keywords and the answer text with the highest security, so that the large language model can not only better understand the context of security-related problems, but also more accurately identify and generate the answer text with the highest security, which can significantly improve the security and reliability of the large language model compared to the general model without fine-tuning.

[0012] The embodiment of the present application also provides an answer output method, including: inputting the question text to be processed into a fine-tuned large language model, so that the fine-tuned large language model outputs the answer text; wherein the fine-tuned large language model is obtained by fine-tuning the model fine-tuning training method described above. In the implementation process of the above scheme, the large language model after fine-tuning is trained by using the prompt text generated by the principle keywords and the answer text with the highest security, so that the large language model can not only better understand the context of security-related issues, but also more accurately identify and generate the answer text with the highest security, which can significantly improve the security and reliability of the large language model compared to the general model that has not been fine-tuned.

[0013] The embodiment of the present application also provides a model fine-tuning training device, including: a question text acquisition module, used to acquire multiple question texts to be processed; a prompt text generation module, used to acquire the principle keywords corresponding to each question text in the multiple question texts, and generate the prompt text corresponding to the question text according to the question text and the principle keywords corresponding to the question text; an answer text output module, used to input the prompt text corresponding to the question text into a large language model, so that the large language model outputs multiple answer texts; an answer text determination module, used to determine the answer text with the highest security from multiple answer texts; a model fine-tuning training module, used to use the prompt text corresponding to each question text as training data and the answer text with the highest security as training label to fine-tune the large language model to obtain the large language model after fine-tuning training.

[0014] Optionally, in an embodiment of the present application, the answer text output module includes: a repeated input and output submodule, which is used to repeatedly input the prompt text corresponding to the question text into the large language model multiple times, so that the large language model outputs multiple answer texts.

[0015] Optionally, in an embodiment of the present application, the model fine-tuning training device also includes: a question text input module, used to input the question text into the large language model so that the large language model outputs a first answer text; an answer text adding module, used to add the first answer text to multiple answer texts.

[0016] Optionally, in an embodiment of the present application, the model fine-tuning training device also includes: a category keyword acquisition module, used to obtain the security category of the question text and the principle keywords corresponding to the security category; a first text determination module, used to determine the second answer text based on the question text, the security category of the question text and the principle keywords corresponding to the security category; a first text adding module, used to add the second answer text to multiple answer texts.

[0017] Optionally, in an embodiment of the present application, the first text determination module includes: a prompt text generation submodule, used to generate a prompt text corresponding to the question text based on the question text, the security category of the question text and the principle keywords corresponding to the security category; a text input and output submodule, used to input the prompt text corresponding to the question text into the large language model, so that the large language model outputs a second answer text.

[0018] Optionally, in an embodiment of the present application, the answer text determination module includes: a selection result acquisition submodule, which is used to use a large language model to select multiple answer texts to obtain a selection result; and an answer text determination submodule, which is used to determine the answer text with the highest security based on the selection result.

[0019] Optionally, in an embodiment of the present application, the selection result is the number of selections of the answer text; the answer text determination submodule includes: a selection number judgment unit, used to judge whether the number of selections of the answer text exceeds a preset threshold; a safe answer determination unit, used to determine the answer text as the answer text with the highest security if the number of selections of the answer text exceeds a preset threshold.

[0020] Optionally, in an embodiment of the present application, the model fine-tuning training device also includes: a fine-tuning model reasoning module, which is used to input the question text to be processed into the fine-tuned large language model so that the fine-tuned large language model outputs the answer text.

[0021] An embodiment of the present application also provides an answer output device, including: a question text input module, used to input the question text to be processed into a large language model after fine-tuning training, so that the large language model after fine-tuning training outputs an answer text; wherein the large language model after fine-tuning training is obtained by fine-tuning training of the model fine-tuning training method described above.

[0022] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0023] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0024] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flow chart of a model fine-tuning training method provided in an embodiment of the present application is shown;

[0027] Figure 2 A flowchart of the answer output method provided in an embodiment of the present application is shown;

[0028] Figure 3 A schematic diagram of the flow chart of the model fine-tuning training device provided in an embodiment of the present application is shown;

[0029] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0031] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings generally here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed, but merely represents the selected embodiments of the embodiments of the present application.

[0032] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0033] Before introducing the model fine-tuning training method provided in the embodiment of the present application, some concepts involved in the embodiment of the present application are first introduced:

[0034] Large Language Models (LLMs) refer to deep learning models trained using large-scale data sets. LLMs gain a deep understanding of language structure, grammar, semantics, and context by learning from large amounts of text data.

[0035] Fine-tuning is a training method in machine learning, which refers to using a pre-trained model (usually a general model that has been trained on a large dataset) to further optimize the training process for a specific task or field. This optimization training process includes adjusting the weights of the pre-trained model to better adapt it to the new task, rather than training a completely new model from scratch.

[0036] It should be noted that the model fine-tuning training method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and non-x86 servers include mainframes, minicomputers and UNIX servers.

[0037] The following is an example of an application scenario in which the model fine-tuning training method is applicable: When using open source LLMs such as chatGLM and MOSS-SFT, it is found that the content generated by these LLMs is sometimes not legal and compliant, and sometimes does not conform to social moral values. In this application scenario, the model fine-tuning training method can be used to perform fine-tuning (fine-turning) training on the large language model LLM specifically for the security field, so that the large language model can not only better understand the context of security-related issues, but also more accurately identify and generate the most secure answer text. Compared with general models that have not been fine-tuned, the security and reliability of the large language model can be significantly improved.

[0038] See also Figure 1 The flowchart of the model fine-tuning training method provided in the embodiment of the present application is shown; the embodiment of the present application provides a model fine-tuning training method, including:

[0039] Step S110: Acquire multiple question texts to be processed.

[0040] The above-mentioned question text may be a text related to LLM security, and may be obtained by collecting the question text from the Internet through an executable application, or may be generated by a self-instruct method, or may be a manually written question text. The above-mentioned multiple question texts may be a set of pre-classified questions, that is, the question set includes the question text and its corresponding category.

[0041] Step S120: for each question text among the multiple question texts, obtain the principle keywords corresponding to the question text, and generate a prompt text corresponding to the question text according to the question text and the principle keywords corresponding to the question text.

[0042] It is understandable that there are many ways to obtain the principle keywords corresponding to the question text as mentioned above. The correspondence between the question text and the principle keywords can be directly stored, and then the principle keywords corresponding to the question text can be directly read; of course, the correspondence between the category and the principle keywords can also be saved through the category of the question text. This correspondence between the question text and the principle keywords is an indirect correspondence.

[0043] Step S130: inputting the prompt text corresponding to the question text into the large language model so that the large language model outputs a plurality of answer texts.

[0044] In order to reduce the output error caused by the randomness inside the large language model (such as the dropout mechanism), in the specific practical operation process, the prompt text corresponding to the question text can be repeatedly input into the large language model, so that the large language model outputs multiple answer texts. Of course, the model parameters of the large language model can also be copied multiple times to obtain multiple model parameters, and the multiple model parameters can be run to obtain multiple large language models with the same model parameters. Then, the prompt text corresponding to the same question text is used to input multiple large language models with the same model parameters. Since the large language model itself has randomness (such as the dropout mechanism), it will obtain multiple different answer texts.

[0045] Step S140: Determine the answer text with the highest security from multiple answer texts.

[0046] It is understandable that, since the above-mentioned multiple answer texts are answers to the same question text by the large language model, and the same question text corresponds to one or more security principles, the large language model can determine the answer text with the highest security from the multiple answer texts based on these one or more security principles. The highest security of the above-mentioned answer text means that its question text has the highest security under the principle keywords corresponding to the question text, rather than the highest security of any keyword in the ordinary sense. For example: a certain answer text has higher security under cultural diversity, but its security under some equal principle keywords may not be higher, so the above-mentioned highest security is relative to the principle keywords of multiple answer texts.

[0047] Step S150: Using the prompt text corresponding to each question text as training data and the answer text with the highest security as training label, fine-tune the large language model to obtain the large language model after fine-tuning.

[0048] In the process of implementing the above solution, the large language model is fine-tuned specifically for the security field through the prompt text generated by the principle keywords corresponding to the question text and the answer text with the highest security. This enables the large language model to not only better understand the context of security-related issues, but also more accurately identify and generate the answer text with the highest security, thereby promoting the self-optimization of the large language model to gradually adapt to the language patterns and logic unique to the security field, discover and learn some implicit knowledge of security experts, and significantly improve the security and reliability of the large language model compared to the general model that has not been fine-tuned.

[0049] As an optional implementation of obtaining the principle keywords corresponding to the question text in the above step S120, if the correspondence between the question text and the principle keywords has been stored in a database or file system, the principle keywords corresponding to the question text can be directly read from the database or file system.

[0050] As another optional implementation method of obtaining the principle keywords corresponding to the question text in the above-mentioned step S120, if the first database or the first file system stores the correspondence between the category of the question text and the security category, and the second database or the second file system stores the correspondence between the security category and the principle keywords, then the security category corresponding to the question text can be first read from the first database or the first file system, and then the principle keywords corresponding to the security category can be read from the second database or the second file system.

[0051] As implementation methods of generating a prompt text corresponding to the question text in the above step S120, the following are included: a first implementation method, manually splicing the question text and the principle keywords corresponding to the question text into a prompt text corresponding to the question text, for example: the question text and the principle keywords corresponding to the question text are merged and spliced ​​into a coherent sentence, and then the coherent sentence is determined as the prompt text corresponding to the above question text, and the style of the prompt text can be that the question text is in the front and the principle keywords are in the back, or the question text is in the back and the principle keywords are in the front. A second implementation method, if a prompt template including a question field to be filled and a keyword field is prepared in advance, then the question text can be filled into the question field to be filled in the prompt template, and the principle keywords corresponding to the question text can be filled into the keyword field of the prompt template, so as to obtain the prompt text corresponding to the question text.

[0052] As an optional implementation of the above step S130, the above implementation of inputting the prompt text corresponding to the question text into the large language model may include:

[0053] Step S131: repeatedly inputting the prompt text corresponding to the question text into the large language model multiple times, so that the large language model outputs multiple answer texts.

[0054] An implementation example of the above step S131 is as follows: first, the prompt text corresponding to the question text is input into the large language model to obtain an answer text, and then the prompt text corresponding to the question text is repeatedly input into the large language model to obtain another answer text. Since the two answer texts are semantically similar, but most likely completely different in content, this method of repeatedly inputting the large language model so that the large language model outputs multiple answer texts can reduce the output error caused by the randomness of the large language model.

[0055] As an optional implementation of the above model fine-tuning training method, before determining the answer text with the highest security from multiple answer texts, the first answer text corresponding to the question text may be added to the above multiple answer texts. This implementation may include:

[0056] Step S132: input the question text into the large language model so that the large language model outputs a first answer text.

[0057] Step S133: Add the first answer text to the multiple answer texts.

[0058] The implementation method of the above-mentioned steps S132 to S133, for example: can interact with the large language model through a programming interface (API), specifically, the question text can be input into a large language model such as chatGLM, MOSS-SFT, etc. through a programming interface (API), so that the large language model such as chatGLM, MOSS-SFT, etc. outputs a first answer text, and the first answer text is added to the fine-tuning data set consisting of multiple answer texts.

[0059] As an optional implementation of the above-mentioned model fine-tuning training method, before determining the answer text with the highest security from multiple answer texts, the second answer text determined by the security category and its principle keywords can be added to the multiple answer texts. This implementation may include:

[0060] Step S134: Obtain the security category of the question text and the principle keywords corresponding to the security category.

[0061] For example, the implementation method of the above step S134 is: when the corresponding relationship between the category and the principle keyword has been saved through the category of the question text, the safety category of the question text can be directly obtained. The safety category here can be set according to the specific situation. If the above safety categories are too many or the setting is unreasonable, manual screening can be performed, and the safety category set after manual screening is used as the final category set, which includes but is not limited to: physical and mental health, public health, cultural differences, etc.

[0062] Step S135: Determine a second answer text according to the question text, the security category of the question text and the principle keywords corresponding to the security category.

[0063] In the actual practice, multiple security categories can be set for the same question text, but it is not advisable to set too many (for example, more than 10). Setting more than 10 security categories for the same question text will make it easy for LLM to focus on keywords when answering questions, rather than on the content of the question text itself.

[0064] Step S136: Add the second answer text to the multiple answer texts.

[0065] The implementation method of the above steps S135 to S136 may include: generating a prompt text corresponding to the question text according to the question text, the security category of the question text and the principle keywords corresponding to the security category. Specifically, the question text, the security category of the question text and the principle keywords corresponding to the security category may be manually spliced ​​into the prompt text corresponding to the question text, or the question text, the security category of the question text and the principle keywords corresponding to the security category may be filled into the corresponding fields of the prompt template. The corresponding fields of the prompt template may be set according to the specific situation, and the prompt text corresponding to the question text may be obtained. Then, the prompt text corresponding to the question text is input into a large language model such as chatGLM, MOSS-SFT, etc., so that the large language model such as chatGLM, MOSS-SFT, etc. outputs a second answer text. Finally, the second answer text is added to the fine-tuning data set composed of multiple answer texts.

[0066] As another optional implementation of the above step S130, the same prompt text can be input into multiple large language models with the same model parameters. For example, the implementation includes: making multiple copies of the model parameters of the large language model to obtain multiple model parameters, and running the multiple model parameters to obtain multiple large language models with the same model parameters. Then, the prompt text corresponding to the same question text is used to input multiple large language models with the same model parameters. Since the large language model itself has randomness, multiple different answer texts will be obtained.

[0067] As an optional implementation of the above step S140, the above implementation of determining the answer text with the highest security from multiple answer texts may include:

[0068] Step S141: Use a large language model to select multiple answer texts to obtain a selection result.

[0069] An implementation example of the above step S141 is: using a large language model to select multiple answer texts based on principle keywords. Specifically, the principle keywords and multiple answer texts can be manually spliced ​​into an answer selection prompt engineering text. The prompt engineering text here is, for example: "In accordance with the cultural diversity principles of equality, tolerance and respect for differences, which of the following multiple answer texts do you think is the answer text that best reflects the above principles? The content of answer text 1, the content of answer text 2, and the content of answer text 3". Then, the above answer selection prompt engineering text is input into the large language model to obtain the selection result output by the large language model. The selection result here represents the answer text with the highest security selected by the large language model, assuming it is answer text 2, and then repeat the above steps until the number of selections of answer text 2 exceeds a preset threshold (for example, 5 times or 10 times, etc.). Finally, the number of selections of each of the above answer texts is taken as the selection result. The following describes how to determine the final answer text with the highest security based on the selection result.

[0070] Step S142: Determine the answer text with the highest security based on the selection result.

[0071] As an optional implementation of the above step S142, the above selection result may be the number of times the answer text is selected; the above implementation of determining the answer text with the highest security according to the selection result may include:

[0072] Step S142a: Determine whether the number of selections of the answer text exceeds a preset threshold.

[0073] Step S142b: If the number of selections of the answer text exceeds a preset threshold, the answer text is determined as the answer text with the highest security.

[0074] An implementation example of the above step S142b is as follows: if the number of times the answer text is selected as the answer text with the highest security by the large language model exceeds a preset threshold (for example, 5 times or 10 times), then the answer text is determined as the answer text with the highest security to be added to the fine-tuning dataset, and finally a fine-tuning dataset including multiple answer texts with the highest security is obtained.

[0075] Step S142c: If the number of selections of the answer text does not exceed the preset threshold, continue to use the large language model to select multiple answer texts.

[0076] An example of an implementation method of the above step S142c is as follows: if the number of times the answer text is selected as the most secure by the large language model does not exceed a preset threshold, then return to execute step S141 to continue using the large language model to select multiple answer texts, so as to continue to add the answer text with the highest security to the fine-tuning data set. It is understandable that the fine-tuning data set may include prompt text (i.e., Prompt text) corresponding to the question text and answer text (i.e., Answer text) whose number of selections exceeds a preset threshold. Such Prompt text and Answer text may form a Prompt-Answer pair. The process of using the above Prompt-Answer pair to train the large language model is actually a process of fine-tuning the large language model using the prompt text corresponding to each question text as training data and the answer text with the highest security as training labels. The fine-tuning training process of the large language model is described below.

[0077] As another optional implementation of the above-mentioned step S140, the prompt text corresponding to each question text in the above-mentioned fine-tuning data set is used as the training data, and the answer text with the highest security in the above-mentioned fine-tuning data set is used as the training label. The large language model is fine-tuned by adopting full fine-tuning, P-tuning v2 or LoRA and other training methods. Therefore, the above-mentioned specific training method should not be used as a limitation of the embodiments of the present application.

[0078] As an optional implementation of the above-mentioned model fine-tuning training method, after obtaining the large language model after fine-tuning training, the large language model after fine-tuning training can also be used to process new question texts. This implementation may include:

[0079] Step S160: input the question text to be processed into the fine-tuned large language model so that the fine-tuned large language model outputs the answer text.

[0080] The implementation method of the above step S160 is, for example: it can be understood that after obtaining the fine-tuned large language model, the security of the fine-tuned large language model can also be verified. The specific verification can be to verify the security of the fine-tuned large language model through a test data set, for example: 70% of the data labels in the test data set are selected by the large language model as the answer text with the highest security. Of course, it is also possible to directly input the problem text to be processed into the fine-tuned large language model, so that the fine-tuned large language model outputs the answer text, thereby manually verifying whether the security of the fine-tuned large language model is improved.

[0081] See also Figure 2The flowchart of the answer output method provided by the embodiment of the present application is shown; the embodiment of the present application provides an answer output method, including:

[0082] Step S210: Obtain the text of the question to be processed.

[0083] Step S220: input the question text to be processed into the fine-tuned large language model so that the fine-tuned large language model outputs the answer text.

[0084] An implementation example of the above steps S210 to S220 is as follows: after the electronic device obtains the text of the question to be processed, the electronic device can directly send the text of the question to be processed to the server, so that the server uses the running fine-tuned large language model to process the question to be processed, and obtains and returns the answer text output by the fine-tuned large language model. The electronic device can receive the answer text output by the fine-tuned large language model sent by the server, and after obtaining the answer text sent by the server, the electronic device can also verify whether the security of the fine-tuned large language model is improved. Among them, the fine-tuned large language model is obtained by fine-tuning training of the model fine-tuning training method described above.

[0085] See also Figure 3 The flowchart of the model fine-tuning training device provided in the embodiment of the present application is shown; the embodiment of the present application provides a model fine-tuning training device 300, including:

[0086] The question text acquisition module 310 is used to acquire multiple question texts to be processed.

[0087] The prompt text generation module 320 is used to obtain the principle keywords corresponding to each question text among the multiple question texts, and generate the prompt text corresponding to the question text according to the question text and the principle keywords corresponding to the question text.

[0088] The answer text output module 330 is used to input the prompt text corresponding to the question text into the large language model so that the large language model outputs multiple answer texts.

[0089] The answer text determination module 340 is used to determine the answer text with the highest security from multiple answer texts.

[0090] The model fine-tuning training module 350 is used to fine-tune the large language model using the prompt text corresponding to each question text as training data and the answer text with the highest security as training label to obtain the large language model after fine-tuning training.

[0091] As an optional implementation of the above device, the answer text output module includes:

[0092] The repeated input and output submodule is used to repeatedly input the prompt text corresponding to the question text into the large language model so that the large language model outputs multiple answer texts.

[0093] As an optional implementation of the above device, the model fine-tuning training device further includes:

[0094] The question text input module is used to input the question text into the large language model so that the large language model outputs a first answer text.

[0095] The answer text adding module is used to add the first answer text to multiple answer texts.

[0096] As an optional implementation of the above device, the model fine-tuning training device further includes:

[0097] The category keyword acquisition module is used to obtain the security category of the question text and the principle keywords corresponding to the security category.

[0098] The first text determination module is used to determine the second answer text according to the question text, the security category of the question text and the principle keywords corresponding to the security category.

[0099] The first text adding module is used to add the second answer text into the multiple answer texts.

[0100] As an optional implementation manner of the above device, the first text determination module includes:

[0101] The prompt text generation submodule is used to generate a prompt text corresponding to the question text according to the question text, the security category of the question text and the principle keywords corresponding to the security category.

[0102] The text input and output submodule is used to input the prompt text corresponding to the question text into the large language model so that the large language model outputs a second answer text.

[0103] As an optional implementation of the above device, the answer text determination module includes:

[0104] The selection result obtaining submodule is used to select multiple answer texts using a large language model to obtain selection results.

[0105] The answer text determination submodule is used to determine the answer text with the highest security according to the selection result.

[0106] As an optional implementation of the above device, the selection result is the number of times the answer text is selected; the answer text determination submodule includes:

[0107] The selection times judging unit is used to judge whether the selection times of the answer text exceeds a preset threshold.

[0108] The secure answer determination unit is used to determine the answer text as the answer text with the highest security if the number of selections of the answer text exceeds a preset threshold.

[0109] As an optional implementation of the above device, the model fine-tuning training device further includes:

[0110] The fine-tuning model inference module is used to input the question text to be processed into the large language model after fine-tuning training, so that the large language model after fine-tuning training outputs the answer text.

[0111] The embodiment of the present application also provides an answer output device, including:

[0112] The question text input module is used to input the question text to be processed into the fine-tuned large language model so that the fine-tuned large language model outputs the answer text.

[0113] Among them, the large language model after fine-tuning training is obtained by fine-tuning training using the model fine-tuning training method described above.

[0114] It should be understood that the device corresponds to the above-mentioned model fine-tuning training method embodiment and can perform the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0115] See also Figure 4 An electronic device 400 provided in an embodiment of the present application includes: a processor 410 and a memory 420, wherein the memory 420 stores machine-readable instructions executable by the processor 410, and when the machine-readable instructions are executed by the processor 410, the above method is executed.

[0116] The embodiment of the present application also provides a computer-readable storage medium 430, on which a computer program is stored, and the computer program is executed by the processor 410 to execute the above method. The computer-readable storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0117] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor.

[0118] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0119] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0120] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradiction.

[0121] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A model fine-tuning training method, characterized in that: include: Get multiple question texts to be processed; For each question text among the multiple question texts, obtain the principle keywords corresponding to the question text, and generate a prompt text corresponding to the question text according to the question text and the principle keywords corresponding to the question text; Inputting a prompt text corresponding to the question text into a large language model so that the large language model outputs a plurality of answer texts; Determining the answer text with the highest security from the multiple answer texts; The prompt text corresponding to each question text is used as training data, and the answer text with the highest security is used as a training label, and the large language model is fine-tuned to obtain a large language model after fine-tuning training.

2. The method according to claim 1, characterized in that: The step of inputting the prompt text corresponding to the question text into the large language model comprises: The prompt text corresponding to the question text is repeatedly input into the large language model multiple times, so that the large language model outputs multiple answer texts.

3. The method according to claim 1, characterized in that: Before determining the answer text with the highest security from the multiple answer texts, the method further includes: Inputting the question text into the large language model so that the large language model outputs a first answer text; The first answer text is added to the multiple answer texts.

4. The method according to claim 1, characterized in that Before determining the answer text with the highest security from the multiple answer texts, the method further includes: Obtain the security category of the question text and the principle keywords corresponding to the security category; Determine a second answer text according to the question text, the security category of the question text and the principle keyword corresponding to the security category; The second answer text is added to the plurality of answer texts.

5. The method according to claim 4, characterized in that The step of determining the second answer text according to the question text, the security category of the question text, and the principle keywords corresponding to the security category includes: Generate a prompt text corresponding to the question text according to the question text, the security category of the question text and the principle keyword corresponding to the security category; The prompt text corresponding to the question text is input into the large language model, so that the large language model outputs the second answer text.

6. The method according to any one of claims 1 to 5, characterized in that: The step of determining the answer text with the highest security from the multiple answer texts includes: Select the multiple answer texts using the large language model to obtain a selection result; The answer text with the highest security is determined according to the selection result.

7. The method according to claim 6, characterized in that The selection result is the number of times the answer text is selected; and determining the answer text with the highest security according to the selection result includes: Determine whether the number of selections of the answer text exceeds a preset threshold; If so, the answer text is determined as the answer text with the highest security.

8. The method according to any one of claims 1 to 5, characterized in that: After obtaining the large language model after fine-tuning training, the method further includes: The question text to be processed is input into the large language model after fine-tuning training, so that the large language model after fine-tuning training outputs the answer text.

9. A method for outputting an answer, characterized in that: include: Inputting the question text to be processed into the fine-tuned large language model so that the fine-tuned large language model outputs the answer text; The large language model after fine-tuning training is obtained by fine-tuning training using the model fine-tuning training method described in any one of claims 1 to 8.

10. A model fine-tuning training device, characterized in that: include: A question text acquisition module is used to acquire multiple question texts to be processed; A prompt text generation module, for obtaining, for each question text among the plurality of question texts, a principle keyword corresponding to the question text, and generating a prompt text corresponding to the question text according to the question text and the principle keyword corresponding to the question text; An answer text output module, used for inputting a prompt text corresponding to the question text into a large language model, so that the large language model outputs a plurality of answer texts; An answer text determination module, used to determine the answer text with the highest security from the multiple answer texts; The model fine-tuning training module is used to use the prompt text corresponding to each question text as training data and the answer text with the highest security as training labels to fine-tune the large language model to obtain the large language model after fine-tuning training.

11. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform any method according to claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is executed.

13. A computer program product, characterized in that include: A computer program or a computer instruction, wherein when the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 9 is executed.

Images