Training set construction method and device and electronic equipment

Abstract

The embodiment of the invention provides a training set construction method and device and electronic equipment, and is used for solving the problems that when a large model is finely adjusted based on RLHF in the related technology, a large amount of user feedback data needs to be collected to determine an accurate sorting result, a high-quality sorting result is difficult to obtain, and the iterative optimization speed of the large model is greatly influenced. In the embodiment of the invention, the electronic equipment obtains a preset number of knowledge documents related to a to-be-identified question based on a website obtained in advance; obtaining a sorting result of each candidate answer based on the large model and each obtained knowledge document; the electronic equipment can obtain the sequence of the candidate answers corresponding to the to-be-recognized questions through the large model, a high-quality sequence result can be obtained without collecting a large amount of user feedback data, and therefore the speed and quality of large model iterative optimization are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a training set construction method, device and electronic device. Background Art

[0002] Large models are developing rapidly. Currently, the industry has developed a variety of base large models and industry large models. Base large models include Large Language Model Meta AI (LLama), Qwen, ChatGlobal Language Model (ChatGLM), etc., and industry large models include news large models and education large models. In order to improve the accuracy of content generation of large models, eliminate illusions, and enhance user experience and satisfaction, OpenAI proposed a large model fine-tuning method based on reinforcement learning from human feedback (RLHF). In RLHF, reinforcement learning is combined with human feedback. Human feedback is used as a reward signal to guide model training and minimize useless, distorted or biased outputs, thereby enhancing the model's understanding and satisfaction of human intentions. RLHF has achieved good results in the ChatGPT series of commercial models, and has been widely used in the fine-tuning of various mainstream large models.

[0003] Although the RLHF method has achieved great success, it still has many problems. Among them, when fine-tuning a large model based on the RLHF method, it is necessary to obtain the ranking results corresponding to the accuracy of multiple candidate answers corresponding to the question. In related technologies, a large amount of user feedback data needs to be collected to determine the accurate ranking results. However, it is extremely difficult to obtain high-quality ranking results because it requires high labor costs for labeling, which seriously restricts the iterative optimization speed of large models. Summary of the invention

[0004] The embodiments of the present application provide a training set construction method, device and electronic device for solving the problem that when the related technology fine-tunes a large model based on RLHF, it is necessary to collect a large amount of user feedback data to determine accurate sorting results, but it is difficult to obtain high-quality sorting results, which greatly affects the speed of iterative optimization of the large model.

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a training set, the method comprising:

[0006] Receiving a question to be identified and each candidate answer corresponding to the question to be identified; acquiring a preset number of knowledge documents related to the question to be identified based on a pre-acquired website, wherein the acquired knowledge documents exist on the website;

[0007] Inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into the big model, and obtaining the sorting result of each candidate answer by the big model; wherein the sorting instruction is used to prompt the big model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy;

[0008] The question to be identified and each ranked candidate answer are added to a human feedback reinforcement learning training set.

[0009] In a second aspect, an embodiment of the present application further provides a training set construction device, the device comprising:

[0010] A receiving and obtaining module, configured to receive a question to be identified and each candidate answer corresponding to the question to be identified; based on a website obtained in advance, obtaining a preset number of knowledge documents related to the question to be identified, wherein the obtained knowledge documents exist in the website;

[0011] A processing module, used to input the problem to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction into the big model, and obtain the sorting result of each candidate answer by the big model; wherein the sorting instruction is used to prompt the big model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy;

[0012] An adding module is used to add the to-be-identified question and each ranked candidate answer to a human feedback reinforcement learning training set.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0014] A memory for storing program instructions;

[0015] The processor is used to call the program instructions stored in the memory and execute the steps included in any of the above-mentioned methods for constructing a training set according to the obtained program instructions.

[0016] In an embodiment of the present application, an electronic device receives a question to be identified and each candidate answer corresponding to the question to be identified; based on a pre-acquired website, a preset number of knowledge documents related to the question to be identified are acquired; the question to be identified, each candidate answer, each acquired knowledge document, and any preset sorting statement are input into a large model to obtain a sorting result of the large model for each candidate answer; wherein the sorting statement is used to prompt the large model to sort each candidate answer based on each knowledge document; the question to be identified and each sorted candidate answer are added to a training set, and the electronic device can obtain the sorting of the candidate answers corresponding to the question to be identified through the large model, without collecting a large amount of user feedback data, and can obtain high-quality sorting results, thereby improving the speed and quality of iterative optimization of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of the process of the training set construction method provided in the embodiment of the present application;

[0019] Figure 2 A schematic diagram of the structure of a fine-tuning large model provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a process for fine-tuning a large model provided in an embodiment of the present application;

[0021] Figure 4 A structural diagram of a training set construction device provided in an embodiment of the present application;

[0022] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0024] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0025] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0026] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.

[0027] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0028] In order to accurately and effectively obtain high-quality sorting results and improve the iterative optimization speed of large models, the embodiments of the present application provide a training set construction method, device and electronic device.

[0029] The training set construction method includes: receiving a question to be identified and each candidate answer corresponding to the question to be identified; based on a pre-acquired website, obtaining a preset number of knowledge documents related to the question to be identified, wherein the acquired knowledge documents exist in the website included in the website; inputting the question to be identified, each candidate answer, each acquired knowledge document and any preset sorting statement into a large model, and obtaining the sorting result of the large model for each candidate answer; wherein the sorting statement is used to prompt the large model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy; adding the question to be identified and each sorted candidate answer to the human feedback reinforcement learning training set.

[0030] Embodiment 1:

[0031] Figure 1 A process diagram of a training set construction method provided in an embodiment of the present application, the process includes the following steps:

[0032] S101: receiving a question to be identified and each candidate answer corresponding to the question to be identified; obtaining a preset number of knowledge documents related to the question to be identified based on a pre-acquired website, wherein the acquired knowledge documents exist in the website included in the website.

[0033] The training set construction method provided in the embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server.

[0034] In order to construct the training set accurately and effectively, the electronic device may first receive the question to be identified and each candidate answer corresponding to the question to be identified.

[0035] The electronic device has websites stored locally, where business personnel can pre-collect and build a collection of websites with credible data, which cover authoritative information sources in the fields of news, education, etc. The electronic device can obtain a preset number of knowledge documents related to the information to be identified based on the website. It should be noted that the website contains the obtained knowledge documents.

[0036] Among them, the online knowledge search component of the electronic device is responsible for retrieving knowledge documents related to the problem to be identified from a trusted source on the Internet. The online knowledge search component of the electronic device calls the search engine application programming interface (Application Programming Interface, API) to search for relevant trusted websites. During the search process, the online knowledge search component will retrieve a set of trusted websites through the search engine to obtain a set of knowledge documents related to the problem to be identified and each candidate answer D = {d1, d2, ..., d n}.

[0037] S102: Input the question to be identified, each candidate answer, each acquired knowledge document and any preset sorting statement into the big model to obtain the sorting result of each candidate answer by the big model; wherein the sorting statement is used to prompt the big model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy.

[0038] In order to accurately and effectively construct the training set, the electronic device can input the information to be identified, each candidate answer, each acquired knowledge stability, and any preset sorting statement into the large model. The large model can be open source or other large models, and a large parameter model with higher accuracy can be used. The large model is a large language model (LLM). LLM is a natural language processing model based on deep learning. It can learn the grammar and semantics of natural language, so as to generate human-readable text.

[0039] The sorting statement is used to prompt the large model to determine the accuracy of each candidate answer based on each knowledge document, and to sort each candidate answer based on the determined accuracy. The sorting statement needs to consider the accuracy of the description of the candidate question and the searched knowledge document, the logical matching degree, etc. Among them, the accuracy of the sort 1 is the highest, and the accuracy of the sort K is the lowest.

[0040] For example, the sorting statement can be: Please refer to the trusted knowledge document, and sort each candidate answer from best to worst based on the accuracy, logical matching, etc. of the comparison between the question to be identified and the events, dates, and characters in the knowledge document, where the one ranked as 1 has the highest accuracy and is the best.

[0041] The electronic device obtains the output of the large model, which is the ranking result of the accuracy of each candidate answer.

[0042] Among them, the online knowledge search component, which can be an electronic device, inputs the acquired knowledge documents as reference materials together with the question to be identified and each candidate answer into the LLM automatic sorting component. The LLM automatic sorting component sends the same question to be identified and multiple candidate answers together with the knowledge documents into the large model, and calls the candidate answers one by one by setting a special system prompt word, that is, the sorting statement described in the embodiment of the present application, and sorts and votes on all candidate answers based on the knowledge document set as a reference.

[0043] S103: Add the question to be identified and each ranked candidate answer to a training set.

[0044] After obtaining the sorting results of each candidate answer, the electronic device may add the information to be recognized and each sorted candidate answer to the corresponding training set for training based on RLHF.

[0045] The method provided by the embodiments of the present application is particularly applicable to fields such as news and education, which have high requirements for the accuracy of generating the text and have a large number of reliable online search knowledge sources.

[0046] In the embodiments of the present application, the electronic device receives the question to be recognized and each candidate answer corresponding to the question to be recognized; obtains a preset number of knowledge documents related to the question to be recognized based on the pre-obtained websites; inputs the question to be recognized, each candidate answer, each obtained knowledge document, and any preset sorting statement into the large model to obtain the sorting results of the large model for each candidate answer; wherein, the sorting statement is used to prompt the large model to sort each candidate answer based on each knowledge document; adds the question to be recognized and each sorted candidate answer to the training set. The electronic device can obtain the sorting of the candidate answers corresponding to the question to be recognized through the large model, and can obtain high-quality sorting results without collecting a large amount of user feedback data, thereby improving the speed and quality of the iterative optimization of the large model.

[0047] Embodiment 2:

[0048] In order to accurately and effectively obtain knowledge documents, based on the above embodiments, in the embodiments of the present application, the obtaining a preset number of knowledge documents related to the question to be recognized based on the pre-obtained websites includes:

[0049] Extract keywords in the information to be recognized, and search for a preset number of knowledge documents corresponding to the keywords based on the pre-obtained websites;

[0050] Determine that the obtained knowledge documents are the preset number of knowledge documents related to the question to be recognized.

[0051] After receiving the question to be recognized, the electronic device can extract keywords from the question to be recognized. These keywords are the core information in the question, and they can summarize the main content of the question. In one possible implementation, the electronic device can calculate the occurrence frequency of each word in the question to be recognized, and determine the words with an occurrence frequency higher than the threshold as keywords. It should be noted that when determining the occurrence frequency of each word, common stop words such as "de", "shi", "zai", etc. can be filtered out first, as these words are not very helpful for understanding the content of the question. In another possible implementation, the electronic device can also input the information to be recognized into a pre-trained extraction model to obtain the keywords in the information to be recognized output by the extraction model.

[0052] After the keywords are acquired, the electronic device can perform targeted searches based on the acquired keywords and use pre-collected and organized websites to search for a preset number of knowledge documents corresponding to the keywords.

[0053] In order to obtain knowledge documents accurately and effectively, based on the above embodiments, in the embodiment of the present application, searching for a preset number of knowledge documents corresponding to the keyword based on the pre-acquired website includes:

[0054] Use the Term Frequency-Inverse Document Frequency (TF-IDF) method to extract the TF-IDF value corresponding to each document in the acquired website and the keyword;

[0055] Each of the documents is sorted based on the TF-IDF value, and a preset number of documents ranked top are determined as a preset number of knowledge documents related to the problem to be identified.

[0056] In order to accurately and effectively obtain knowledge documents, electronic devices can use TF-IDF to quantify the strength of association between each document in the obtained website and the keywords of the question to be identified. Specifically, the electronic device first calculates the term frequency (TF) of the keyword in each document, that is, the ratio of the number of times the keyword appears in the document to the total number of words in the document, which reflects the importance of the keyword in the document. In addition, the electronic device also calculates the inverse document frequency (IDF), that is, the inverse logarithm of the number of documents containing the keyword in the total number of documents, which reflects the uniqueness and information content of the keyword in the entire document collection.

[0057] By combining TF and IDF, the electronic device can obtain the TF-IDF value of the keyword in each document. After obtaining the TF-IDF value corresponding to each document, the electronic device can sort each document from large to small based on the TF-IDF value. This process ensures that the documents that are most relevant to the keywords of the problem to be identified and have the highest information content can be ranked at the top. Based on the sorting results, the electronic device selects a preset number of documents that are ranked at the top, and determines the obtained documents as knowledge documents related to the problem to be identified.

[0058] In a possible implementation, the electronic device may use the TF-IDF method to extract keywords for searching some relatively long questions to be identified and corresponding candidate answers, and finally obtain knowledge documents.

[0059] Embodiment 3:

[0060] In order to accurately and effectively construct a training set, based on the above embodiments, in the embodiment of the present application, after obtaining a preset number of knowledge documents related to the problem to be identified, before inputting the problem to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction into the large model, the method further includes:

[0061] For each acquired knowledge document, if the text length in the knowledge document exceeds a preset length, the knowledge document is split into multiple sub-knowledge documents;

[0062] For each of the split sub-knowledge documents, a subsequent step is performed to input the question to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction into the large model.

[0063] In practical applications, if the text length of knowledge documents is too long, it may cause the large model to be unable to effectively read key information, making it difficult to build an effective training set. To solve this problem, electronic devices can split the text that is too long.

[0064] Specifically, the electronic device can first determine the text length of each acquired knowledge document. If the text length of a knowledge document exceeds the preset length, the electronic device will automatically trigger the splitting process to split the knowledge document into multiple sub-knowledge documents. Ensure that the text length of each sub-knowledge document is within the range that the large model can handle, thereby improving the efficiency of information extraction and model training.

[0065] In a possible implementation, the electronic device may split the knowledge document based on paragraphs. Specifically, the electronic device may identify paragraph separators in the knowledge document, such as line breaks, blank lines, or specific paragraph marks, and split the document into multiple sub-documents based on these separators.

[0066] In another possible implementation, the electronic device can also split the knowledge document based on chapters. Specifically, for a knowledge document containing multiple chapters or subsections, the electronic device can identify chapter titles or chapter separators, such as "Chapter 1", "Chapter 2", etc., and split the document into multiple sub-documents based on these titles or separators. This method can maintain the integrity and independence of the sub-documents and help preserve the original structure of the document.

[0067] That is to say, in the embodiment of the present application, when the knowledge document is long, the document is divided into chunks, and only the retrieved document chunks are sent to the large model.

[0068] Embodiment 4:

[0069] In order to accurately and effectively sort each candidate answer, based on the above embodiments, in the embodiment of the present application, the question to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction are input into the big model to obtain the sorting result of each candidate answer by the big model, including:

[0070] Inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into at least two large models, and obtaining the sorting results of each candidate answer by the at least two large models;

[0071] After obtaining the ranking results of each candidate answer by the at least two large models and before adding the question to be identified and each ranked candidate answer to a training set, the method further includes:

[0072] Based on the ranking results of each candidate answer output by each large model and the scores of different ranking results saved in advance, the target score corresponding to each candidate answer is determined, and each candidate answer is ranked based on the target score.

[0073] Since each candidate answer is ranked only by one large model, the ranking result may be inaccurate. In order to accurately and effectively rank the candidate answers, the electronic device can rank the candidate answers based on the ranking results of each candidate answer by multiple large models.

[0074] Specifically, the electronic device can input the question to be identified, each candidate answer, each acquired knowledge document, and any preset sorting result into at least two large models, and obtain the sorting results of each candidate answer by at least two large models.

[0075] After obtaining the ranking results output by each large model, the electronic device can determine a specific target score for each candidate answer based on the ranking results output by each large model for each candidate answer and in combination with the scores corresponding to the different ranking results saved in advance. Specifically, when there are two large models to sort the candidate answers, if a candidate answer ranks first in the ranking results output by the first model and ranks second in the ranking results output by the second model, the electronic device adds the pre-saved scores corresponding to the first and second positions of the candidate answer, and the sum is used as the target score of the candidate answer.

[0076] After determining the target score for each candidate answer, the electronic device can sort the candidate answers according to the obtained target score. Specifically, the electronic device can sort each candidate answer from large to small according to the corresponding target score. Among them, the training set sample is:

[0077] (prompt, response1, response2, response3,…)

[0078] Among them, prompt is the question to be identified, response1 is the most accurate and optimal candidate answer, and the subsequent ones are second.

[0079] The LLM automatic sorting component in the embodiment of the present application sends the same question to be identified and multiple candidate answers together with knowledge documents into multiple large models for sorting by voting. n} as a reference, all candidate answers are ranked and voted. In the case of a long knowledge document, for a document chunk, only the retrieved document chunk, ie, the sub-knowledge document described in the embodiment of the present application, is sent to the large model.

[0080] Embodiment 5:

[0081] In order to accurately and effectively adjust the parameters of the large model, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0082] Based on the problem to be identified and each ranked candidate answer, the reward model is trained; based on the trained reward model, the parameters of the large model to be adjusted are adjusted.

[0083] After obtaining the question to be identified and each of its corresponding ranked candidate answers, the electronic device can first train the reward model. This training process can enable the reward model to accurately understand the core requirements of the question and learn to assign corresponding reward values ​​based on the quality, relevance or other key indicators of the candidate answers. By deeply analyzing the ranked candidate answers, the reward model can learn which features or attributes make the answers better, thereby better guiding the screening and optimization of answers.

[0084] After the reward model training is completed, the electronic device can use the trained reward model to adjust the large model parameters to be adjusted.

[0085] In order to ensure the effectiveness and accuracy of parameter adjustment, electronic devices can use methods such as gradient descent, regularization, cross-validation, etc. to adjust the parameters of large models.

[0086] Embodiment 6:

[0087] In order to accurately and effectively train the reward model, on the basis of the above embodiments, in the embodiment of the present application, the reward model is trained based on the question to be identified and each ranked candidate answer, including:

[0088] Input the question to be identified and the first candidate answer after sorting into the reward model to obtain a first reward value output by the reward model; input the question to be identified and the last second candidate answer after sorting into the reward model to obtain a second reward value output by the reward model; determine a sorting loss value based on the deviation between the first reward value and the second reward value; and determine a penalty loss value based on the sum and difference of the preset penalty value and the first reward value respectively;

[0089] A loss value is determined according to the sorting loss value and the penalty loss value and their corresponding weights; and the reward model is trained based on the loss value.

[0090] In order to accurately and effectively train the reward model, the electronic device may input the question to be identified and the first candidate answer after sorting into the reward model. For the sake of distinction, the first candidate answer may be referred to as the first candidate answer, and the first reward value output by the reward model is obtained. Furthermore, the electronic device inputs the question to be identified and the last candidate answer after sorting into the reward model. For the sake of distinction, the last candidate answer may be referred to as the second candidate answer, and the second reward value output by the reward model is obtained.

[0091] After obtaining the first reward value and the second reward value, the electronic device may determine the sorting loss value based on the deviation between the first reward value and the second reward value. The electronic device may determine the output value of the Sigmoid function when the input value is the deviation, and determine the sorting loss value based on the output value. Specifically, the electronic device may determine the sorting loss value by the following formula:

[0092] L rank =-(1-2×max(0,P i,j -1 / 2)) γ log(P i,j )

[0093] Among them, P i,j =σ(r θ (x,y i )-r θ (x,y j )), P i,j is the output value of the Sigmoid function, σ represents the sigmoid function, r θ (x,y i ) is to input the question to be identified and the first candidate answer after sorting into the reward model, and obtain the first reward value output by the reward model, r θ (x,y j) is to input the question to be identified and the last second candidate answer after sorting into the reward model to obtain the second reward value output by the reward model, γ is a preset value, for example, it can be 2, L rank is the ranking loss value, i,j are the indices that distinguish different candidate answers.

[0094] And the electronic device determines the penalty loss value based on the sum and difference of the preset penalty value and the first reward value; specifically, the electronic device can determine the penalty loss value by the following formula:

[0095] L penalty =-(log(m+d)+log(dm))

[0096] Among them, m is the first reward value, L penalty is the penalty loss value, d is the preset penalty value, and the penalty value can also be called a hyperparameter for adjusting the penalty interval. Specifically, the value of d can be set according to actual needs.

[0097] It should be noted that the base of the logarithmic function in the above formula may be a preset value.

[0098] The electronic device can determine the loss value according to the sorting loss value and the penalty loss value and the corresponding weights; and train the reward model based on the loss value.

[0099] Specifically, the electronic device can determine the loss value by the following formula:

[0100] L loss =L rank +λL penalty

[0101] Among them, L loss is the loss value, L rank is the sorting loss value, L penalty is the penalty loss value, λ is the weight corresponding to the penalty loss value, and can also be called a hyperparameter.

[0102] Among them, the search feedback reward model training component of the electronic device is used to train a reward model that guides the fine-tuning process, so that the final fine-tuned large model output is closer to the preferred answer.

[0103] Figure 2 A schematic diagram of the structure of a fine-tuning large model provided in an embodiment of the present application.

[0104] Depend on Figure 2It can be seen that the electronic device can first obtain the question to be identified and each candidate answer, determine the search feedback data, that is, each candidate answer after sorting, based on the online knowledge search component and the LLM automatic sorting component, and train the reward model based on the reward model training component. Based on the reward model and the fine-tuning training set, the model fine-tuning component of reinforcement learning is used to fine-tune the large model to obtain the fine-tuned large model. Moreover, the large model is pre-trained by the pre-trained training set, and the supervised fine-tuning (SFT) large model is obtained by the fine-tuning training set.

[0105] Figure 3 A schematic diagram of a process for fine-tuning a large model provided in an embodiment of the present application, the process comprising the following steps:

[0106] S301: The online knowledge search component calls the search engine API to search related trusted websites to obtain knowledge documents.

[0107] S302: Call the LLM automatic sorting component to sort all candidate answers.

[0108] S303: Call the search feedback reward model training component to train the reward model.

[0109] S304: Call the reinforcement learning-based model fine-tuning component to fine-tune the SFT large model.

[0110] The embodiment of the present application proposes a large model fine-tuning system based on search feedback reinforcement learning (RLSF), which mainly includes four components: an online knowledge search component, an LLM automatic sorting component, a search feedback reward model (RM) training component, and a model fine-tuning component based on reinforcement learning. RLSF automatically sorts question and answer data pairs based on online knowledge search combined with LLM, so that the fine-tuned large model text answers are closer to the facts and reduce the occurrence of hallucinations. Among them, RLSF is a method that guides the training of the model by incorporating the feedback from online search into the training process, thereby enhancing the model's understanding and satisfaction of human intentions. Unlike the traditional RLHF method of collecting user feedback data to train a reward model, RLSF automatically sorts and scores question and answer data pairs based on online knowledge search combined with LLM, so that the fine-tuned large model text answers are closer to the facts and reduce the occurrence of hallucinations.

[0111] Embodiment 7:

[0112] In order to accurately and effectively adjust the parameters of the large model, based on the above embodiments, in the embodiment of the present application, the reward model based on the training is used to adjust the parameters of the large model to be adjusted, including:

[0113] Obtain any pre-saved sample question, input the sample question into the large model to be adjusted, and obtain an output answer output by the large model to be adjusted;

[0114] Inputting the sample question and the output answer into a trained reward model to obtain a target reward value output by the reward model;

[0115] Based on the pre-saved standard reward value and the target reward value, the parameters of the large model to be adjusted are adjusted.

[0116] In order to adjust the parameters of the large model to be adjusted, the electronic device pre-stores multiple sample questions of different types. The electronic device can obtain any pre-stored sample question, input the sample question into the large model to be adjusted, and obtain the answer output by the large model to be adjusted. For the sake of distinction, the answer can be called the output answer.

[0117] After obtaining the output answer, the electronic device inputs the sample question and the output answer into the trained reward model to obtain the target reward value output by the reward model. It should be noted that when the output answer is sufficiently standard, the target reward value output by the reward model will be close to or even equal to the standard reward value, and the accuracy of the output answer can be identified through the reward model.

[0118] The electronic device may adjust the parameters of the large model to be adjusted based on the pre-saved standard reward value and the target reward value.

[0119] Among them, the reinforcement learning-based model fine-tuning component of the electronic device is similar to the RLHF method. RLSF also uses the reinforcement learning (Proximal Policy Optimization, PPO) algorithm combined with the reward model to fine-tune the SFT large model, and generate a more accurate large model after fine-tuning.

[0120] Embodiment 8:

[0121] Based on the same inventive concept, the present application embodiment provides a training set construction device, please refer to Figure 4 , the device comprises:

[0122] The receiving and obtaining module 401 is used to receive the question to be identified and each candidate answer corresponding to the question to be identified; based on the website obtained in advance, obtain a preset number of knowledge documents related to the question to be identified, wherein the obtained knowledge documents exist in the website;

[0123] Processing module 402, used to input the problem to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction into the big model, and obtain the sorting result of each candidate answer by the big model; wherein the sorting instruction is used to prompt the big model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy;

[0124] The adding module 403 is used to add the question to be identified and each ranked candidate answer to the human feedback reinforcement learning training set.

[0125] In a possible implementation, the receiving and acquiring module 401 is specifically used to extract keywords from the information to be identified, and based on a pre-acquired website, search for a preset number of knowledge documents corresponding to the keywords; and determine that the acquired knowledge documents are a preset number of knowledge documents related to the problem to be identified.

[0126] In a possible implementation, the receiving and acquiring module 401 is specifically used to extract the TF-IDF value corresponding to the keyword for each document in the acquired website using the TF-IDF method; sort each document based on the TF-IDF value, and determine a preset number of documents with the highest sorting as a preset number of knowledge documents related to the problem to be identified.

[0127] In a possible implementation, the processing module 402 is also used to split each acquired knowledge document into multiple sub-knowledge documents if the text length in the knowledge document exceeds a preset length; and for each of the split sub-knowledge documents, execute subsequent steps of inputting the question to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into the large model.

[0128] In a possible implementation, the processing module 402 is specifically configured to input the question to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into at least two large models, and obtain the sorting results of each candidate answer by the at least two large models;

[0129] The processing module 402 is also used to determine the target score corresponding to each candidate answer based on the ranking result of each candidate answer output by each large model and the scores of different ranking results saved in advance, and to rank each candidate answer based on the target score.

[0130] In a possible implementation, the processing module 402 is further configured to train a reward model based on the question to be identified and each ranked candidate answer;

[0131] Adjust the parameters of the large model to be adjusted based on the trained reward model.

[0132] In one possible implementation, the processing module 402 is specifically used to input the question to be identified and the first first candidate answer after sorting into the reward model to obtain the first reward value output by the reward model; and input the question to be identified and the last second candidate answer after sorting into the reward model to obtain the second reward value output by the reward model; determine the sorting loss value based on the deviation between the first reward value and the second reward value; and determine the penalty loss value based on the sum and difference of the preset penalty value and the first reward value; determine the loss value according to the sorting loss value and the penalty loss value and the corresponding weights; and train the reward model based on the loss value.

[0133] In one possible implementation, the processing module 402 is specifically used to obtain any pre-saved sample question, input the sample question into the large model to be adjusted, and obtain an output answer output by the large model to be adjusted; input the sample question and the output answer into a trained reward model to obtain a target reward value output by the reward model; and adjust the parameters of the large model to be adjusted based on a pre-saved standard reward value and the target reward value.

[0134] Embodiment 9:

[0135] Based on the same inventive concept, the present application embodiment provides an electronic device, which can implement the function of constructing the training set in the previous text. Please refer to Figure 5 The device includes a processor 501, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 communicate with each other via the communication bus 505.

[0136] The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 performs the following steps:

[0137] Receiving a question to be identified and each candidate answer corresponding to the question to be identified; acquiring a preset number of knowledge documents related to the question to be identified based on a pre-acquired website, wherein the acquired knowledge documents exist on the website;

[0138] Inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into the big model, and obtaining the sorting result of each candidate answer by the big model; wherein the sorting instruction is used to prompt the big model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy;

[0139] The question to be identified and each ranked candidate answer are added to a human feedback reinforcement learning training set.

[0140] In a possible implementation manner, acquiring a preset number of knowledge documents related to the problem to be identified based on the pre-acquired website includes:

[0141] Extracting keywords from the information to be identified, and searching for a preset number of knowledge documents corresponding to the keywords based on the pre-acquired websites;

[0142] It is determined that the acquired knowledge documents are a preset number of knowledge documents related to the problem to be identified.

[0143] In a possible implementation, searching a preset number of knowledge documents corresponding to the keyword based on the pre-acquired website includes:

[0144] Use the TF-IDF method to extract the TF-IDF value corresponding to each document in the acquired website and the keyword;

[0145] Each of the documents is sorted based on the TF-IDF value, and a preset number of documents ranked top are determined as a preset number of knowledge documents related to the problem to be identified.

[0146] In a possible implementation, after obtaining a preset number of knowledge documents related to the problem to be identified, and before inputting the problem to be identified, each candidate answer, each obtained knowledge document, and any preset sorting instruction into the large model, the method further includes:

[0147] For each acquired knowledge document, if the text length in the knowledge document exceeds a preset length, the knowledge document is split into multiple sub-knowledge documents;

[0148] For each of the split sub-knowledge documents, a subsequent step is performed to input the question to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction into the large model.

[0149] In a possible implementation, inputting the question to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into a large model, and obtaining a sorting result of each candidate answer by the large model, includes:

[0150] Inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into at least two large models, and obtaining the sorting results of each candidate answer by the at least two large models;

[0151] After obtaining the ranking results of each candidate answer by the at least two large models and before adding the question to be identified and each ranked candidate answer to a training set, the method further includes:

[0152] Based on the ranking results of each candidate answer output by each large model and the scores of different ranking results saved in advance, the target score corresponding to each candidate answer is determined, and each candidate answer is ranked based on the target score.

[0153] In a possible implementation, the method further includes:

[0154] Based on the problem to be identified and each ranked candidate answer, the reward model is trained; based on the trained reward model, the parameters of the large model to be adjusted are adjusted.

[0155] In a possible implementation, the training of the reward model based on the question to be identified and each ranked candidate answer includes:

[0156] Input the question to be identified and the first candidate answer after sorting into the reward model to obtain a first reward value output by the reward model; input the question to be identified and the last second candidate answer after sorting into the reward model to obtain a second reward value output by the reward model; determine a sorting loss value based on the deviation between the first reward value and the second reward value; and determine a penalty loss value based on the sum and difference of the preset penalty value and the first reward value respectively;

[0157] A loss value is determined according to the sorting loss value and the penalty loss value and their corresponding weights; and the reward model is trained based on the loss value.

[0158] In a possible implementation, adjusting the parameters of the large model to be adjusted based on the trained reward model includes:

[0159] Obtain any pre-saved sample question, input the sample question into the large model to be adjusted, and obtain an output answer output by the large model to be adjusted;

[0160] Inputting the sample question and the output answer into a trained reward model to obtain a target reward value output by the reward model;

[0161] Based on the pre-saved standard reward value and the target reward value, the parameters of the large model to be adjusted are adjusted.

[0162] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0163] The communication interface 502 is used for communication between the electronic device and other devices.

[0164] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0165] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0166] Embodiment 10:

[0167] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing:

[0168] Receiving a question to be identified and each candidate answer corresponding to the question to be identified; acquiring a preset number of knowledge documents related to the question to be identified based on a pre-acquired website, wherein the acquired knowledge documents exist on the website;

[0169] Inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into the big model, and obtaining the sorting result of each candidate answer by the big model; wherein the sorting instruction is used to prompt the big model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy;

[0170] The question to be identified and each ranked candidate answer are added to a human feedback reinforcement learning training set.

[0171] In a possible implementation manner, acquiring a preset number of knowledge documents related to the problem to be identified based on the pre-acquired website includes:

[0172] Extracting keywords from the information to be identified, and searching for a preset number of knowledge documents corresponding to the keywords based on the pre-acquired websites;

[0173] It is determined that the acquired knowledge documents are a preset number of knowledge documents related to the problem to be identified.

[0174] In a possible implementation, searching a preset number of knowledge documents corresponding to the keyword based on the pre-acquired website includes:

[0175] Use the TF-IDF method to extract the TF-IDF value corresponding to each document in the acquired website and the keyword;

[0176] Each of the documents is sorted based on the TF-IDF value, and a preset number of documents ranked top are determined as a preset number of knowledge documents related to the problem to be identified.

[0177] In a possible implementation, after obtaining a preset number of knowledge documents related to the problem to be identified, and before inputting the problem to be identified, each candidate answer, each obtained knowledge document, and any preset sorting instruction into the large model, the method further includes:

[0178] For each acquired knowledge document, if the text length in the knowledge document exceeds a preset length, the knowledge document is split into multiple sub-knowledge documents;

[0179] For each of the split sub-knowledge documents, a subsequent step is performed to input the question to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction into the large model.

[0180] In a possible implementation, inputting the question to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into a large model, and obtaining a sorting result of each candidate answer by the large model, includes:

[0181] Inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into at least two large models, and obtaining the sorting results of each candidate answer by the at least two large models;

[0182] After obtaining the ranking results of each candidate answer by the at least two large models and before adding the question to be identified and each ranked candidate answer to a training set, the method further includes:

[0183] Based on the ranking results of each candidate answer output by each large model and the scores of different ranking results saved in advance, the target score corresponding to each candidate answer is determined, and each candidate answer is ranked based on the target score.

[0184] In a possible implementation, the method further includes:

[0185] Based on the problem to be identified and each ranked candidate answer, the reward model is trained; based on the trained reward model, the parameters of the large model to be adjusted are adjusted.

[0186] In a possible implementation, the training of the reward model based on the question to be identified and each ranked candidate answer includes:

[0187] Input the question to be identified and the first candidate answer after sorting into the reward model to obtain a first reward value output by the reward model; input the question to be identified and the last second candidate answer after sorting into the reward model to obtain a second reward value output by the reward model; determine a sorting loss value based on the deviation between the first reward value and the second reward value; and determine a penalty loss value based on the sum and difference of the preset penalty value and the first reward value respectively;

[0188] A loss value is determined according to the sorting loss value and the penalty loss value and their corresponding weights; and the reward model is trained based on the loss value.

[0189] In a possible implementation, adjusting the parameters of the large model to be adjusted based on the trained reward model includes:

[0190] Obtain any pre-saved sample question, input the sample question into the large model to be adjusted, and obtain an output answer output by the large model to be adjusted;

[0191] Inputting the sample question and the output answer into a trained reward model to obtain a target reward value output by the reward model;

[0192] Based on the pre-saved standard reward value and the target reward value, the parameters of the large model to be adjusted are adjusted.

[0193] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0194] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0195] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0196] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0197] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A training set construction method, characterized in that: The method comprises: Receiving a question to be identified and each candidate answer corresponding to the question to be identified; acquiring a preset number of knowledge documents related to the question to be identified based on a pre-acquired website, wherein the acquired knowledge documents exist on the website; Inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into the big model, and obtaining the sorting result of each candidate answer by the big model; wherein the sorting instruction is used to prompt the big model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy; The question to be identified and each ranked candidate answer are added to a human feedback reinforcement learning training set.

2. The method according to claim 1, characterized in that The step of obtaining a preset number of knowledge documents related to the problem to be identified based on the pre-acquired website includes: Extracting keywords from the information to be identified, and searching for a preset number of knowledge documents corresponding to the keywords based on the pre-acquired websites; It is determined that the acquired knowledge documents are a preset number of knowledge documents related to the problem to be identified.

3. The method according to claim 2, characterized in that The step of searching a preset number of knowledge documents corresponding to the keyword based on the pre-acquired website includes: Use the term frequency-inverse document frequency TF-IDF method to extract the TF-IDF value corresponding to each document in the acquired website and the keyword; Each of the documents is sorted based on the TF-IDF value, and a preset number of documents ranked top are determined as a preset number of knowledge documents related to the problem to be identified.

4. The method according to claim 1, characterized in that After obtaining a preset number of knowledge documents related to the problem to be identified, and before inputting the problem to be identified, each candidate answer, each obtained knowledge document, and any preset sorting instruction into the large model, the method further includes: For each acquired knowledge document, if the text length in the knowledge document exceeds a preset length, the knowledge document is split into multiple sub-knowledge documents; For each of the split sub-knowledge documents, a subsequent step is performed to input the question to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction into the large model.

5. The method according to claim 1, characterized in that The step of inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into the big model, and obtaining the sorting result of each candidate answer by the big model, includes: Inputting the problem to be identified, each candidate answer, each acquired knowledge document, and any preset sorting instruction into at least two large models, and obtaining the sorting results of each candidate answer by the at least two large models; After obtaining the ranking results of each candidate answer by the at least two large models and before adding the question to be identified and each ranked candidate answer to a training set, the method further includes: Based on the ranking results of each candidate answer output by each large model and the scores of different ranking results saved in advance, the target score corresponding to each candidate answer is determined, and each candidate answer is ranked based on the target score.

6. The method according to claim 1, characterized in that The method further comprises: Based on the problem to be identified and each ranked candidate answer, the reward model is trained; based on the trained reward model, the parameters of the large model to be adjusted are adjusted.

7. The method according to claim 6, characterized in that The reward model is trained based on the question to be identified and each ranked candidate answer, including: Input the question to be identified and the first candidate answer after sorting into the reward model to obtain a first reward value output by the reward model; input the question to be identified and the last second candidate answer after sorting into the reward model to obtain a second reward value output by the reward model; determine a sorting loss value based on the deviation between the first reward value and the second reward value; and determine a penalty loss value based on the sum and difference of the preset penalty value and the first reward value respectively; A loss value is determined according to the sorting loss value and the penalty loss value and their corresponding weights; and the reward model is trained based on the loss value.

8. The method according to claim 7, characterized in that The reward model that has been trained is used to adjust the parameters of the large model to be adjusted, including: Obtain any pre-saved sample question, input the sample question into the large model to be adjusted, and obtain an output answer output by the large model to be adjusted; Inputting the sample question and the output answer into a trained reward model to obtain a target reward value output by the reward model; Based on the pre-saved standard reward value and the target reward value, the parameters of the large model to be adjusted are adjusted.

9. A training set construction device, characterized in that: The device comprises: A receiving and obtaining module, configured to receive a question to be identified and each candidate answer corresponding to the question to be identified; based on a website obtained in advance, obtaining a preset number of knowledge documents related to the question to be identified, wherein the obtained knowledge documents exist in the website; A processing module, used to input the problem to be identified, each candidate answer, each acquired knowledge document and any preset sorting instruction into the big model, and obtain the sorting result of each candidate answer by the big model; wherein the sorting instruction is used to prompt the big model to determine the accuracy corresponding to each candidate answer based on each knowledge document, and sort each candidate answer based on the determined accuracy; An adding module is used to add the to-be-identified question and each ranked candidate answer to a human feedback reinforcement learning training set.

10. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method for constructing a training set according to any one of claims 1 to 8 according to the obtained program instructions.

Images