Method, device, storage medium and electronic device for training a model

Abstract

Embodiments of the present specification disclose a method and device for training a model, a storage medium and an electronic device. A first output of a target model for a query sample is obtained without relying on an external knowledge source, and a second output obtained by searching the query sample by the external knowledge source. A knowledge boundary division result corresponding to the query sample is determined according to the first output, the second output and label information corresponding to the query sample. A preference data set is constructed according to a third output of the target model for the query sample, the label information and the knowledge boundary division result. The target model is trained based on the preference data set by a direct preference optimization method, and a trained target model is obtained, so that the trained target model can output the rejection response information when a target query input exceeds the parameter knowledge boundary and the search knowledge boundary.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a method and device for training a model, a storage medium and an electronic device. BACKGROUND

[0002] With the development of natural language processing technology, large language models (LLM) can significantly improve their performance in natural language processing tasks by integrating external knowledge sources when combined with retrieval systems, making responses more accurate and contextually rich. However, existing retrieval augmented fine-tuning (RAFT, a training method that improves the robustness of models to suboptimal retrieval results by fine-tuning with retrieval contexts containing noise) methods, while improving the robustness of models to noisy retrieval, still generate answers when there is a lack of reliable knowledge, resulting in hallucinations and weakening the reliability of the model. Therefore, a method is needed to enhance the refusal ability of the model when faced with uncertain or noisy retrieval results to improve its credibility in high-risk areas. SUMMARY

[0003] The purpose of the embodiments of the present specification is to provide a method and device for training a model, a storage medium and an electronic device.

[0004] The embodiments of the present specification provide a method for training a model, which accurately identifies whether a query is within the knowledge range of a model by dividing the knowledge boundary, constructs a high-quality data set for direct preference optimization according to the knowledge boundary division result corresponding to the query sample, and trains the model based on the direct preference optimization method by using the high-quality data set, so that the model can learn to generate accurate answers within the knowledge boundary, and at the same time, the model can be given the refusal ability to answer "I don't know" when the query exceeds the knowledge boundary, effectively balancing the accuracy and refusal ability of the model, enhancing the robustness and credibility of the model in high-risk areas, effectively solving the hallucination problem existing in the existing retrieval augmented fine-tuning method, and the method comprises:

[0005] Obtaining a first output of a target model for a query sample without relying on an external knowledge source, and a second output obtained by retrieving the query sample by the external knowledge source;

[0006] According to the first output, the second output and the label information corresponding to the query sample, determining a knowledge boundary division result corresponding to the query sample, wherein the knowledge boundary division result is used to indicate whether the query sample is located within the parameter knowledge boundary of the target model, and whether it is located within the retrieval knowledge boundary of the external knowledge source;

[0007] constructing a preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge boundary division result, wherein the preference dataset comprises the query sample, a preferred response and a non-preferred response corresponding to the query sample, and one of the preferred response and the non-preferred response comprises the rejection response information;

[0008] training the target model based on the preference dataset by a direct preference optimization method, to obtain a trained target model, so that the trained target model can output the rejection response information when an input target query exceeds the parameter knowledge boundary and the retrieval knowledge boundary.

[0009] Further, the determining the knowledge boundary division result corresponding to the query sample according to the first output, the second output, and the label information corresponding to the query sample comprises:

[0010] determining the knowledge boundary division result corresponding to the query sample according to whether the first output is consistent with the label information corresponding to the query sample and whether the second output is consistent with the label information.

[0011] Further, the determining the knowledge boundary division result corresponding to the query sample according to whether the first output is consistent with the label information corresponding to the query sample and whether the second output is consistent with the label information comprises:

[0012] allocating the query sample to one of four knowledge quadrants according to whether the first output is consistent with the label information corresponding to the query sample and whether the second output is consistent with the label information, wherein each knowledge quadrant corresponds to a different knowledge boundary division result.

[0013] wherein the constructing the preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge boundary division result comprises:

[0014] constructing the preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge quadrant to which the query sample belongs.

[0015] Further, the constructing the preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge quadrant to which the query sample belongs comprises:

[0016] determine answer indication information corresponding to the third output according to whether the third output of the target model for the query sample is consistent with the label information, wherein the answer indication information is used to indicate whether the third output is a correct answer corresponding to the query sample;

[0017] construct a rule based on the answer indication information and preference data corresponding to the knowledge quadrant to which the query sample belongs, and construct a preference data set according to the third output and the rejection response information.

[0018] Further, the method further comprises:

[0019] obtain performance data of the target model in each of the four knowledge quadrants with respect to one or more preset indicators;

[0020] If the performance data of the target model in at least one knowledge quadrant does not satisfy a preset condition, clean up the query sample assigned to the at least one knowledge quadrant.

[0021] Further, the loss function used in the training process of the target model includes a direct preference optimization loss, and further includes at least one of a supervision fine-tuning loss and a knowledge boundary classification loss.

[0022] Further, the method further comprises:

[0023] If the similarity between the second output and the label information is less than a first preset threshold, obtain a fourth output obtained by performing semantic matching retrieval on the query sample by using the external knowledge source:

[0024] If the similarity between the fourth output and the label information is greater than or equal to the first preset threshold, and a retrieval confidence corresponding to the fourth output is greater than or equal to a second preset threshold, it is determined that the second output is consistent with the label information, otherwise, it is determined that the second output is inconsistent with the label information.

[0025] Further, the method further comprises:

[0026] If the similarity between the first output and the label information is less than a third preset threshold, obtain a fifth output of the target model for at least one similar query satisfying a preset similarity condition with the query sample without relying on the external knowledge source;

[0027] If the similarity between the fifth output and the label information is greater than or equal to the third preset threshold, and a model confidence corresponding to the fifth output is greater than or equal to a fourth preset threshold, it is determined that the first output is consistent with the label information, otherwise, it is determined that the first output is inconsistent with the label information.

[0028] Further, the method further comprises:

[0029] if the similarity between the first output and the label information is greater than or equal to a fifth preset threshold, obtaining inference step information of the target model about the first output;

[0030] determining whether the first output is consistent with the label information corresponding to the query sample according to a logical rationality of the inference step information for the query sample.

[0031] Further, the method further comprises:

[0032] inputting a target query into the trained target model to obtain answer response information corresponding to the target query output by the trained target model, wherein if the target query exceeds the parameter knowledge boundary and the retrieval knowledge boundary, the answer response information includes a refusal response information

[0033] The embodiments of the present specification also provide a device for training a model, comprising:

[0034] an obtaining module, configured to obtain a first output of a target model for a query sample without relying on an external knowledge source, and a second output obtained by the external knowledge source by retrieving the query sample;

[0035] a knowledge boundary division module, configured to determine a knowledge boundary division result corresponding to the query sample according to the first output, the second output, and label information corresponding to the query sample, wherein the knowledge boundary division result is used to indicate whether the query sample is located within a parameter knowledge boundary of the target model, and whether the query sample is located within a retrieval knowledge boundary of the external knowledge source;

[0036] a data set construction module, configured to construct a preference data set according to a third output of the target model for the query sample, the label information, and the knowledge boundary division result, wherein the preference data set includes the query sample, a preferred response and a non-preferred response corresponding to the query sample, and one of the preferred response and the non-preferred response includes a refusal response information;

[0037] a model training module, configured to train the target model based on the preference data set by a direct preference optimization method to obtain a trained target model, so that the trained target model can output the refusal response information in a case where an input target query exceeds the parameter knowledge boundary and the retrieval knowledge boundary.

[0038] The embodiments of the present specification also provide a storage medium, which stores a computer program suitable for being loaded and executed by a processor to perform the steps of the above-mentioned method.

[0039] The embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is adapted to be loaded by the processor and execute the steps of the above method.

[0040] The embodiment of the present specification also provides a computer program product having at least one instruction stored thereon, which, when executed by a processor, implements the steps of the above method.

[0041] According to the scheme of the embodiment of the present specification, the knowledge boundary is divided to accurately identify whether the query is within the knowledge range of the model, a high-quality data set for direct preference optimization is constructed according to the knowledge boundary division result corresponding to the query sample, and the model is trained based on the direct preference optimization method by using the high-quality data set, so that the model can learn to generate accurate answers within the knowledge boundary, and the model can be endowed with the ability to refuse to answer "I don't know" when the query exceeds the knowledge boundary. The accuracy and refusal ability of the model can be effectively balanced, the robustness and credibility of the model in the high-risk field are enhanced, and the illusion problem existing in the existing retrieval enhancement fine-tuning method can be effectively solved. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flowchart of a method for training a model is provided for the embodiment of the present specification;

[0043] Figure 2 A structural diagram of a device for training a model is provided for the embodiment of the present specification;

[0044] Figure 3 A structural diagram of an electronic device is provided for the embodiment of the present specification. DETAILED DESCRIPTION

[0045] To make the purpose, technical scheme and advantages of the present specification clearer, the technical scheme of the present specification will be described clearly and completely in combination with the specific embodiments of the present specification and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present specification, not all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present specification.

[0046] Please refer to Figure 1 A flowchart of a method for training a model is provided for the embodiment of the present specification. In the embodiment of the present specification, the method for training a model is applied to the device for training a model (hereinafter referred to as "model training device") or the electronic device configured with the model training device described in the embodiment of the present specification. The following will be described with respect toFigure 1 The flowchart is illustrated in detail. The method for training the model can specifically include the following steps:

[0047] S102, obtaining a first output of the target model for a query sample without relying on an external knowledge source, and a second output obtained by searching the query sample by the external knowledge source.

[0048] In some embodiments, the target model refers to a large language model, which is a large parameter language model pre-trained based on a large-scale corpus dataset, and trained through instruction fine-tuning and artificial feedback reinforcement learning. The large language model has strong text generation capability. In some embodiments, the target model includes a RAG (Retrieval-Augmented Generation) system. The RAG system integrates an external knowledge source (such as a document library or a database, and the specific type of the external knowledge source is not specially limited in the example embodiment) to enhance the context information of the model, thereby generating more accurate and more contextualized text responses.

[0049] In some embodiments, the query sample refers to training data for training the target model, which contains query information. The query information refers to instructions or questions input by a user to the target model, such as “what medicine to take for a cold”. The target model will generate corresponding question and answer responses based on the query information. In some embodiments, the query information can only include text content, or the query information can include other forms of content such as image content, audio content, video content, icon content, etc. in addition to text content, and the example embodiment does not specially limit this.

[0050] In some embodiments, the query sample is input into the target model without relying on the external knowledge source. At this time, the target model will not use the external knowledge source to perform external retrieval on the query sample input into the target model, but will directly generate and output corresponding question and answer responses (i.e., the first output). In some embodiments, without inputting the target model, the external knowledge source is used to perform retrieval operations on the query sample to obtain corresponding retrieval results (i.e., the second output).

[0051] S104, determining a knowledge boundary division result corresponding to the query sample according to the first output, the second output, and label information corresponding to the query sample, wherein the knowledge boundary division result is used to indicate whether the query sample is located within a parameter knowledge boundary of the target model, and whether it is located within a retrieval knowledge boundary of the external knowledge source.

[0052] In some embodiments, the parameter knowledge boundary of the model refers to, in the case that the model does not rely on an external knowledge source, if a query is located within the knowledge boundary, the model can output a corresponding correct answer for the query, that is, the parameter knowledge boundary of the model is used to represent whether the query can be correctly answered by the parameter knowledge of the model, and the retrieval knowledge boundary of the external knowledge source refers to, if a query is located within the knowledge boundary, the external knowledge source can retrieve a corresponding correct answer for the query, that is, the retrieval knowledge boundary is used to represent whether the query can be correctly answered by the retrieval knowledge of the external knowledge source.

[0053] In some embodiments, the query sample is labeled training data, and the query sample has corresponding label information, which refers to the question and answer response label corresponding to the query information (instruction or question) contained in the query sample. In some embodiments, whether the query sample can be correctly answered by the parameter knowledge of the target model can be determined according to the first output and the label information corresponding to the query sample, whether the query sample can be correctly answered by the retrieval knowledge of the external knowledge source can be determined according to the second output and the label information corresponding to the query sample, and the knowledge boundary division result corresponding to the query sample can be determined according to whether the query sample can be correctly answered by the parameter knowledge of the target model and the retrieval knowledge of the external knowledge source, wherein the knowledge boundary division result includes four results, specifically, the query sample is located within the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, the query sample is located within the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source, the query sample is located outside the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, and the query sample is located outside the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source. In some embodiments, if the query sample can be correctly answered by the parameter knowledge of the target model and by the retrieval knowledge of the external knowledge source, the corresponding knowledge boundary division result is that the query sample is located within the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, if the query sample can be correctly answered by the parameter knowledge of the target model but cannot be correctly answered by the retrieval knowledge of the external knowledge source, the corresponding knowledge boundary division result is that the query sample is located within the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source, if the query sample cannot be correctly answered by the parameter knowledge of the target model but can be correctly answered by the retrieval knowledge of the external knowledge source, the corresponding knowledge boundary division result is that the query sample is located outside the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, and if the query sample cannot be correctly answered by the parameter knowledge of the target model and by the retrieval knowledge of the external knowledge source, the corresponding knowledge boundary division result is that the query sample is located outside the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source.

[0054] S106, constructing a preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge boundary division result, wherein the preference dataset comprises the query sample, a preferred response corresponding to the query sample, and a non-preferred response, one of the preferred response and the non-preferred response comprising a rejection response information.

[0055] In some embodiments, the query sample is input into the target model, and the target model uses an external knowledge source to perform external retrieval on the query sample input into the target model to obtain a corresponding retrieval result, and then generates and outputs a corresponding question and answer response (i.e., the third output) based on the query sample and the retrieval result.

[0056] In some embodiments, a customized preference dataset is constructed according to whether the third output is consistent with the label information corresponding to the query sample and a knowledge boundary division result corresponding to the query sample, and the preference dataset comprises the query sample, a preferred response corresponding to the query sample, and a non-preferred response. Whether the third output is consistent with the label information can be determined according to a similarity between the third output and the label information. For example, if the similarity between the third output and the label information is greater than or equal to a preset threshold, it can be determined that the third output is consistent with the label information, and if the similarity between the third output and the label information is less than the preset threshold, it can be determined that the third output is not consistent with the label information.

[0057] In some embodiments, if the query sample is located within the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, the rejection response information (the rejection response information is used to represent that the target model refuses to answer the query sample, for example, the rejection response information can be a "reject" text information, and the specific content of the rejection response information is not specially limited in the example embodiment) is taken as the non-preferred response, if the third output is consistent with the label information, the third output is taken as the preferred response, and if not, the preferred response is defaulted. If the query sample is located within the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source, the rejection response information is taken as the non-preferred response, if the third output is consistent with the label information, the third output is taken as the preferred response, and if not, the third output is also taken as the non-preferred response, and the preferred response is defaulted. If the query sample is located outside the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, the rejection response information is taken as the non-preferred response, if the third output is consistent with the label information, the third output is taken as the preferred response, and if not, the third output is also taken as the non-preferred response, and the preferred response is defaulted. If the query sample is located outside the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source, the rejection response information is taken as the preferred response, and regardless of whether the third output is consistent with the label information, the third output is taken as the non-preferred response.

[0058] In S108, the target model is trained based on the preference dataset by using a direct preference optimization method, and a trained target model is obtained, so that the trained target model can output the rejection response information when an input target query exceeds the parameter knowledge boundary and the retrieval knowledge boundary.

[0059] In some embodiments, direct preference optimization (DPO) is a language model training method that replaces the traditional reinforcement learning alignment. The method directly optimizes the model output by using human preference data, without explicitly constructing a reward model. The core idea is to convert preference learning into an optimization problem of probability distribution, and directly adjust the model parameters by maximizing the difference in log-likelihood between the preferred response and the non-preferred response in the preference data, so as to achieve preference alignment.

[0060] In some embodiments, the target model is trained by using the constructed preference dataset by using a direct preference optimization method, and a trained target model is obtained, so that the output preference of the target model is optimized, and the target model can better generate accurate answers within the knowledge boundary, and select rejection outside the knowledge boundary. In some embodiments, the loss function used in the training process of the target model includes a DPO loss (direct preference optimization loss), which is used to learn the difference between preference pairs, so that the model can distinguish between preferred and non-preferred outputs. In some embodiments, the target query to be answered is input into the trained target model. If the target query exceeds the parameter knowledge boundary of the target model and the retrieval knowledge boundary of the external knowledge source, the target model outputs the rejection response information. Otherwise (the target query is within the parameter knowledge boundary and within the retrieval knowledge boundary, or the target query is within the parameter knowledge boundary or within the retrieval knowledge boundary), the target model generates and outputs the corresponding correct answer.

[0061] According to the scheme of the embodiments of the present specification, the knowledge boundary is divided to accurately identify whether the query is within the knowledge range of the model, a high-quality dataset for direct preference optimization is constructed according to the knowledge boundary division result corresponding to the query sample, and the model is trained based on the direct preference optimization method by using the high-quality dataset, so that the model can learn to generate accurate answers within the knowledge boundary, and the model can be given the ability to answer "I don't know" when the query exceeds the knowledge boundary. The accuracy and rejection ability of the model can be effectively balanced, the robustness and credibility of the model in high-risk (such as medical, legal, etc.) fields can be enhanced, and the illusion problem existing in the existing retrieval enhancement fine-tuning method can be effectively solved.

[0062] In some embodiments, the determining the knowledge boundary division result corresponding to the query sample according to the first output, the second output, and label information corresponding to the query sample comprises: determining the knowledge boundary division result corresponding to the query sample according to whether the first output is consistent with the label information corresponding to the query sample and whether the second output is consistent with the label information. In some embodiments, the knowledge boundary division result corresponding to the query sample can be determined according to whether the first output is consistent with the label information corresponding to the query sample and whether the second output is consistent with the label information corresponding to the query sample, wherein if the first output is consistent with the label information and the second output is consistent with the label information, the corresponding knowledge boundary division result is that the query sample is located within the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, if the first output is consistent with the label information and the second output is inconsistent with the label information, the corresponding knowledge boundary division result is that the query sample is located within the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source, if the first output is inconsistent with the label information and the second output is consistent with the label information, the corresponding knowledge boundary division result is that the query sample is located outside the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, and if the first output is inconsistent with the label information and the second output is inconsistent with the label information, the corresponding knowledge boundary division result is that the query sample is located outside the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source. In some embodiments, whether the first output or the second output is consistent with the label information can be determined according to the similarity between the first output or the second output and the label information, for example, if the similarity between the first output or the second output and the label information is greater than or equal to a preset threshold, it can be determined that the first output or the second output is consistent with the label information, and if the similarity between the first output or the second output and the label information is less than the preset threshold, it can be determined that the first output or the second output is inconsistent with the label information.

[0063] In some embodiments, the determining the knowledge boundary division result corresponding to the query sample according to whether the first output is consistent with the label information corresponding to the query sample and whether the second output is consistent with the label information comprises: according to whether the first output is consistent with the label information corresponding to the query sample and whether the second output is consistent with the label information, assigning the query sample to one of four knowledge quadrants, wherein each knowledge quadrant corresponds to a different knowledge boundary division result; and wherein the constructing the preference data set according to the third output of the target model for the query sample, the label information, and the knowledge boundary division result comprises: constructing the preference data set according to the third output of the target model for the query sample, the label information, and the knowledge quadrant to which the query sample belongs. In some embodiments, the query samples are divided into four preset knowledge quadrants, each of which corresponds to a different knowledge boundary division result. For example, the query samples in the first knowledge quadrant are located within the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, the query samples in the second knowledge quadrant are located within the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source, the query samples in the third knowledge quadrant are located outside the parameter knowledge boundary of the target model and within the retrieval knowledge boundary of the external knowledge source, and the query samples in the fourth knowledge quadrant are located outside the parameter knowledge boundary of the target model and outside the retrieval knowledge boundary of the external knowledge source. It should be noted that the above correspondence between the knowledge quadrants and the knowledge boundary division results is only an example and is not limiting. Those skilled in the art should understand that any correspondence can be included in the protection scope of the present specification, and the present example embodiment does not specially limit this. In some embodiments, according to whether the first output is consistent with the label information corresponding to the query sample and whether the second output is consistent with the label information, the query sample is assigned to one of the four preset knowledge quadrants based on a preset assignment rule. If the first output is consistent with the label information, it indicates that the query sample can be correctly answered by the parameter knowledge of the target model. If the first output is inconsistent with the label information, it indicates that the query sample cannot be correctly answered by the parameter knowledge of the target model. If the second output is consistent with the label information, it indicates that the query sample can be correctly answered by the retrieval knowledge of the external knowledge source. If the second output is inconsistent with the label information, it indicates that the query sample cannot be correctly answered by the retrieval knowledge of the external knowledge source. That is, by analyzing whether the query sample can be correctly answered by the parameter knowledge of the target model or the retrieval knowledge of the external knowledge source, the query sample is assigned to the corresponding knowledge quadrant. By dividing the query samples into four knowledge quadrants and constructing a customized preference data set for each knowledge quadrant, a high-quality data set for direct preference optimization (DPO) training is formed, which can effectively improve the rejection capability of the target model outside the knowledge boundary.In some embodiments, based on a preset assignment rule, if the first output is consistent with the label information and the second output is consistent with the label information, the query sample is assigned to the first knowledge quadrant; if the first output is consistent with the label information and the second output is inconsistent with the label information, the query sample is assigned to the second knowledge quadrant; if the first output is inconsistent with the label information and the second output is consistent with the label information, the query sample is assigned to the third knowledge quadrant; if the first output is inconsistent with the label information and the second output is inconsistent with the label information, the query sample is assigned to the fourth knowledge quadrant. It should be noted that the above assignment rule is only an example and is not limited. Those skilled in the art should understand that any assignment rule can be included in the protection scope of the present specification, and the present example embodiment does not specially limit it. In some embodiments, the query sample is input into the target model, at this time the target model uses the external knowledge source to perform external retrieval on the query sample input into the target model to obtain the corresponding retrieval result, and then the target model generates and outputs the corresponding question and answer response (i.e., the third output) based on the query sample and the retrieval result. In some embodiments, according to whether the third output is consistent with the label information and the knowledge quadrant to which the query sample belongs, a customized preference data set is constructed, which includes the query sample, the preferred response and the non-preferred response corresponding to the query sample. If the query sample belongs to the first knowledge quadrant, the refusal response information is taken as the non-preferred response; if the third output is consistent with the label information, the third output is taken as the preferred response; if not, the preferred response is defaulted. If the query sample belongs to the second knowledge quadrant, the refusal response information is taken as the non-preferred response; if the third output is consistent with the label information, the third output is taken as the preferred response; if not, the third output is also taken as the non-preferred response, and the preferred response is defaulted. If the query sample belongs to the third knowledge quadrant, the refusal response information is taken as the non-preferred response; if the third output is consistent with the label information, the third output is taken as the preferred response; if not, the third output is also taken as the non-preferred response, and the preferred response is defaulted. If the query sample belongs to the fourth knowledge quadrant, the refusal response information is taken as the preferred response, and regardless of whether the third output is consistent with the label information, the third output is taken as the non-preferred response.

[0064] In some embodiments, the constructing the preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge quadrant to which the query sample belongs comprises: determining answer indication information corresponding to the third output according to whether the third output of the target model for the query sample is consistent with the label information, wherein the answer indication information is used to indicate whether the third output is a correct answer corresponding to the query sample; and constructing the preference dataset according to the third output and the rejection response information based on the answer indication information and a rule corresponding to the preference data of the knowledge quadrant to which the query sample belongs. In some embodiments, the answer indication information corresponding to the third output is determined according to whether the third output is consistent with the label information corresponding to the query sample, and the answer indication information is used to indicate whether the third output is a correct answer or an incorrect answer corresponding to the query sample. Then, the preference dataset is constructed according to the third output and the rejection response information based on the answer indication information and a rule corresponding to the preference data of the knowledge quadrant to which the query sample belongs. For example, the rule corresponding to the preference data of the first knowledge quadrant is "selecting a correct answer as a preferred response and taking'rejection' as a non-preferred response", the rule corresponding to the preference data of the second knowledge quadrant is "selecting a correct answer as a preferred response and taking an incorrect answer and'rejection' as non-preferred responses", the rule corresponding to the preference data of the third knowledge quadrant is "selecting a correct answer as a preferred response and taking an incorrect answer and'rejection' as non-preferred responses", and the rule corresponding to the preference data of the fourth knowledge quadrant is "selecting'rejection' as a preferred response and taking an incorrect answer and a correct answer as non-preferred responses". It should be noted that the above correspondence between the knowledge quadrant and the rule for constructing the preference data is only an example and is not limited. Those skilled in the art should understand that any correspondence can be included in the protection scope of the present specification, and the present example embodiment does not specially limit the correspondence. In some embodiments, if the query sample belongs to the first knowledge quadrant, the rejection response information is taken as a non-preferred response, the third output is taken as a preferred response if the third output is a correct answer, and the preferred response is defaulted if the third output is an incorrect answer. If the query sample belongs to the second knowledge quadrant, the rejection response information is taken as a non-preferred response, the third output is taken as a preferred response if the third output is a correct answer, and the third output is also taken as a non-preferred response if the third output is an incorrect answer, and the preferred response is defaulted. If the query sample belongs to the third knowledge quadrant, the rejection response information is taken as a non-preferred response, the third output is taken as a preferred response if the third output is a correct answer, and the third output is also taken as a non-preferred response if the third output is an incorrect answer, and the preferred response is defaulted. If the query sample belongs to the fourth knowledge quadrant, the rejection response information is taken as a preferred response, and the third output is taken as a non-preferred response regardless of whether the third output is a correct answer or an incorrect answer.

[0065] In some embodiments, the method further comprises: obtaining performance data of the target model in each of the four knowledge quadrants with respect to one or more preset indicators; and cleaning the query samples assigned to at least one knowledge quadrant if the performance data of the target model in the at least one knowledge quadrant does not meet a preset condition. In some embodiments, the performance data of the target model in each of the four preset knowledge quadrants with respect to one or more preset indicators includes, but is not limited to, accuracy, rejection rate, rejection accuracy, etc., and the present example embodiment does not make special limitations thereon. If the performance data of the target model in at least one knowledge quadrant does not meet a preset condition, the query samples assigned to the at least one knowledge quadrant are cleaned to ensure that the target model generates accurate answers within the knowledge boundary and correctly rejects outside the knowledge boundary, where the preset condition includes, but is not limited to, accuracy less than or equal to a preset threshold, rejection accuracy less than or equal to a preset threshold, etc., and the present example embodiment does not make special limitations thereon. In some embodiments, if the query samples corresponding to the target knowledge quadrant are cleaned, a new batch of query samples is obtained, and each query sample in the new batch of query samples is assigned to the knowledge quadrant to which it belongs until the number of query samples assigned to the target knowledge quadrant meets a preset condition (e.g., reaches a preset number threshold).

[0066] In some embodiments, the loss function used in the training process of the target model includes a direct preference optimization loss, and at least one of a supervised fine-tuning loss and a knowledge boundary classification loss. In some embodiments, in addition to including a direct preference optimization (DPO) loss, the loss function used in the training process of the target model also needs to combine a supervised fine-tuning (SFT) loss and a knowledge boundary classification loss to further comprehensively improve the performance of the target model through a multi-objective training method, so that the target model generates accurate answers within the knowledge boundary and correctly rejects outside the knowledge boundary. The SFT loss is used to improve the ability of the target model to generate preference output, and the knowledge boundary classification loss is used to enhance the perception ability of the target model to the knowledge boundary by predicting the knowledge boundary division result corresponding to the query.

[0067] In some embodiments, the method further comprises: if the similarity between the second output and the label information is less than a first preset threshold, obtaining a fourth output obtained by performing semantic matching retrieval on the query sample by the external knowledge source; if the similarity between the fourth output and the label information is greater than or equal to the first preset threshold, and a retrieval confidence corresponding to the fourth output is greater than or equal to a second preset threshold, determining that the second output is consistent with the label information, otherwise, determining that the second output is inconsistent with the label information. In some embodiments, if the similarity between the second output and the label information is less than the first preset threshold, it is not directly determined that the second output is inconsistent with the label information, but the semantic matching retrieval operation is performed on the query sample by using the external knowledge source without inputting the target model to obtain the corresponding retrieval result (i.e., the fourth output). Semantic Matching Retrieval is a retrieval technology based on semantic understanding, which aims to find the most relevant results in the semantic level from large-scale data, rather than relying on mechanical matching of keywords. It improves the accuracy and relevance of retrieval by deeply understanding the semantic information of the text (such as intent, context, entity relationship, etc.). In some embodiments, if the similarity between the fourth output and the label information is still less than the first preset threshold, it is determined that the second output is inconsistent with the label information, and if the similarity between the fourth output and the label information is greater than or equal to the first preset threshold, it is further determined whether the retrieval confidence corresponding to the fourth output is greater than or equal to the second preset threshold. If yes, it is determined that the second output is consistent with the label information, otherwise, it is determined that the second output is inconsistent with the label information. Retrieval Confidence is a quantitative evaluation index for measuring the relevance or reliability of the retrieval result and the query content, usually in the form of probability value, score or level.

[0068] In some embodiments, the method further comprises: if the similarity between the first output and the label information is less than a third preset threshold, obtaining a fifth output of the target model for at least one similar query that meets a preset similarity condition with the query sample without relying on an external knowledge source; if the similarity between the fifth output and the label information is greater than or equal to the third preset threshold, and a model confidence corresponding to the fifth output is greater than or equal to a fourth preset threshold, determining that the first output is consistent with the label information, otherwise, determining that the first output is inconsistent with the label information. In some embodiments, if the similarity between the first output and the label information is less than the third preset threshold, it is not directly determined that the first output is inconsistent with the label information, but at least one similar query that meets a preset similarity condition (for example, the corresponding similarity is greater than or equal to a preset threshold) with the query sample is obtained first, and then the at least one similar query is input into the target model without relying on an external knowledge source, to obtain at least one question and answer response (i.e., the fifth output) generated and output by the target model for the at least one similar query. If there is no question and answer response in the at least one question and answer response that has a similarity greater than or equal to the third preset threshold with the label information, it is determined that the first output is inconsistent with the label information. If there is a target question and answer response in the at least one question and answer response that has a similarity greater than or equal to the third preset threshold with the label information, it is necessary to further determine whether a model confidence corresponding to the target question and answer response is greater than or equal to the fourth preset threshold. If yes, it can be determined that the first output is consistent with the label information, otherwise, it is determined that the first output is inconsistent with the label information. The model confidence refers to the degree of certainty of the target model for its prediction result, which is usually presented in the form of a probability value, a score or a confidence interval.

[0069] In some embodiments, the method further comprises: if the similarity between the first output and the label information is greater than or equal to a fifth preset threshold, obtaining inference step information of the target model about the first output; and determining whether the first output is consistent with the label information corresponding to the query sample according to a logical reasonableness of the inference step information for the query sample. In some embodiments, if the similarity between the first output and the label information is greater than or equal to the fifth preset threshold, it is not directly determined that the first output is consistent with the label information, but the inference step information of the target model about the first output needs to be obtained first, the inference step information including a series of intermediate and step-by-step inference steps of the target model in the process of generating the first output based on the query sample, and then a logical reasonableness of the inference step information for the query sample is obtained, if the logical reasonableness is greater than or equal to a preset threshold, it is determined that the first output is consistent with the label information, and if the logical reasonableness is less than the preset threshold, it is determined that the first output is not consistent with the label information. It should be noted that the logical reasonableness can be obtained in any manner, for example, by a human (e.g., a training personnel) or by inputting the inference step information into a trained logical judgment model to obtain the logical reasonableness output by the logical judgment model. It should be noted that the above-mentioned manner of obtaining the logical reasonableness is only an example, and not a limitation. Any obtaining manner can be included in the protection scope of the present specification, and the present example embodiment does not specially limit the obtaining manner.

[0070] In some embodiments, the method further comprises: inputting a target query into the trained target model to obtain answer response information corresponding to the target query output by the trained target model, wherein if the target query exceeds the parameter knowledge boundary and the retrieval knowledge boundary, the answer response information includes a refusal response information. In some embodiments, a target query to be answered is input into a trained target model, if the target query exceeds the parameter knowledge boundary of the target model and the retrieval knowledge boundary of the external knowledge source, the target model outputs a refusal response information, otherwise (the target query is within the parameter knowledge boundary and within the retrieval knowledge boundary, or the target query is within the parameter knowledge boundary or within the retrieval knowledge boundary), the target model generates and outputs a corresponding correct answer.

[0071] Figure 2 A structural schematic diagram of a device for training a model is provided in the embodiments of the present specification. The device for training a model (hereinafter referred to as “model training device 1”) can be realized as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the model training device 1 includes an obtaining module 11, a knowledge boundary division module 12, a data set construction module 13, and a model training module 14.

[0072] The obtaining module 11 is configured to obtain a first output of a target model for a query sample without relying on an external knowledge source, and a second output obtained by searching the query sample by using the external knowledge source;

[0073] The knowledge boundary division module 12 is configured to determine a knowledge boundary division result corresponding to the query sample according to the first output, the second output, and label information corresponding to the query sample, wherein the knowledge boundary division result is used to indicate whether the query sample is located within a parameter knowledge boundary of the target model, and whether the query sample is located within a search knowledge boundary of the external knowledge source.

[0074] The dataset construction module 13 is configured to construct a preference dataset according to a third output of the target model for the query sample, the label information, and the knowledge boundary division result, wherein the preference dataset includes the query sample, a preferred response corresponding to the query sample, and a non-preferred response, one of the preferred response and the non-preferred response includes rejection response information.

[0075] The model training module 14 is configured to train the target model based on the preference dataset by using a direct preference optimization method, to obtain a trained target model, so that the trained target model can output the rejection response information when an input target query exceeds the parameter knowledge boundary and the search knowledge boundary.

[0076] In some embodiments, the determination of the knowledge boundary division result corresponding to the query sample according to the first output, the second output, and the label information corresponding to the query sample includes: determining the knowledge boundary division result corresponding to the query sample according to whether the first output is consistent with the label information corresponding to the query sample, and whether the second output is consistent with the label information.

[0077] In some embodiments, the determination of the knowledge boundary division result corresponding to the query sample according to whether the first output is consistent with the label information corresponding to the query sample, and whether the second output is consistent with the label information includes: assigning the query sample to one of four knowledge quadrants according to whether the first output is consistent with the label information corresponding to the query sample, and whether the second output is consistent with the label information, wherein each knowledge quadrant corresponds to a different knowledge boundary division result; and the construction of the preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge boundary division result includes: constructing the preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge quadrant to which the query sample belongs.

[0078] In some embodiments, the constructing the preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge quadrant to which the query sample belongs comprises: determining answer indication information corresponding to the third output according to whether the third output of the target model for the query sample is consistent with the label information, wherein the answer indication information is used to indicate whether the third output is the correct answer corresponding to the query sample; and constructing a preference dataset according to the third output and the rejection response information based on the answer indication information and the preference data corresponding to the knowledge quadrant to which the query sample belongs.

[0079] In some embodiments, the model training apparatus 1 is further configured to: obtain performance data of the target model in each of the four knowledge quadrants with respect to one or more preset indicators; and clean query samples assigned to at least one knowledge quadrant if the performance data of the target model in the at least one knowledge quadrant does not satisfy a preset condition.

[0080] In some embodiments, the loss function used in the training process of the target model comprises a direct preference optimization loss, and further comprises at least one of a supervision fine-tuning loss and a knowledge boundary classification loss.

[0081] In some embodiments, the model training apparatus 1 is further configured to: if the similarity between the second output and the label information is less than a first preset threshold, obtain a fourth output obtained by performing semantic matching retrieval on the query sample by using the external knowledge source; if the similarity between the fourth output and the label information is greater than or equal to the first preset threshold, and a retrieval confidence corresponding to the fourth output is greater than or equal to a second preset threshold, determine that the second output is consistent with the label information, otherwise, determine that the second output is inconsistent with the label information.

[0082] In some embodiments, the model training apparatus 1 is further configured to: if the similarity between the first output and the label information is less than a third preset threshold, obtain a fifth output of the target model for at least one similar query that satisfies a preset similarity condition with the query sample without relying on the external knowledge source; if the similarity between the fifth output and the label information is greater than or equal to the third preset threshold, and a model confidence corresponding to the fifth output is greater than or equal to a fourth preset threshold, determine that the first output is consistent with the label information, otherwise, determine that the first output is inconsistent with the label information.

[0083] In some embodiments, the model training apparatus 1 is further configured to: if the similarity between the first output and the label information is greater than or equal to a fifth preset threshold, obtain inference step information of the target model with respect to the first output; and determine whether the first output is consistent with the label information corresponding to the query sample according to a logical rationality of the query sample with respect to the inference step information.

[0084] In some embodiments, the model training apparatus 1 is further configured to: input a target query into the trained target model to obtain answer response information corresponding to the target query output by the trained target model, wherein if the target query is beyond the parameter knowledge boundary and the retrieval knowledge boundary, the answer response information comprises a refusal response information.

[0085] The device embodiments correspond to the method embodiments described above, and specific descriptions can be referred to the descriptions of the method embodiments, which will not be repeated here. The device embodiments are based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. Specific descriptions can be referred to the corresponding method embodiments.

[0086] The embodiments of the present specification also provide a computer storage medium, which can store a plurality of instructions. The instructions are suitable for being loaded by a processor and executed to perform the method of the embodiments of the present specification.

[0087] The embodiments of the present specification also provide a computer program product, which stores at least one instruction. The at least one instruction is loaded by the processor and executed to perform the method of the embodiments of the present specification.

[0088] The embodiments of the present specification also provide a computer program product, which stores at least one instruction. The at least one instruction is loaded by the processor and executed to perform the method of the embodiments of the present specification. Figure 3 The structure schematic diagram of the electronic device is shown. As shown in the figure, Figure 3 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course can also include other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs to implement the above method.

[0089] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer may, for example, be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0090] Those skilled in the art will appreciate that embodiments of the present description can be readily used as a method, a system or a computer program product. Accordingly, the present description can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0091] The present description is described in reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present description. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0092] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks. Figure 1 one or more functions specified in the flow diagram and / or block diagram block or blocks.

[0094] It is also to be noted that the terms "comprising", "including", and "having" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0095] The specification can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0096] The various embodiments in the specification are described in progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0097] The above only describes the embodiments of the specification and does not limit the specification. The specification can have various changes and variations for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the specification shall be included in the scope of claims of the specification.

Claims

1. A method for training a model, comprising: obtaining a first output of a target model for a query sample without relying on an external knowledge source, and a second output of the query sample retrieved by the external knowledge source; allocating the query sample to one of four knowledge quadrants according to whether the first output is consistent with label information corresponding to the query sample, and whether the second output is consistent with the label information, wherein each knowledge quadrant corresponds to a different knowledge boundary division result, and the knowledge boundary division result is used to indicate whether the query sample is located within a parameter knowledge boundary of the target model and within a retrieval knowledge boundary of the external knowledge source; constructing a preference dataset according to a third output of the target model for the query sample, the label information, and the knowledge quadrant to which the query sample belongs, wherein the preference dataset includes the query sample, a preferred response and a non-preferred response corresponding to the query sample, and one of the preferred response and the non-preferred response includes rejection response information; training the target model based on the preference dataset by a direct preference optimization method to obtain a trained target model, so that the trained target model can output the rejection response information when an input target query exceeds the parameter knowledge boundary and the retrieval knowledge boundary.

2. The method of claim 1, wherein the constructing of the preference dataset according to the third output of the target model for the query sample, the label information, and the knowledge quadrant to which the query sample belongs comprises: determining answer indication information corresponding to the third output according to whether the third output is consistent with the label information, wherein the answer indication information is used to indicate whether the third output is a correct answer corresponding to the query sample; constructing the preference dataset according to the third output and the rejection response information based on the answer indication information and a preference data construction rule corresponding to the knowledge quadrant to which the query sample belongs.

3. The method of claim 2, further comprising: obtaining performance data of the target model in each of the four knowledge quadrants with respect to one or more preset indicators; if the performance data of the target model in at least one knowledge quadrant does not satisfy a preset condition, cleaning up the query sample allocated to the at least one knowledge quadrant.

4. The method of claim 1, wherein a loss function used in the training process of the target model includes a direct preference optimization loss, and further includes at least one of a supervision fine-tuning loss and a knowledge boundary classification loss.

5. The method of claim 1, further comprising: if a similarity between the second output and the label information is less than a first preset threshold, obtaining a fourth output of the query sample obtained by performing semantic matching retrieval on the query sample by the external knowledge source. If the similarity between the fourth output and the label information is greater than or equal to the first preset threshold, and the retrieval confidence corresponding to the fourth output is greater than or equal to a second preset threshold, it is determined that the second output is consistent with the label information, otherwise, it is determined that the second output is inconsistent with the label information.

6. The method of claim 1 or 5, further comprising: if the similarity between the first output and the label information is less than a third preset threshold, obtaining a fifth output of the target model for at least one similar query that satisfies a preset similarity condition with the query sample without relying on an external knowledge source; if the similarity between the fifth output and the label information is greater than or equal to the third preset threshold, and a model confidence corresponding to the fifth output is greater than or equal to a fourth preset threshold, it is determined that the first output is consistent with the label information, otherwise, it is determined that the first output is inconsistent with the label information.

7. The method of claim 1, further comprising: if the similarity between the first output and the label information is greater than or equal to a fifth preset threshold, obtaining inference step information of the target model for the first output; determining whether the first output is consistent with the label information corresponding to the query sample according to a logical rationality of the query sample based on the inference step information.

8. The method of claim 1, further comprising: inputting a target query into the trained target model to obtain answer response information corresponding to the target query output by the trained target model, wherein if the target query exceeds the parameter knowledge boundary and the retrieval knowledge boundary, the answer response information includes a refusal response information.

9. An apparatus for training a model, comprising: an obtaining module configured to obtain a first output of a target model for a query sample without relying on an external knowledge source, and a second output obtained by the external knowledge source by retrieving the query sample; a knowledge boundary division module configured to divide the query sample into one of four knowledge quadrants according to whether the first output is consistent with label information corresponding to the query sample, and whether the second output is consistent with the label information, wherein each knowledge quadrant corresponds to a different knowledge boundary division result, and the knowledge boundary division result is used to indicate whether the query sample is within a parameter knowledge boundary of the target model, and whether the query sample is within a retrieval knowledge boundary of the external knowledge source; a dataset construction module configured to construct a preference dataset according to a third output of the target model for the query sample, the label information, and a knowledge quadrant to which the query sample belongs, wherein the preference dataset includes the query sample, a preferred response and a non-preferred response corresponding to the query sample, and one of the preferred response and the non-preferred response includes a refusal response information. A model training module configured to train the target model based on the preference dataset by a direct preference optimization method, to obtain a trained target model, so that the trained target model is capable of outputting the rejection response information when an input target query exceeds the parameter knowledge boundary and the retrieval knowledge boundary.

10. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by a processor, implements the steps of the method of any one of claims 1-8.

11. An electronic device, comprising: Comprising: A processor and a memory; wherein the memory stores a computer program, the computer program being adapted to be loaded and executed by the processor to implement the steps of the method of any one of claims 1-8.

12. A computer program product having stored thereon at least one instruction, the computer program product comprising: The at least one instruction, when executed by a processor, implements the steps of the method of any one of claims 1-8.

Images