Method, device and equipment for optimizing model training sample, medium and product

Abstract

The invention provides a method for optimizing a model training sample. The method comprises the following steps: acquiring a first negative sample; wherein the first negative sample comprises a first cue word and a first output, and the first output is the output of the target model when the first cue word is used as the input of the target model; negative factor identification is carried out on the first output, negative factor content in the first output is determined, and the negative factor content is partial content of the first output; generating a second negative sample according to the negative cause content; wherein the second negative sample comprises a second cue word and a second output, the second cue word comprises the first cue word and at least part of other contents except the negative cause content in the first output, the second output comprises the negative cause content, and the second negative sample is used for training the target model. In the method, the training sample is optimized, and when the optimized negative sample is applied to the post-training stage, the probability of generating the negative cause content can be pointedly and accurately reduced, and the target model is directionally optimized.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product for optimizing model training samples. Background Art

[0002] A generative model is an artificial intelligence model capable of generating new content and can be applied to multiple different fields such as text generation, image generation, and speech generation. The training process of a generative model generally includes a pre-training stage and a post-training stage. In the pre-training stage, the ability of the generative model to generate the next token is trained, and in the post-training stage, the inference performance of the generative model in a specific task is improved.

[0003] Specifically, the post-training stage usually includes two parts: supervised fine tuning (SFT) and reinforcement learning from human feedback (RLHF). In the related art, in RLHF of the post-training stage, the training samples are usually negative samples, and the generative model is fine-tuned using the negative samples so that the generative model better meets user requirements. Among them, the negative sample includes a prompt and an output, that is, when the prompt is input into the generative model, the output is the model output that does not meet the expectation.

[0004] However, in the above method, the output part in the negative sample will be suppressed as a whole, that is, when the prompt is used as the model input, the probability of generating all tokens in the output will decrease, and it is difficult to achieve targeted and precise model fine-tuning. Summary of the Invention

[0005] This application provides a method for optimizing model training samples. This method can optimize the training samples in the post-training stage so as to perform targeted optimization on the target model. This application also provides a corresponding apparatus, an electronic device, a computer-readable storage medium, and a computer program product for the above method.

[0006] In a first aspect, this application provides a method for optimizing model training samples, and the method includes:

[0007] Obtain a first negative sample; wherein, the first negative sample includes: a first prompt and a first output, and the first output is the output of the target model when the first prompt is used as the input of the target model;

[0008] Perform negative cause identification on the first output, and determine the negative cause content in the first output, where the negative cause content is part of the first output;

[0009] Generate a second negative sample according to the negative cause content; wherein, the second negative sample includes: a second prompt word and a second output, the second prompt word includes: at least part of the first prompt word and other content in the first output except the negative cause content, and the second output includes the negative cause content, and the second negative sample is used to train the target model.

[0010] In a second aspect, the present application provides an apparatus for optimizing model training samples, and the apparatus includes:

[0011] An acquisition module, configured to acquire a first negative sample; wherein, the first negative sample includes: a first prompt word and a first output, and the first output is the output of the target model when the first prompt word is used as the input of the target model;

[0012] An identification module, configured to perform negative cause identification on the first output, and determine the negative cause content in the first output, where the negative cause content is part of the first output;

[0013] An optimization module, configured to generate a second negative sample according to the negative cause content; wherein, the second negative sample includes: a second prompt word and a second output, the second prompt word includes: at least part of the first prompt word and other content in the first output except the negative cause content, and the second output includes the negative cause content, and the second negative sample is used to train the target model.

[0014] In a third aspect, the present application provides an electronic device, and the electronic device includes a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute instructions stored in the memory, so that the electronic device executes the method for optimizing model training samples in the first aspect or any implementation manner of the first aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, and instructions are stored in the computer-readable storage medium, and the instructions instruct the electronic device to execute the method for optimizing model training samples in the first aspect or any implementation manner of the first aspect.

[0016] In a fifth aspect, the present application provides a computer program product containing instructions, and when it runs on an electronic device, it causes the electronic device to execute the method for optimizing model training samples in the first aspect or any implementation manner of the first aspect.

[0017] Based on the implementation manners provided in the above aspects, the present application can be further combined to provide more implementation manners.

[0018] As can be seen from the above technical solutions, the present application has the following advantages:

[0019] The present application provides a method for optimizing model training samples. The method first obtains a first negative sample, where the first negative sample includes: a first prompt and a first output. The first output is the output of a target model when the first prompt is used as the input of the target model. Then, negative factor identification is performed on the first output to determine the negative factor content in the first output. The negative factor content is part of the first output. According to the negative factor content, a second negative sample is generated, where the second negative sample includes: a second prompt and a second output. The second prompt includes at least part of the first prompt and other content in the first output except the negative factor content. The second output includes the negative factor content. The second negative sample is used to train the target model.

[0020] In this method, for the training samples in the post-training stage of a target model (such as a generative model), by performing negative factor identification on the output part of the first negative sample, the part of the content that causes it to be a negative factor is accurately found, and the negative sample is reconstructed, so that the output in the reconstructed negative sample (i.e., the second negative sample) only includes the negative factor content. In this way, the optimization of the training samples is achieved. When the optimized negative sample is applied to the post-training stage, the probability of generating negative factor content can be targeted and accurately reduced, the target model can be optimized directionally, the model fine-tuning efficiency can be improved, and the inference performance of the fine-tuned target model is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below.

[0022] Figure 1 It is a schematic flowchart of a method for optimizing model training samples provided by an embodiment of the present application;

[0023] Figure 2 It is a schematic structural diagram of an apparatus for optimizing model training samples provided by an embodiment of the present application;

[0024] Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] The terms "first" and "second" in the embodiments of the present application are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0026] First, some technical terms and application scenarios involved in the embodiments of the present application are introduced.

[0027] A generative model is an artificial intelligence model that can generate new content based on existing data. Among them, the existing data can be different types of data, such as text data, audio data, video data, image data, etc. In other words, the generative model can be used to generate multimodal content.

[0028] Generally, a generative model generates new content based on "next token prediction", that is, the generative model predicts the next most likely token based on the existing data sequence. Specifically, in implementation, the existing data sequence is tokenized and decomposed into the smallest units (i.e., multiple tokens) that the generative model can understand. The generative model converts each token into an embedding vector so that the embedding vector can represent the content meaning of the token and the position of the token in the data sequence. Since the generative model is usually based on the transformer architecture, through the self-attention mechanism of the transformer architecture, when the generative model generates each token, it can fully consider the context information of all previous tokens (including the tokens in the existing data sequence and the tokens that have been generated), and then generate accurate and coherent content.

[0029] The training process of the generative model is usually divided into a pre-training stage and a post-training stage. In the pre-training stage, the generative model learns the ability to predict the next token by training on a data set. The pre-training stage is usually self-supervised learning, that is, the generative model uses the first N tokens in the data set to predict the generation probability of the (N + 1)th token, and judges whether the generation probability of the (N + 1)th token is the same as the actual (N + 1)th token in the data set, and then conducts self-training without externally labeled training data.

[0030] In the post-training stage, the inference performance of the generative model is improved in specific tasks. Specifically, the post-training stage generally includes two parts: supervised fine tuning (SFT) and reinforcement learning from human feedback (RLHF). In SFT, the pre-trained generative model is fine-tuned using labeled training data, enabling the generative model to learn the style of specific tasks, understand the task requirements of specific tasks, and improve the performance of the generative model on specific tasks. In RLHF, human subjective feedback is introduced to guide model fine-tuning, making the output of the generative model more in line with user needs and further optimizing the output quality of the generative model.

[0031] In related technologies, in RLHF of the post-training stage, the training samples are usually negative samples, that is, the generative model is fine-tuned using negative samples. Among them, the negative sample includes a prompt and an output, that is, when the prompt is input into the generative model, the output is the model output that does not meet the expectations. By using negative samples to optimize the generative model, when the prompt is used as the input, the generation probability of tokens in the output is reduced, thereby reducing the generation of content that does not meet the expectations and aligning the model output with user needs.

[0032] However, in the above method, the output part in the negative sample will be suppressed as a whole, that is, when the prompt is used as the model input, the probability of generating all tokens in the output will be reduced. In actual scenarios, it is often not the case that each token in the negative sample does not meet the expectations, and it is difficult to achieve targeted and precise model fine-tuning using the above method. Moreover, due to the overall suppression of negative samples, a large amount of training resources are consumed, resulting in low model fine-tuning efficiency.

[0033] In view of this, the present application provides a method for optimizing model training samples. The method first obtains a first negative sample, where the first negative sample includes: a first prompt and a first output. The first output is the output of the target model when the first prompt is used as the input of the target model. Then, negative cause identification is performed on the first output to determine the negative cause content in the first output. The negative cause content is part of the first output. According to the negative cause content, a second negative sample is generated, where the second negative sample includes: a second prompt and a second output. The second prompt includes at least part of the first prompt and other content in the first output except the negative cause content. The second output includes the negative cause content, and the second negative sample is used to train the target model.

[0034] In this method, for the training samples in the post-training stage of the target model (such as a generative model), by identifying the negative factors in the output part of the first negative sample, the part of the content that causes it to be a negative factor is accurately found, and the negative sample is reconstructed so that the output in the reconstructed negative sample (i.e., the second negative sample) only includes the content of the negative factor. In this way, the optimization of the training sample is realized. When the optimized negative sample is applied to the post-training stage, the probability of generating the content of the negative factor can be reduced specifically and accurately, the target model is optimized directionally, the model fine-tuning efficiency is improved, and the inference performance of the fine-tuned target model is better.

[0035] To facilitate the understanding of the technical solutions provided in the embodiments of the present application, the following will be described with reference to the accompanying drawings. See Figure 1 The flowchart of a model fine-tuning method shown, the method specifically includes:

[0036] S101: Obtain the first negative sample.

[0037] In the embodiments of the present application, the negative samples (including the first negative sample and the second negative sample) are training samples for model fine-tuning of the target model. Among them, the target model can be a generative model that generates new content based on the prediction of the next token, which can be understood as a pre-trained generative model with a model fine-tuning requirement. In other words, the target model can be a generative model that has gone through the pre-training stage, and in the post-training stage, the target model is fine-tuned.

[0038] It should be noted that the embodiments of the present application do not limit the data types that the target model can process. For example, the target model can be a generative model for processing multi-modal data such as text data, audio data, picture data, and video data. In other words, the optimized training samples in the embodiments of the present application can be applied to generative models of different modalities and have good universality.

[0039] In the post-training stage, the training samples usually consist of a prompt and an output. Among them, the training samples composed of the prompt and the output can also be called training question-and-answer pairs, and the prompt and the output can also be called instructions and answers. In SFT, the model is fine-tuned with the training samples composed of the prompt and the output above. When the prompt in the training sample is used as the model input, the generative model can increase the probability of generating the output in the training sample. In RLHF, the training samples can also include negative samples composed of the prompt and the output. When the prompt in the negative sample is used as the model input, the generative model can reduce the probability of generating the output in the negative sample.

[0040] In the embodiments of the present application, the first negative sample includes: a first prompt and a first output. The first output is the output of the target model when the first prompt is used as the input of the target model. And since the first prompt and the first output form the first negative sample, the first output can also be understood as the model output that does not meet the expectations and does not match the user's needs when the first prompt is used as the model input. That is to say, when the first prompt is input into the target model, it is not expected that the target model returns the first output.

[0041] The embodiments of the present application do not limit the acquisition method of the first negative sample. In some embodiments, multiple outputs corresponding to the first prompt are labeled, for example, manually labeled by annotators or automatically labeled by a classification model, and the output characterized as a negative sample in the labeling result is determined as the first output, forming the first negative sample composed of the first prompt and the first output. In other embodiments, a negative sample rule is preset in advance, and the negative sample rule can be used to describe the model output belonging to the negative sample. The multiple outputs corresponding to the first prompt are judged based on the negative sample rule, and the first output that hits the negative sample rule is screened out to form the first negative sample composed of the first prompt and the first output. In still other embodiments, a reward model is pre-trained in advance, and the reward model is used to judge the multiple outputs corresponding to the first prompt to determine the first output, forming the first negative sample composed of the first prompt and the first output.

[0042] S102: Identify the negative reasons for the first output to determine the negative reason content in the first output.

[0043] In the RLHF in the post-training stage, different from the traditional method of taking the overall output in the negative sample as the object to be suppressed, in the method provided by the embodiments of the present application, the negative reason for the first output in the first negative sample is identified to find out the negative reason content that causes the first output to be a negative sample, so as to optimize the first negative sample and achieve refined model fine-tuning.

[0044] Among them, the negative factor can be understood as the factor that causes the first output to be a negative sample (for example, it does not meet the expectations and does not match the user's needs). The content of the negative factor can be understood as the content in the first output that meets the negative factor. The content of the negative factor can be all the content of the first output (which can also be called the global negative factor), or it can be part of the content of the first output (which can also be called the local negative factor). It can be understood that when the content of the negative factor is all the content of the first output, it indicates that all the content of the first output is the content that causes the first output to be a negative sample, and the first output as a whole does not meet the expectations and does not match the user's needs. For example, the first output is too long or too short as a whole, or the first output does not include specific key information, etc. In the RLHF in the post-training stage, it is necessary to suppress all the content of the first output, reduce the probability of generating all the content of the first output when the first prompt is used as the model input, which is no different from the traditional model fine-tuning method. Therefore, the embodiments of the present application focus on the case where the content of the negative factor is part of the content of the first output, that is, after identifying the negative factor of the first output, the content of the negative factor in the first output is part of the content of the first output, such as some tokens.

[0045] In some possible implementation manners, a negative factor identification model is used to identify negative factors. Specifically, the first output is sent to the negative factor identification model, and the content of the negative factor in the first output returned by the negative factor identification model is received.

[0046] Among them, the negative factor identification model can be understood as a model with the ability to identify negative factors. For example, the negative factor identification model can be a model trained using text marked with the content of the negative factor. By sending the first output to the negative factor identification model, the negative factor identification model can find out the reason for the first output to be a negative sample and output the content of the negative factor in the first output, realizing negative factor identification.

[0047] In some other possible implementation manners, the negative factors of the first output can also be identified according to the set negative factor rules, and the content of the negative factors in the first output is determined.

[0048] Among them, the negative factor rules can be used to indicate the reasons for the output to be a negative sample. That is to say, by formulating negative factor rules in advance and combining the negative factor rules to judge which content in the first output does not meet the expectations and does not match the user's needs, the negative factor identification is completed.

[0049] In the actual application process, the negative factor rules can be flexibly defined based on the specific scenario and the specific model fine-tuning requirements. The negative factor identification processes in different scenarios and for different model requirements are described below.

[0050] In some embodiments, the negative factor rules include the first negative factor rule. The first prompt is described in the target language. According to the first negative factor rule, the negative factors of the first output are identified, and the content in the first output that is not in the target language is determined as the content of the negative factor in the first output.

[0051] That is, the user hopes that the target model will reply in the same language as the question (i.e., the first prompt), and the first negative factor rule can be understood as that the language of the first prompt does not match the language of the first output, and the reason for the first output being a negative sample is that there is content in the first output that is different from the language of the first prompt. In this case, the negative factor content in the first output is the content in the first output that does not belong to the target language. It is precisely because there is content in the first output that does not belong to the target language that the first output is a negative sample output.

[0052] In some other embodiments, the negative factor rule includes a second negative factor rule. According to the second negative factor rule, negative factor identification is performed on the first output, and the content in the first output that hits the keyword set is determined as the negative factor content in the first output.

[0053] Among them, the keyword set may include multiple pre-configured keywords, and the keyword can be understood as a word that is not expected to appear in the model output. That is, the user hopes that the model output generated by the target model does not contain the keywords in the keyword set. The second negative factor rule can be understood as that the first output hits the keyword set, and the reason for the first output being a negative sample is that there are keywords in the keyword set in the first output. In this case, the negative factor content in the first output is the content in the first output that hits the keyword set (i.e., the keywords existing in the first output). It is precisely because there are keywords in the keyword set in the first output that the first output is a negative sample output.

[0054] In some other embodiments, the negative factor rule includes a third negative factor rule. According to the third negative factor rule, negative factor identification is performed on the first output, and the garbled content in the first output whose length meets the length threshold is determined as the negative factor content in the first output.

[0055] Among them, the garbled content can be understood as content without punctuation marks, difficult to read, or without practical meaning (such as content composed of symbols such as "#", "￥", "%", etc.). That is, considering that the target model, especially the target model after SFT, may generate garbled content, the user hopes that the model output generated by the target model does not contain large segments of garbled content. The third negative factor rule can be understood as identifying that there is garbled content of a certain length in the first output, and the reason for the first output being a negative sample is that there is garbled content of a certain length in the first output. In this case, the negative factor content in the first output is the garbled content of a certain length in the first output. It is precisely because there is garbled content in the first output that the first output is a negative sample output.

[0056] In some other embodiments, the target model performs inference using the retrieval-augmented generation (RAG) technique. Among them, the RAG technique is divided into two stages: the knowledge retrieval stage and the response generation stage. Among them, the knowledge retrieval stage is used to retrieve knowledge information related to the question text from the knowledge base, and the response generation stage is used to generate a response text by combining the knowledge information related to the question text.

[0057] In the embodiments of the present application, the knowledge base may include multiple knowledge articles. By retrieving in the knowledge base, knowledge articles related to the question are recalled, and the knowledge articles recalled from the knowledge base are added to the first prompt. The target model performs inference based on the knowledge articles recalled from the knowledge base in the first prompt. For example, the most relevant target knowledge article that can be used to solve the problem is determined from the knowledge articles recalled from the knowledge base.

[0058] For the above scenario, the negative factor rule includes the fourth negative factor rule. The first output includes first knowledge information, and the first knowledge information includes at least one of the following: the title of the first knowledge article, the indication information for locating the first knowledge article, and the content of the first knowledge article. According to the fourth negative factor rule, negative factor identification is performed on the first output, and at least one of the following in the first output is determined as the negative factor content in the first output: the title of the first knowledge article that is empty or has an error, the indication information for locating the first knowledge article that is empty or has an error, and the content of the first knowledge article with a format error.

[0059] Among them, the indication information for locating the first knowledge article may be the link of the first knowledge article, a uniform resource locator (URL), etc. That is to say, it is hoped that the knowledge information citation situation in the model output generated by the target model is correct. The fourth negative factor rule can be understood as the situation where there is an incorrect knowledge information citation in the first output. The reason for the first output to be a negative sample is that there is an incorrect citation title in the first output (for example, it is empty or different from the title in the first prompt), incorrect indication information (for example, it is empty or different from the indication information in the first prompt), incorrect format, etc. In this case, the negative factor content in the first output is the incorrect knowledge information citation content in the first output. It is precisely because there is incorrect knowledge information citation content in the first output that the first output is a negative sample output.

[0060] Illustrated with a specific example, the first prompt may be as follows:

[0061] "As a Q&A expert, your job is to answer the provided questions based on the relevant parts of the given knowledge articles.

[0062] Question: How to repurchase a computer?

[0063] Knowledge Article 1:

[0064] URL: http: / / url.for.article.1.com

[0065] Title: How to Apply for the Expired Repurchase of a Laptop

[0066] Content: This article will introduce the expired repurchase policy of laptops, the description and application method of expired paid repurchase...

[0067] Knowledge Article 2:

[0068] URL: http: / / url.for.article.2.com

[0069] Title: Frequently Asked Questions about the Expired Repurchase of Computers

[0070] Content: This article will introduce the common questions about the expired repurchase of computers for your reference...

[0071] Among them, the first knowledge article is Knowledge Article 1 and Knowledge Article 2.

[0072] ① When the negative factor rule includes the first negative factor rule, if the first output is as follows:

[0073] "To repurchase a computer, the specific method is as follows:

[0074] First, make sure you are qualified to repurchase a computer;

[0075] Then..."

[0076] Since the first prompt word is described in Chinese, that is, the target language is Chinese, and the first output includes non-Chinese content, that is, the first output includes non-target language content. By identifying the negative factors in the first output, it is determined that the negative factor content in the first output is non-target language content.

[0077] ② When the negative factor rule includes the fourth negative factor rule, if the first output is as follows:

[0078] "To repurchase a computer, the specific method is as follows:

[0079] First, you need to make sure you are eligible to apply for the repurchase;

[0080] Secondly...

[0081] Finally...

[0082] For details, please refer to: [](http: / / url.for.article.1.com) and [Frequently Asked Questions about the Full-Term Repurchase of Computers](http: / / url.for.article.2.com)”

[0083] Or, if the first output is as follows:

[0084] “To repurchase a computer, the specific method is as follows:

[0085] First, you need to ensure that you are eligible to apply for the repurchase;

[0086] Second...

[0087] Finally...

[0088] For details, please refer to: [How to Apply for the Full-Term Repurchase of a Laptop](http: / / url.for.article.1.com) and [Common Questions about the Full-Term Repurchase of Computers](http: / / url.for.article.2.com)”

[0089] Since the title of the first knowledge article in the first output is empty ([]), or the title of the first knowledge article is incorrect ([Frequently Asked Questions about the Full-Term Repurchase of Computers]), by identifying the negative factors in the first output, it is determined that the negative factor content in the first output is the empty or incorrect title of the first knowledge article.

[0090] ③ When the negative factor rule includes the fourth negative factor rule, if the first output is as follows:

[0091] “To repurchase a computer, the specific method is as follows:

[0092] First, you need to ensure that you are eligible to apply for the repurchase;

[0093] Second...

[0094] Finally...

[0095] For details, please refer to: [How to Apply for the Full-Term Repurchase of a Laptop]() and [Frequently Asked Questions about the Full-Term Repurchase of Computers](http: / / url.for.article.2.com)”

[0096] Or, if the first output is as follows:

[0097] “To repurchase a computer, the specific method is as follows:

[0098] First, you need to ensure that you are eligible to apply for the repurchase;

[0099] Second...

[0100] Finally...

[0101] For details, please refer to: [How to Apply for the Expired Repurchase of Laptop](http: / / url.for.article.1.com) and [Frequently Asked Questions about the Expired Repurchase of Computer](http: / / other.url.com)”

[0102] Since there is empty or incorrect indication information (()) or (http: / / other.url.com) for locating the first knowledge article in the first output, by performing negative factor identification on the first output, it is determined that the negative factor content in the first output is the empty or incorrect indication information for locating the first knowledge article.

[0103] ④ When the negative factor rule includes the fourth negative factor rule, if the first output is as follows:

[0104] “To repurchase a computer, the specific methods are as follows:

[0105] First, you need to ensure that you are eligible to apply for the repurchase;

[0106] Second...

[0107] Finally...

[0108] For details, please refer to: http: / / url.for.article.1.com and http: / / url.for.article.2.com, How to Apply for the Expired Repurchase of Laptop, Frequently Asked Questions about the Expired Repurchase of Computer”

[0109] Since there is content of the first knowledge article with incorrect format (the content citing knowledge information is not presented in markdown format) in the first output, by performing negative factor identification on the first output, it is determined that the negative factor content in the first output is the content of the first knowledge article with incorrect format.

[0110] In this way, by performing negative factor identification on the first output, the negative factor content that causes the first output to be a negative sample (such as not meeting the user's expectations or not matching the user's needs) can be accurately found, so as to precisely suppress the negative factor content in subsequent RLHF, rather than suppressing the entire first output.

[0111] In some embodiments, the first output hits multiple negative factor rules, that is, there is negative factor content corresponding to multiple different negative factor rules in the first output. For the above situation, one of the multiple negative factor contents existing in the first output can be selected as the final negative factor content.

[0112] Specifically, according to the set negative factor rules, perform negative factor recognition on the first output, determine multiple candidate negative factor contents in the first output that hit the negative factor rules, respectively determine the starting positions of the multiple candidate negative factor contents in the first output, and determine the candidate negative factor content with the smallest starting position as the negative factor content in the first output.

[0113] Illustrating with an example, let the length of the first output be represented by output_len, and the negative factor content be represented by <pos, len>, where pos is the starting position of the negative factor content in the first output, and len is the length of the negative factor content. For each candidate negative factor content belonging to local negative factors, there are len < output_len and pos + len ≤ output_len. When selecting the final negative factor content from multiple candidate negative factor contents, the candidate negative factor content with the smallest starting position, that is, the smallest pos and closest to the first token in the first output, is determined as the final negative factor content of the first output.

[0114] Since the target model is a generative model that generates new content based on the prediction of the next token, by determining the candidate negative factor content with the smallest starting position as the final negative factor content of the first output, after precisely suppressing the negative factor content subsequently, the probability of the target model generating the negative factor content is reduced, and the generation probabilities of other candidate negative factor contents originally after the negative factor content in the first output also decrease accordingly. In this way, it is not necessary to suppress each candidate negative factor content in the first output. By suppressing the candidate negative factor content with the most forward position, the generation probabilities of other unexpected contents in the first output are reduced together, saving training resources and achieving efficient model fine-tuning.

[0115] S103: Generate a second negative sample according to the negative factor content.

[0116] After performing negative factor recognition on the first output and determining the negative factor content that causes the first output to be a negative sample, reconstruct a negative sample (i.e., the second negative sample). In the embodiments of the present application, the second negative sample includes: a second prompt word and a second output. The second prompt word includes at least part of the first prompt word and other content in the first output except the negative factor content. The second output includes the negative factor content.

[0117] That is to say, the original first prompt word and other content in the first output that do not belong to the negative factor content are constructed into a new second prompt word, and the negative factor content in the original first output is constructed into a new second output. In this way, in the subsequent RLHF in the training stage, it is possible to specifically suppress the negative factor content, and when the second prompt word is given, the probability of generating the negative factor content is precisely reduced.

[0118] In specific implementation, the starting position of the negative factor content in the first output is the Mth token, and the length of the negative factor content is N tokens, where M and N are positive integers. In this case, based on the first prompt and the content before the Mth token in the first output, a second prompt is generated, and based on the content from the Mth token to the M+N-1th token in the first output, a second output is generated.

[0119] For example, M is 3 and N is 2, that is, the starting position of the negative factor content in the first output is the 3rd token, and the length of the negative factor content is 2 tokens. By reconstructing the negative sample, the second prompt includes the first prompt and the first 2 tokens in the first output, and the second output includes the 3rd token to the 4th token in the first output.

[0120] Since the starting position of the negative factor content in the first output is the Mth token, it means that the first M-1 tokens in the first output are in line with expectations and are the correct model outputs. Therefore, when reconstructing the negative sample, the first M-1 tokens in the first output are added to the second prompt, indicating that there is no need to suppress the first M-1 tokens in the first output. The content from the Mth token to the M+N-1th token in the first output is the negative factor content itself. Using the content from the Mth token to the M+N-1th token in the first output to construct the second output indicates that only the part of the first output that belongs to the negative factor content needs to be suppressed. Compared with the first negative sample, the length of the second output in the second negative sample is less than the length of the first output in the first negative sample, realizing the optimization of the negative sample and enhancing the accuracy of model fine-tuning.

[0121] Illustrated with a specific example, the first prompt can be as follows:

[0122] "As a Q&A expert, your job is to answer the provided questions based on the relevant parts of the given knowledge article.

[0123] Question: How to repurchase a computer?

[0124] Knowledge Article 1:

[0125] URL: http: / / url.for.article.1.com

[0126] Title: How to Apply for the Expired Repurchase of a Laptop

[0127] Content: This article will introduce the expired repurchase policy of laptops, the description and application method of expired paid repurchase...

[0128] Knowledge Article 2:

[0129] URL: http: / / url.for.article.2.com

[0130] Title: Frequently Asked Questions about the Expired Repurchase of Computers

[0131] Content: This article will introduce the common questions about the expired repurchase of computers for your reference...

[0132] ① If the first output is as follows:

[0133] "To repurchase a computer, the specific method is as follows:

[0134] First, make sure you are qualified to repurchase a computer;

[0135] Then..."

[0136] The negative content in the first output is the content in a non-target language. By reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows:", and the second output in the second negative sample is "First, make sure you are qualified to repurchase a computer;".

[0137] ② If the first output is as follows:

[0138] "To repurchase a computer, the specific method is as follows:

[0139] First, the first keyword,

[0140] Then..."

[0141] The negative content in the first output is the content that hits the keyword set (i.e., the first keyword). By reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows: First,", and the second output in the second negative sample is "the first keyword".

[0142] ③ If the first output is as follows:

[0143] "To repurchase a computer, the specific method is as follows:

[0144] First, please refer to: [](http: / / url.for.article.1.com)

[0145] Then, please refer to: [Frequently Asked Questions about the Expired Repurchase of Computers](http: / / url.for.article.2.com)"

[0146] The negative factor content in the first output is the title of the first knowledge article that is empty. By reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows: First, please refer to: [", and the second output in the second negative sample is "]".

[0147] It should be noted that in the above situation, the negative factor content is the title of the first knowledge article that is empty. However, since the left half part "[" of the symbol "[]" used to represent the title of the first knowledge article is the correct model output, therefore, "[" is added to the second prompt word, and the second output only includes the right half part "]" of the symbol "[]" used to represent the title of the first knowledge article.

[0148] ④ If the first output is as follows:

[0149] "To repurchase a computer, the specific method is as follows:

[0150] First, please refer to: [How to Apply for the Expired Repurchase of a Laptop](http: / / url.for.article.1.com)

[0151] Then, please refer to: [Frequently Asked Questions about the Expired Repurchase of Computers]()""

[0152] The negative factor content in the first output is the indication information used to locate the first knowledge article that is empty. By reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows: First, please refer to: [How to Apply for the Expired Repurchase of a Laptop](http: / / url.for.article.1.com) Then, please refer to: [Frequently Asked Questions about the Expired Repurchase of Computers](", and the second output in the second negative sample is ")".

[0153] Similarly, in the above situation, the negative factor content is the indication information used to locate the first knowledge article that is empty. However, since the left half part "(" of the symbol "()" used to represent the indication information used to locate the first knowledge article is the correct model output, therefore, "(" is added to the second prompt word, and the second output only includes the right half part ")" of the symbol "()" used to represent the indication information used to locate the first knowledge article.

[0154] ⑤ If the first output is as follows:

[0155] "To repurchase a computer, the specific method is as follows:

[0156] First..., please refer to: [How to Repurchase a Computer](http: / / url.for.article.3.com);

[0157] Then..., please refer to: [Frequently Asked Questions about Computer Expiration Repurchase](http: / / url.for.article.2.com)”

[0158] The negative factor content in the first output is the indication information for locating the first knowledge article that has an error. In some possible implementation manners, by reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and “To repurchase a computer, the specific method is as follows: First..., please refer to: [How to Repurchase a Computer](”, and the second output in the second negative sample is “http: / / url.for.article.3.com”.

[0159] In some other possible implementation manners, by reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and “To repurchase a computer, the specific method is as follows: First..., please refer to: [”, and the second output in the second negative sample is “How to Repurchase a Computer](http: / / url.for.article.3.com”.

[0160] Furthermore, the second negative sample can be used to train the target model. Specifically, after reconstructing the negative sample, use the second negative sample to perform RLHF on the target model to complete the post-training stage.

[0161] In specific implementation, obtain positive samples, use the positive samples and the second negative sample as training samples, and fine-tune the target model. Among them, the positive samples include: the third prompt word and the third output. For example, the third output can be the model output that meets the expectations and the user's needs when the third prompt word is used as the model input.

[0162] In this way, use the positive samples and the second negative sample together as the training samples for RLHF in the post-training stage. For a given prompt word, the target model after RLHF can generate a model output close to the third output in the positive samples, and avoid generating a model output similar to the second output in the second negative sample, aligning with the user's needs.

[0163] In some possible implementation manners, RLHF in the post-training stage can adopt non-pairwise training, that is, the third prompt word in the positive sample is different from the second prompt word in the second negative sample. In this case, if the target model has completed SFT in the post-training stage, the training samples in SFT can be used as positive samples, and combined with the second negative sample to form the training samples in RLHF, and use a training method that supports both positive and negative samples, such as the kahneman-tversky optimization (KTO) method, to perform RLHF.

[0164] In some other possible implementation manners, the post-training stage may adopt paired sample training, that is, the third prompt word in the positive sample is the same as the second prompt word in the second negative sample. In this case, different ways may be adopted to obtain positive samples. In some embodiments, the second prompt word is determined as the third prompt word in the positive sample, and according to the negative factor rule, the second output is corrected to obtain the third output in the positive sample.

[0165] Illustratively, the negative factor rule includes the first negative factor rule, and the first negative factor rule can be understood as that the first prompt word does not match the language of the first output. In this case, the second output described in a non-target language is corrected to the content described in the target language to obtain the third output.

[0166] That is to say, by correcting the second output in the second negative sample, the second output is corrected to the model output that meets the expectations, and then the second prompt word and the corrected second output in the second negative sample are used to form a positive sample.

[0167] In some other embodiments, the second prompt word is determined as the third prompt word in the positive sample, the second prompt word is input into the correction model multiple times, multiple candidate outputs returned by the correction model are received, and the candidate output that meets the requirements of the positive sample among the multiple candidate outputs is determined as the third output in the positive sample.

[0168] Among them, the correction model may be a model capable of correcting training samples to obtain positive samples that meet the expectations and the user requirements. In some possible implementation manners, the correction model may be a generative model different from the target model. For example, the correction model may be a self-developed generative model that has completed the pre-training stage and the post-training stage and can be actually used.

[0169] By using the second prompt word in the second negative sample as the model input of the correction model, the correction model generates a series of candidate outputs, and the candidate output that does not hit the negative factor rule and meets the expectations is selected from the candidate outputs and determined as the third output to form a positive sample.

[0170] In paired sample training, methods such as direct preference optimization (DPO) can be used for RLHF.

[0171] In this method, for the training samples in the post-training stage of the target model (such as a generative model), by identifying the negative factors in the output part of the first negative sample, the part of the content that causes it to be a negative factor is accurately found, and the negative sample is reconstructed so that the output in the reconstructed negative sample (i.e., the second negative sample) only includes the negative factor content. In this way, the optimization of the training sample is achieved. When the optimized negative sample is applied to the post-training stage, it can specifically and accurately reduce the probability of generating negative factor content, perform targeted optimization on the target model, improve the model fine-tuning efficiency, and make the inference performance of the fine-tuned target model better.

[0172] As described above in connection with Figure 1 the method for optimizing model training samples provided in the embodiments of the present application has been introduced in detail. Next, the devices and equipment provided in the embodiments of the present application will be introduced with reference to the accompanying drawings.

[0173] See Figure 2 the structural schematic diagram of the device for optimizing model training samples shown in the figure. The device 20 includes:

[0174] An acquisition module 201, configured to acquire a first negative sample; wherein, the first negative sample includes: a first prompt and a first output, and the first output is the output of the target model when the first prompt is used as the input of the target model;

[0175] An identification module 202, configured to identify negative factors in the first output and determine the negative factor content in the first output, where the negative factor content is part of the first output;

[0176] An optimization module 203, configured to generate a second negative sample according to the negative factor content; wherein, the second negative sample includes: a second prompt and a second output, the second prompt includes at least part of the first prompt and other content in the first output except the negative factor content, the second output includes the negative factor content, and the second negative sample is used to train the target model.

[0177] In some possible implementation manners, the identification module 202 is specifically configured to:

[0178] Send the first output to a negative factor identification model and receive the negative factor content in the first output returned by the negative factor identification model; or,

[0179] Identify negative factors in the first output according to a set negative factor rule and determine the negative factor content in the first output; wherein, the negative factor rule is used to indicate the reason for the output to be a negative sample.

[0180] In some possible implementation manners, the identification module 202 is specifically configured to:

[0181] According to the set negative factor rules, perform negative factor recognition on the first output to determine multiple candidate negative factor contents in the first output that hit the negative factor rules;

[0182] Respectively determine the starting positions of the multiple candidate negative factor contents in the first output;

[0183] Determine the candidate negative factor content with the smallest starting position as the negative factor content in the first output.

[0184] In some possible implementation manners, the starting position of the negative factor content in the first output is the Mth token, and the length of the negative factor content is N tokens, where M and N are integers greater than 0; the optimization module 203 is specifically configured to:

[0185] Generate a second prompt word according to the first prompt word and the content in the first output before the Mth token; and,

[0186] Generate a second output according to the content from the Mth token to the M+N-1th token in the first output.

[0187] In some possible implementation manners, the negative factor rules include a first negative factor rule, and the first prompt word is described in the target language. The recognition module 202 is specifically configured to:

[0188] Perform negative factor recognition on the first output according to the first negative factor rule, and determine the content in the first output that is not in the target language as the negative factor content in the first output.

[0189] In some possible implementation manners, the negative factor rules include a second negative factor rule. The recognition module 202 is specifically configured to:

[0190] Perform negative factor recognition on the first output according to the second negative factor rule, and determine the content in the first output that hits the keyword set as the negative factor content in the first output.

[0191] In some possible implementation manners, the negative factor rules include a third negative factor rule. The recognition module 202 is specifically configured to:

[0192] Perform negative factor recognition on the first output according to the third negative factor rule, and determine the garbled content in the first output whose length meets the length threshold as the negative factor content in the first output.

[0193] In some possible implementation manners, the negative factor rule includes a fourth negative factor rule, the first output includes first knowledge information, and the first knowledge information includes at least one of the following: the title of the first knowledge article, the indication information for locating the first knowledge article, and the content of the first knowledge article; the recognition module 202 is specifically configured to:

[0194] According to the fourth negative factor rule, perform negative factor recognition on the first output, and determine at least one of the following in the first output as the negative factor content in the first output: the title of the first knowledge article that is empty or has an error, the indication information for locating the first knowledge article that is empty or has an error, and the content of the first knowledge article with a format error.

[0195] In some possible implementation manners, the apparatus 20 further includes a training module, and the training module is configured to:

[0196] Obtain positive samples; wherein, the positive samples include: a third prompt word and a third output;

[0197] Use the positive samples and the second negative samples as training samples to fine-tune the target model.

[0198] In some possible implementation manners, the training module is specifically configured to:

[0199] Determine the second prompt word as the third prompt word in the positive samples;

[0200] According to the negative factor rule, correct the second output to obtain the third output in the positive samples; or, input the second prompt word into the correction model multiple times, receive multiple candidate outputs returned by the correction model, and determine the candidate output that meets the requirements of the positive samples among the multiple candidate outputs as the third output in the positive samples.

[0201] The apparatus 20 for optimizing the model training samples according to the embodiments of the present application may correspond to execute the methods described in the embodiments of the present application, and the above and other operations and / or functions of each module / unit of the apparatus 20 for optimizing the model training samples respectively implement Figure 1 the corresponding processes of the respective methods in the illustrated embodiments, and for the sake of brevity, will not be described in detail herein.

[0202] Embodiments of the present application further provide an electronic device. This electronic device is specifically configured to implement the functions of the apparatus 20 for optimizing the model training samples as shown in Figure 2 the illustrated embodiments.

[0203] Figure 3 A schematic structural diagram of an electronic device 300 is provided, as shown in Figure 3As shown, the electronic device 300 includes a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0204] The bus 301 can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0205] The processor 302 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0206] The communication interface 303 is used for external communication. For example, the communication interface 303 can be used to communicate with a terminal.

[0207] The memory 304 can include volatile memory, such as random access memory (RAM). The memory 304 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0208] The memory 304 stores executable code, and the processor 302 executes the executable code to perform the method for optimizing the model training samples described above.

[0209] Specifically, in the case of implementing Figure 2 the embodiments shown, and Figure 2 when each module or unit of the device 20 for optimizing the model training samples described in the embodiments is implemented by software, execute Figure 2The software or program code required for the functions of each module / unit in [the context] can be stored partially or entirely in the memory 304. The processor 302 executes the program code corresponding to each unit stored in the memory 304 to execute the aforesaid method for optimizing the model training samples.

[0210] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive), etc. The computer-readable storage medium includes instructions that direct the computing device to execute the method for optimizing the model training samples of the device 20 applied above.

[0211] The embodiments of the present application also provide a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, they generate, in whole or in part, the processes or functions according to the embodiments of the present application.

[0212] The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, or data center to another website, computer, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.).

[0213] When the computer program product is executed by a computer, the computer executes any of the aforesaid methods for optimizing the model training samples. The computer program product can be a software installation package. In the case where any of the methods for optimizing the model training samples is needed, the computer program product can be downloaded and executed on the computer.

[0214] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0215] The units involved in the embodiments described in the present application can be implemented in software or in hardware. Among them, the name of the unit / module does not constitute a limitation to the unit itself in some cases.

[0216] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0217] In the context of the embodiments of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatuses, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM or Flash Memory), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0218] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple. For the relevant parts, reference can be made to the descriptions in the method section.

[0219] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expressions refer to any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0220] It should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0221] The steps of the methods or algorithms described in connection with the embodiments disclosed herein can be implemented directly in hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in a random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0222] The foregoing description of the disclosed embodiments enables those skilled in the art to practice or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for optimizing model training samples, characterized in that: The method comprises: Acquire a first negative sample; wherein the first negative sample includes: a first prompt word and a first output, the first output being the output of the target model when the first prompt word is used as an input of the target model; Performing negative cause identification on the first output to determine negative cause content in the first output, where the negative cause content is part of the first output; A second negative sample is generated according to the negative content; wherein the second negative sample includes: a second prompt word and a second output, the second prompt word includes: the first prompt word and at least part of the other content in the first output except the negative content, the second output includes the negative content, and the second negative sample is used to train the target model.

2. The method according to claim 1, characterized in that The step of identifying the negative cause of the first output and determining the negative cause content in the first output includes: Sending the first output to a negative cause identification model, and receiving negative cause content in the first output returned by the negative cause identification model; or, According to the set negative cause rule, negative cause identification is performed on the first output to determine the negative cause content in the first output; wherein the negative cause rule is used to indicate the reason why the output is a negative sample.

3. The method according to claim 2, characterized in that The step of identifying the negative cause of the first output according to the set negative cause rule and determining the negative cause content in the first output includes: According to the set negative cause rule, negative cause identification is performed on the first output to determine a plurality of candidate negative cause contents in the first output that match the negative cause rule; Determine the starting positions of the plurality of candidate negative cause contents in the first output respectively; The candidate negative cause content with the smallest starting position is determined as the negative cause content in the first output.

4. The method according to claim 1, characterized in that: The starting position of the negative cause content in the first output is the Mth token, the length of the negative cause content is N tokens, and M and N are integers greater than 0; The step of generating a second negative sample according to the negative content includes: Generate a second prompt word according to the first prompt word and the content before the Mth token in the first output; as well as, Generate a second output according to the contents of the Mth token to the M+N-1th token in the first output.

5. The method according to claim 2, characterized in that: The negative cause rule includes a first negative cause rule, the first prompt word is described in a target language, and the negative cause recognition is performed on the first output according to the set negative cause rule to determine the negative cause content in the first output, including: According to the first negative cause rule, negative cause identification is performed on the first output, and content in the first output that is not in the target language is determined as negative cause content in the first output.

6. The method according to claim 2, characterized in that The negative cause rule includes a second negative cause rule, and the negative cause identification is performed on the first output according to the set negative cause rule to determine the negative cause content in the first output, including: According to the second negative cause rule, negative cause identification is performed on the first output, and the content that hits the keyword set in the first output is determined as the negative cause content in the first output.

7. The method according to claim 2, characterized in that: The negative cause rule includes a third negative cause rule, and the negative cause identification is performed on the first output according to the set negative cause rule to determine the negative cause content in the first output, including: According to the third negative cause rule, negative cause identification is performed on the first output, and garbled content in the first output whose length meets the length threshold is determined as negative cause content in the first output.

8. The method according to claim 2, characterized in that: The negative cause rule includes a fourth negative cause rule, the first output includes first knowledge information, and the first knowledge information includes at least one of the following: a title of a first knowledge article, indication information for locating the first knowledge article, and content of the first knowledge article; The step of identifying the negative cause of the first output according to the set negative cause rule and determining the negative cause content in the first output includes: According to the fourth negative cause rule, negative cause identification is performed on the first output, and at least one of the following items in the first output is determined as the negative cause content in the first output: the title of the first knowledge article that is empty or erroneous, the indication information for locating the first knowledge article that is empty or erroneous, and the content of the first knowledge article with a format error.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Acquire a positive sample; wherein the positive sample includes: a third prompt word and a third output; The positive sample and the second negative sample are used as training samples to fine-tune the target model.

10. The method according to claim 9, characterized in that The obtaining of positive samples comprises: Determine the second prompt word as the third prompt word in the positive sample; According to the negative cause rule, the second output is corrected to obtain the third output in the positive sample; or, the second prompt word is input into the correction model multiple times, multiple candidate outputs returned by the correction model are received, and the candidate output that meets the positive sample requirements among the multiple candidate outputs is determined as the third output in the positive sample.

11. A device for optimizing model training samples, characterized in that: The device comprises: An acquisition module, configured to acquire a first negative sample; wherein the first negative sample comprises: a first prompt word and a first output, wherein the first output is the output of the target model when the first prompt word is used as an input of the target model; an identification module, configured to perform negative cause identification on the first output, and determine negative cause content in the first output, wherein the negative cause content is part of the content of the first output; An optimization module is used to generate a second negative sample based on the negative content; wherein the second negative sample includes: a second prompt word and a second output, the second prompt word includes: the first prompt word and at least part of the other content in the first output except the negative content, the second output includes the negative content, and the second negative sample is used to train the target model.

12. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct an electronic device to execute the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product comprises computer readable instructions for implementing the method according to any one of claims 1 to 10.

Images