A method, apparatus, device, medium, and product for optimizing model training samples.

Abstract

This application provides a method for optimizing model training samples. The method includes: obtaining a first negative sample; wherein the first negative sample includes a first prompt word and a first output, the first output being the output of the target model when the first prompt word is used as input; identifying negative causes in the first output to determine the negative cause content, the negative cause content being a portion of the first output; generating a second negative sample based on the negative cause content; wherein the second negative sample includes a second prompt word and a second output, the second prompt word including at least a portion of the first prompt word and other content in the first output besides the negative cause content, the second output including the negative cause content, and the second negative sample being used to train the target model. In this method, optimizing the training samples allows for targeted and precise reduction of the probability of generating negative cause content when the optimized negative samples are applied to the post-training stage, thus enabling directional optimization of the target model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for optimizing model training samples. Background Technology

[0002] Generative models are artificial intelligence models capable of generating new content, applicable to various fields such as text generation, image generation, and speech generation. The training process for generative models typically consists of a pre-training phase and a post-training phase. In the pre-training phase, the generative model's ability to generate the next token is trained; in the post-training phase, the inference performance of the generative model in specific tasks is improved.

[0003] Specifically, the post-training phase typically includes two parts: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). In related technologies, during the RLHF phase, training samples are usually negative samples. These negative samples are used to fine-tune the generative model, allowing it to better meet user needs. The negative samples include prompts and outputs; that is, when the prompt is input into the generative model, the output is an unexpected model output.

[0004] However, in the above method, the output portion of the negative samples is suppressed as a whole. That is, when the prompt word is used as the model input, the probability of generating all tokens in the output will be reduced, making it difficult to achieve targeted and precise model fine-tuning. Summary of the Invention

[0005] This application provides a method for optimizing model training samples. This method can optimize training samples in the post-training stage to enable targeted optimization of the target model. This application also provides apparatus, electronic devices, computer-readable storage media, and computer program products corresponding to the above method.

[0006] Firstly, this application provides a method for optimizing model training samples, the method comprising:

[0007] Obtain a first negative sample; wherein the first negative sample includes: a first prompt word and a first output, the first output being the output of the target model when the first prompt word is used as input to the target model;

[0008] The first output is subjected to negative cause identification to determine the negative cause content in the first output, wherein the negative cause content is a part of the content of the first output.

[0009] Based on the negative cause content, a second negative sample is generated; wherein, the second negative sample includes: a second prompt word and a second output, the second prompt word includes: at least part of the content other than the negative cause content in the first prompt word and the first output, the second output includes the negative cause content, and the second negative sample is used to train the target model.

[0010] Secondly, this application provides an apparatus for optimizing model training samples, the apparatus comprising:

[0011] An acquisition module is used to acquire a first negative sample; wherein, the first negative sample includes: a first prompt word and a first output, and the first output is the output of the target model when the first prompt word is used as input to the target model;

[0012] The identification module is used to identify negative causes in the first output and determine the negative cause content in the first output, wherein the negative cause content is a part of the content of the first output.

[0013] An optimization module is used to generate a second negative sample based on the negative cause content; wherein the second negative sample includes: a second prompt word and a second output, the second prompt word includes: at least part of the content other than the negative cause content in the first prompt word and the first output, the second output includes the negative cause content, and the second negative sample is used to train the target model.

[0014] Thirdly, this application provides an electronic device including a processor and a memory. The processor and the memory communicate with each other. The processor is configured to execute instructions stored in the memory to cause the electronic device to perform a method for optimizing model training samples as described in the first aspect or any implementation thereof.

[0015] Fourthly, this application provides a computer-readable storage medium storing instructions that instruct an electronic device to perform the method for optimizing model training samples as described in the first aspect or any implementation thereof.

[0016] Fifthly, this application provides a computer program product containing instructions that, when run on an electronic device, causes the electronic device to execute the method for optimizing model training samples as described in the first aspect or any implementation thereof.

[0017] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0018] As can be seen from the above technical solutions, this application has the following advantages:

[0019] This application provides a method for optimizing model training samples. The method first obtains a first negative sample, which includes a first prompt word and a first output. The first output is the output of the target model when the first prompt word is used as input. Then, negative cause identification is performed on the first output to determine the negative cause content in the first output. The negative cause content is a part of the content of the first output. Based on the negative cause content, a second negative sample is generated. The second negative sample includes a second prompt word and a second output. The second prompt word includes at least a part of the first prompt word and other content in the first output besides the negative cause content. The second output includes the negative cause content. The second negative sample is used to train the target model.

[0020] In this method, for training samples in the post-training phase of the target model (e.g., a generative model), negative factors are identified in the output of the first negative sample to accurately pinpoint the content that causes it to be negative. The negative sample is then reconstructed so that its output (i.e., the second negative sample) contains only the negative factor content. This optimizes the training samples. When the optimized negative samples are applied to the post-training phase, the probability of generating negative factors can be reduced specifically and precisely. This targeted optimization of the target model improves the efficiency of model fine-tuning, resulting in better inference performance of the fine-tuned target model. Attached Figure Description

[0021] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0022] Figure 1 A flowchart illustrating a method for optimizing model training samples provided in an embodiment of this application;

[0023] Figure 2 A schematic diagram of the structure of an apparatus for optimizing model training samples provided in an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0026] First, some technical terms and application scenarios involved in the embodiments of this application will be introduced.

[0027] Generative models are artificial intelligence models that can generate new content based on existing data. The existing data can be of different types, such as text, audio, video, and image data; in other words, generative models can be used to generate multimodal content.

[0028] Typically, generative models generate new content based on "next token prediction." That is, based on an existing data sequence, the generative model predicts the next most likely token. In practice, the existing data sequence is tokenized, decomposed into the smallest unit (multiple tokens) that the generative model can understand. Each token is then converted into an embedding vector, which represents the token's meaning and its position within the data sequence. Since generative models are usually based on a transformer architecture, the transformer's self-attention mechanism allows the model to fully consider the contextual information of all previous tokens (including tokens from the existing data sequence and already generated tokens) when generating each token, thus producing accurate and coherent content.

[0029] The training process of generative models is typically divided into a pre-training phase and a post-training phase. In the pre-training phase, the generative model learns to predict the next token by training on the dataset. This pre-training phase is usually self-supervised learning; that is, the generative model uses the first N tokens in the dataset to predict the generation probability of the (N+1)th token, and then, based on the actual (N+1)th token in the dataset, determines whether the generation probability of the (N+1)th token is the same as that of the actual (N+1)th token, thus performing self-training without the need for externally labeled training data.

[0030] In the post-training phase, the inference performance of the generative model on specific tasks is improved. Specifically, the post-training phase typically includes two parts: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). In SFT, the pre-trained generative model is fine-tuned using labeled training data, allowing the model to learn the style of the specific task, understand the task requirements, and improve its performance on that task. In RLHF, subjective human feedback is introduced to guide model fine-tuning, making the generative model's output more aligned with user needs and further optimizing the output quality.

[0031] In related technologies, during the post-training phase of RLHF, training samples are typically negative samples; that is, negative samples are used to fine-tune the generative model. Negative samples include prompts and outputs; when the prompt is input into the generative model, the output is an unexpected model output. By optimizing the generative model using negative samples, the model reduces the probability of generating tokens in the output when the prompt is used as input, thereby reducing the generation of unexpected content and aligning the model output with user needs.

[0032] However, in the above method, the output portion of negative samples is suppressed as a whole; that is, when the prompt word is used as model input, the probability of generating all tokens in the output is reduced. In real-world scenarios, not every token in the negative samples is undesirable, making it difficult for the above method to achieve targeted and precise model fine-tuning. Furthermore, suppressing negative samples as a whole consumes a large amount of training resources, resulting in low efficiency for model fine-tuning.

[0033] In view of this, this application provides a method for optimizing model training samples. The method first obtains a first negative sample, wherein the first negative sample includes a first prompt word and a first output. The first output is the output of the target model when the first prompt word is used as input. Then, negative cause identification is performed on the first output to determine the negative cause content in the first output, wherein the negative cause content is a part of the content of the first output. Based on the negative cause content, a second negative sample is generated, wherein the second negative sample includes a second prompt word and a second output. The second prompt word includes at least a part of the first prompt word and other content in the first output besides the negative cause content. The second output includes the negative cause content. The second negative sample is used to train the target model.

[0034] In this method, for training samples in the post-training phase of the target model (e.g., a generative model), negative factors are identified in the output of the first negative sample to accurately pinpoint the content that causes it to be negative. The negative sample is then reconstructed so that its output (i.e., the second negative sample) contains only the negative factor content. This optimizes the training samples. When the optimized negative samples are applied to the post-training phase, the probability of generating negative factors can be reduced specifically and precisely. This targeted optimization of the target model improves the efficiency of model fine-tuning, resulting in better inference performance of the fine-tuned target model.

[0035] To facilitate understanding of the technical solutions provided in the embodiments of this application, the following description will be provided in conjunction with the accompanying drawings. See also... Figure 1 The diagram shown illustrates a model fine-tuning method, which specifically includes:

[0036] S101: Obtain the first negative sample.

[0037] In this embodiment, negative samples (including the first negative sample and the second negative sample) are training samples used for fine-tuning the target model. The target model can be a generative model that generates new content based on the prediction of the next token; it can be understood as a pre-trained generative model that requires fine-tuning. In other words, the target model can be a generative model that has undergone a pre-training phase, and in the post-training phase, the target model is fine-tuned.

[0038] It should be noted that the embodiments of this application do not limit the data types that the target model can process. For example, the target model can be a generative model for processing multimodal data such as text data, audio data, image data, and video data. In other words, the optimized training samples in the embodiments of this application can be applied to generative models of different modalities and have good universality.

[0039] In the post-training phase, training samples typically consist of prompts and outputs. These training samples, consisting of prompts and outputs, can also be called training question-answer pairs, and the prompts and outputs can be referred to as instructions and responses. In SFT, by fine-tuning the model using these training samples, the generative model increases the probability of generating outputs from the training samples when the prompts from the training samples are used as model input. In RLHF, training samples can also include negative samples consisting of prompts and outputs. When the prompts from the negative samples are used as model input, the generative model decreases the probability of generating outputs from the negative samples.

[0040] In this embodiment, the first negative sample includes a first prompt word and a first output. The first output is the output of the target model when the first prompt word is used as input. Since the first prompt word and the first output constitute the first negative sample, the first output can also be understood as the model output that does not meet expectations or user needs when the first prompt word is used as input. In other words, when the first prompt word is input into the target model, it is undesirable for the target model to return the first output.

[0041] This application does not limit the method of obtaining the first negative sample. In some embodiments, multiple outputs corresponding to the first prompt word are labeled, for example, manually labeled by annotators or automatically labeled by a classification model. The outputs that represent negative samples are determined as the first outputs, forming the first negative sample consisting of the first prompt word and the first output. In other embodiments, negative sample rules are pre-set. These rules can be used to describe model outputs that belong to negative samples. Based on the negative sample rules, multiple outputs corresponding to the first prompt word are judged, and the first output that matches the negative sample rules is selected, forming the first negative sample consisting of the first prompt word and the first output. In still other embodiments, a reward model is pre-trained, and the reward model is used to judge multiple outputs corresponding to the first prompt word to determine the first output, forming the first negative sample consisting of the first prompt word and the first output.

[0042] S102: Perform negative factor identification on the first output to determine the negative factor content in the first output.

[0043] In the post-training phase of RLHF, unlike the traditional method that treats the overall output of the negative sample as the target of suppression, the method provided in this application embodiment identifies the negative cause of the first output in the first negative sample, finds the negative cause content that causes the first output to be a negative sample, so as to optimize the first negative sample and achieve fine-tuning of the model.

[0044] In this context, "negative cause" can be understood as the factor that causes the first output to be a negative sample (e.g., it does not meet expectations or user needs), and "negative cause content" can be understood as the content in the first output that satisfies the negative cause. The negative cause content can be all of the first output (also called global negative cause) or only a portion of the first output (also called local negative cause). It is understood that when the negative cause content is all of the first output, it indicates that all of the first output's content is what causes it to be a negative sample, meaning the first output as a whole does not meet expectations or user needs, such as being too long or too short, or lacking specific key information. In the post-training RLHF stage, it is necessary to suppress all of the first output's content to reduce the probability of generating all of the first output's content when the first prompt word is used as model input. This is no different from traditional model fine-tuning methods. Therefore, this application focuses on the case where the negative cause content is only a portion of the first output's content; that is, after identifying the negative cause of the first output, the negative cause content in the first output is only a portion of the first output's content, such as a portion of the token.

[0045] In some possible implementations, a negative cause identification model is used for negative cause identification. Specifically, a first output is sent to the negative cause identification model, and the negative cause content in the first output returned by the negative cause identification model is received.

[0046] The negative cause identification model can be understood as a model with the ability to identify negative causes. For example, the negative cause identification model can be a model trained using text labeled with negative cause content. By sending the first output to the negative cause identification model, the negative cause identification model can find the reason why the first output is a negative sample, output the negative cause content in the first output, and realize negative cause identification.

[0047] In some other possible implementations, the negative factors in the first output can be identified according to the set negative factor rules to determine the negative factor content in the first output.

[0048] Among them, the negative cause rule can be used to indicate the reason why the output is a negative sample. That is, by pre-determining the negative cause rule, the negative cause identification is completed by determining which contents in the first output do not meet expectations or user needs.

[0049] In practical applications, negative cause rules can be flexibly defined based on specific scenarios and model fine-tuning requirements. The negative cause identification process under different scenarios and model requirements is explained below.

[0050] In some embodiments, the negative cause rule includes a first negative cause rule, the first prompt word is described by the target language, and the first output is identified according to the first negative cause rule, and the non-target language content in the first output is determined as the negative cause content in the first output.

[0051] In other words, the user expects the target model to respond (i.e., the first output) in the same language as the question (i.e., the first prompt word). The first negative cause rule can be understood as the first output being a negative sample because the language of the first prompt word and the first output do not match. In this case, the negative cause content in the first output is content that does not belong to the target language. It is precisely because the first output contains content that does not belong to the target language that it becomes a negative sample.

[0052] In other embodiments, the negative cause rule includes a second negative cause rule. According to the second negative cause rule, the first output is identified as a negative cause, and the content in the first output that hits the keyword set is determined as the negative cause content in the first output.

[0053] The keyword set can include multiple pre-configured keywords, which can be understood as words that are not expected to appear in the model output. In other words, the user wants the target model's output to not contain keywords from the keyword set. The second negative cause rule can be understood as the first output hitting the keyword set, causing the first output to be a negative sample because it contains keywords from the keyword set. In this case, the negative cause content in the first output is the content of the first output that hits the keyword set (i.e., the keywords present in the first output). It is precisely because the first output contains keywords from the keyword set that it becomes a negative sample.

[0054] In other embodiments, the negative cause rule includes a third negative cause rule. According to the third negative cause rule, the first output is identified as a negative cause, and the garbled content in the first output whose length meets the length threshold is determined as the negative cause content in the first output.

[0055] In this context, garbled content can be understood as content without punctuation, difficult to read, or meaningless (e.g., content composed of symbols such as "#", "￥", or "%"). In other words, considering the possibility of the target model, especially after SFT, generating garbled content, users expect the target model's output to not contain large sections of garbled content. The third negative cause rule can be understood as identifying garbled content of a certain length in the first output, causing the first output to be a negative sample because of this garbled content. In this case, the negative cause in the first output is the garbled content of a certain length in the first output; it is precisely because of the presence of garbled content in the first output that it is a negative sample.

[0056] In other embodiments, the target model employs retrieval-augmented generation (RAG) technology for reasoning. RAG technology consists of two phases: a knowledge retrieval phase and a response generation phase. The knowledge retrieval phase retrieves knowledge information related to the question text from a knowledge base, while the response generation phase combines this knowledge information to generate a response text.

[0057] In this embodiment of the application, the knowledge base may include multiple knowledge articles. By searching the knowledge base, knowledge articles related to the problem are recalled. The knowledge articles recalled from the knowledge base are added to the first prompt words. The target model performs reasoning based on the knowledge articles recalled from the knowledge base in the first prompt words. For example, it determines the most relevant target knowledge articles that can be used to solve the problem from the knowledge articles recalled from the knowledge base.

[0058] For the above scenario, the negative cause rule includes a fourth negative cause rule. The first output includes first knowledge information, which includes at least one of the following: the title of the first knowledge article, the instruction information used to locate the first knowledge article, and the content of the first knowledge article. According to the fourth negative cause rule, negative cause identification is performed on the first output, and at least one of the following in the first output is determined as the negative cause content in the first output: the title of the first knowledge article that is empty or contains an error, the instruction information used to locate the first knowledge article that is empty or contains an error, and the content of the first knowledge article that contains a formatting error.

[0059] The indicative information used to locate the first knowledge article can be a link to the first knowledge article, a Uniform Resource Locator (URL), etc. In other words, the user expects all knowledge information references in the model output generated by the target model to be correct. The fourth negative cause rule can be understood as the existence of incorrect knowledge information references in the first output. The reason the first output is a negative sample is due to incorrect title (e.g., empty, or different from the title in the first prompt), incorrect indicative information (e.g., empty, or different from the indicative information in the first prompt), incorrect formatting, etc. In this case, the negative cause content in the first output is the incorrect knowledge information reference content in the first output. It is precisely because of the incorrect knowledge information reference content in the first output that the first output is a negative sample.

[0060] To illustrate with specific examples, the first prompt word could be as follows:

[0061] "As a question-and-answer expert, your job is to answer provided questions based on relevant sections of a given knowledge article."

[0062] Question: How to repurchase a computer?

[0063] Knowledge Article 1:

[0064] URL: http: / / url.for.article.1.com

[0065] Title: How to apply for a laptop buyback program at maturity

[0066] Content: This article will introduce the laptop buyback policy upon maturity, including instructions and application methods for a paid buyback upon maturity...

[0067] Knowledge Article 2:

[0068] URL: http: / / url.for.article.2.com

[0069] Title: Frequently Asked Questions about Computer Buyback Programs at Maturity

[0070] Content: This article will introduce common questions regarding computer buyback programs upon maturity, for your reference…

[0071] Among them, the first knowledge articles are Knowledge Article 1 and Knowledge Article 2.

[0072] ① When the negative cause rule includes the first negative cause rule, if the first output is as follows:

[0073] "To buy back the computer, here are the specific steps:"

[0074] First, make sure you are qualified to repurchase a computer;

[0075] Then……"

[0076] Since the first prompt word is described in Chinese, i.e., the target language is Chinese, the first output includes non-Chinese content, i.e., the first output includes non-target language content. By performing negative factor identification on the first output, it is determined that the negative factor content in the first output is non-target language content.

[0077] ② When the negative cause rule includes the fourth negative cause rule, if the first output is as follows:

[0078] To buy back your computer, here are the specific steps:

[0079] First, you need to ensure that you are eligible to apply for a buyback;

[0080] Secondly...

[0081] at last……

[0082] For details, please refer to: [](http: / / url.for.article.1.com) and [FAQ on Computer Maturity Buyback](http: / / url.for.article.2.com)”

[0083] Alternatively, if the first output is as follows:

[0084] To buy back your computer, here are the specific steps:

[0085] First, you need to ensure that you are eligible to apply for a buyback;

[0086] Secondly...

[0087] at last……

[0088] For details, please refer to: "[How to apply for a laptop buyback program at maturity](http: / / url.for.article.1.com)" and "[Common issues with laptop buyback programs at maturity](http: / / url.for.article.2.com)".

[0089] Since the first output contains an empty title of the first knowledge article ([]) or an incorrect title of the first knowledge article ([Computer Maturity Buyback Common FAQs]), by identifying the negative cause in the first output, it is determined that the negative cause content in the first output is an empty or incorrect title of the first knowledge article.

[0090] ③ When the negative cause rule includes the fourth negative cause rule, if the first output is as follows:

[0091] To buy back your computer, here are the specific steps:

[0092] First, you need to ensure that you are eligible to apply for a buyback;

[0093] Secondly...

[0094] at last……

[0095] For details, please refer to: [How to apply for a laptop buyback program at maturity]() and [Frequently Asked Questions about Laptop Buyback Programs at Maturity](http: / / url.for.article.2.com)”

[0096] Alternatively, if the first output is as follows:

[0097] To buy back your computer, here are the specific steps:

[0098] First, you need to ensure that you are eligible to apply for a buyback;

[0099] Secondly...

[0100] at last……

[0101] For details, please refer to: [How to apply for a laptop buyback program at maturity](http: / / url.for.article.1.com) and [Frequently Asked Questions about Laptop Buyback Programs at Maturity](http: / / other.url.com)

[0102] Since the first output contains empty or erroneous indication information for locating the first knowledge article (http: / / other.url.com), the negative cause content in the first output is determined to be empty or erroneous indication information for locating the first knowledge article by identifying the negative cause in the first output.

[0103] ④ When the negative cause rule includes the fourth negative cause rule, if the first output is as follows:

[0104] To buy back your computer, here are the specific steps:

[0105] First, you need to ensure that you are eligible to apply for a buyback;

[0106] Secondly...

[0107] at last……

[0108] For details, please refer to: http: / / url.for.article.1.com and http: / / url.for.article.2.com, "How to apply for laptop buyback upon maturity, Frequently Asked Questions about laptop buyback upon maturity".

[0109] Because the first output contains content from a first knowledge article with a formatting error (the knowledge information referenced is not presented in Markdown format), the negative cause content in the first output is determined to be the content from the first knowledge article with a formatting error by identifying the negative cause in the first output.

[0110] In this way, by identifying the negative causes of the first output, the negative causes that make the first output a negative sample (such as not meeting user expectations or user needs) can be accurately identified, so that the negative causes can be precisely suppressed in the subsequent RLHF, rather than suppressing the first output as a whole.

[0111] In some embodiments, the first output matches multiple negative cause rules, meaning the first output contains negative cause content corresponding to multiple different negative cause rules. In this case, one of the multiple negative cause contents present in the first output can be selected as the final negative cause content.

[0112] Specifically, according to the set negative factor rules, negative factor recognition is performed on the first output to determine multiple candidate negative factor contents in the first output that hit the negative factor rules, respectively determine the starting positions of the multiple candidate negative factor contents in the first output, and determine the candidate negative factor content with the smallest starting position as the negative factor content in the first output.

[0113] Illustrating with an example, let the length of the first output be represented by output_len, and the negative factor content be represented by <pos, len>, where pos is the starting position of the negative factor content in the first output and len is the length of the negative factor content. For each candidate negative factor content that belongs to a local negative factor, there is len < output_len and pos + len ≤ output_len. When selecting the final negative factor content from multiple candidate negative factor contents, the candidate negative factor content with the smallest starting position, that is, the smallest pos and the closest to the first token in the first output, is determined as the final negative factor content in the first output.

[0114] Since the target model is a generative model that generates new content based on the prediction of the next token, by determining the candidate negative factor content with the smallest starting position as the final negative factor content in the first output, after accurately suppressing the negative factor content subsequently, the probability of the target model generating the negative factor content is reduced, and the generation probabilities of other candidate negative factor contents originally after the negative factor content in the first output also decrease accordingly. In this way, there is no need to suppress each candidate negative factor content in the first output. By suppressing the candidate negative factor content with the earliest position, the generation probabilities of other unexpected contents in the first output are reduced together, saving training resources and achieving efficient model fine-tuning.

[0115] S103: Generate a second negative sample according to the negative factor content.

[0116] After performing negative factor recognition on the first output to determine the negative factor content that causes the first output to be a negative sample, a negative sample (i.e., the second negative sample) is reconstructed. In the embodiment of the present application, the second negative sample includes: a second prompt word and a second output. The second prompt word includes: at least part of the content of the first prompt word and the other content in the first output except the negative factor content. The second output includes the negative factor content.

[0117] That is to say, the original first prompt word and some other content in the first output that does not belong to the negative factor content are constructed into a new second prompt word, and the negative factor content in the original first output is constructed into a new second output. In this way, in the subsequent RLHF in the training stage, targeted suppression can be performed on the negative factor content, and when the second prompt word is given, the probability of generating the negative factor content is accurately reduced.

[0118] In the specific implementation, the negative cause content starts at the Mth token in the first output, and its length is N tokens, where M and N are integers greater than 0. In this case, a second prompt word is generated based on the first prompt word and the content in the first output before the Mth token, and a second output is generated based on the content of the Mth token to the (M+N-1)th token in the first output.

[0119] For example, M is 3 and N is 2. That is, the starting position of the negative cause content in the first output is the 3rd token, and the length of the negative cause content is 2 tokens. By reconstructing the negative sample, the second prompt word includes the first prompt word and the first 2 tokens in the first output, and the second output includes the 3rd to 4th tokens in the first output.

[0120] Since the negative cause content starts at the Mth token in the first output, it means that the first M-1 tokens in the first output are as expected and are the correct model output. Therefore, when reconstructing the negative sample, the first M-1 tokens in the first output are added to the second prompt, indicating that there is no need to suppress the first M-1 tokens in the first output. The content of the Mth to M+N-1th tokens in the first output is the negative cause content itself. The second output is constructed using the content of the Mth to M+N-1th tokens in the first output, indicating that only the part of the first output that belongs to the negative cause content needs to be suppressed. Compared with the first negative sample, the length of the second output in the second negative sample is shorter than the length of the first output in the first negative sample, thus optimizing the negative sample and enhancing the accuracy of model fine-tuning.

[0121] To illustrate with specific examples, the first prompt word could be as follows:

[0122] "As a question-and-answer expert, your job is to answer provided questions based on relevant sections of a given knowledge article."

[0123] Question: How to repurchase a computer?

[0124] Knowledge Article 1:

[0125] URL: http: / / url.for.article.1.com

[0126] Title: How to apply for a laptop buyback program at maturity

[0127] Content: This article will introduce the laptop buyback policy upon maturity, including instructions and application methods for a paid buyback upon maturity...

[0128] Knowledge Article 2:

[0129] URL: http: / / url.for.article.2.com

[0130] Title: Frequently Asked Questions about Computer Buyback Programs at Maturity

[0131] Content: This article will introduce common questions regarding computer buyback programs upon maturity, for your reference…

[0132] ①If the first output is as follows:

[0133] To buy back your computer, here are the specific steps:

[0134] First, make sure you are qualified to repurchase a computer;

[0135] Then……"

[0136] The negative factor content in the first output is non-target language content. By reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows:". The second output of the second negative sample is "First, make sure you are qualified to repurchase a computer;".

[0137] ②If the first output is as follows:

[0138] To buy back your computer, here are the specific steps:

[0139] First, the first keyword.

[0140] Then……"

[0141] The negative factor content in the first output is the content of the hit keyword set (i.e., the first keyword). By reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows: First, ". The second output in the second negative sample is "the first keyword".

[0142] ③If the first output is as follows:

[0143] To buy back your computer, here are the specific steps:

[0144] First, please refer to: [](http: / / url.for.article.1.com)

[0145] Then, please refer to: [Frequently Asked Questions about Computer Buyback Programs at Maturity](http: / / url.for.article.2.com)

[0146] The negative factor content in the first output is the title of the first knowledge article that is empty. By reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows: First, please refer to: []", and the second output in the second negative sample is "]".

[0147] It should be noted that in the above case, the negative cause content is the title of the first knowledge article which is empty. However, since the left half of the symbol “[]” used to represent the title of the first knowledge article is the correct model output, the “[” is added to the second prompt word. The second output only includes the right half of the symbol “[]” used to represent the title of the first knowledge article.

[0148] ④ If the first output is as follows:

[0149] To buy back your computer, here are the specific steps:

[0150] First, please refer to: [How to apply for a laptop buyback program at maturity](http: / / url.for.article.1.com)

[0151] Then, please refer to: [Frequently Asked Questions about Computer Buyback at Maturity]()”

[0152] The negative factor content in the first output is an empty instruction for locating the first knowledge article. By reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To buy back a computer, the specific methods are as follows: First, please refer to: [How to apply for a laptop buyback upon expiration](http: / / url.for.article.1.com) Then, please refer to: [Computer Buyback Upon Expiration Common FAQs](", the second output in the second negative sample is ")".

[0153] Similarly, in the above case, the negative cause content is the existence of empty indication information for locating the first knowledge article. However, since the left half of the symbol “()” used to indicate the indication information for locating the first knowledge article is the correct model output, the “()” is added to the second prompt word. The second output only includes the right half of the symbol “()” used to indicate the indication information for locating the first knowledge article.

[0154] ⑤ If the first output is as follows:

[0155] To buy back your computer, here are the specific steps:

[0156] First... please refer to: [How to Buy Back a Computer](http: / / url.for.article.3.com);

[0157] Then... please refer to: [Frequently Asked Questions about Computer Buyback Programs at Maturity](http: / / url.for.article.2.com)

[0158] The negative content in the first output is the erroneous instruction information used to locate the first knowledge article. In some possible implementations, by reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows: First..., please refer to: [How to repurchase a computer](", the second output in the second negative sample is "http: / / url.for.article.3.com".

[0159] In some other possible implementations, by reconstructing the negative sample, the second prompt word in the second negative sample includes the first prompt word and "To repurchase a computer, the specific method is as follows: First..., please refer to: []", and the second output in the second negative sample is "How to repurchase a computer" (http: / / url.for.article.3.com).

[0160] Furthermore, the second negative sample can be used to train the target model. Specifically, after reconstructing the negative sample, the target model is subjected to RLHF using the second negative sample, thus completing the training phase.

[0161] In practice, positive samples are obtained, and the positive samples and the second negative sample are used as training samples to fine-tune the target model. The positive samples include a third prompt word and a third output. For example, the third output can be the model output that meets expectations and user needs when the third prompt word is used as model input.

[0162] In this way, by using both positive and second negative samples as training samples for RLHF in the post-training stage, the target model after RLHF can generate a model output that is close to the third output in the positive sample for a given prompt word, avoiding the generation of a model output that is similar to the second output in the second negative sample, thus aligning with user needs.

[0163] In some possible implementations, RLHF in the post-training phase can employ pairwise training, meaning the third cue word in a positive sample differs from the second cue word in the second negative sample. In this case, if the target model has already undergone the SFT in the post-training phase, the training samples from the SFT can be used as positive samples, combined with the second negative sample to form the training samples for RLHF, using a training method that supports both positive and negative samples, such as the Kahneman-Tversky optimization (KTO) method for RLHF.

[0164] In other possible implementations, the post-training phase can employ pairwise training, where the third cue word in a positive sample is the same as the second cue word in a second negative sample. In this case, positive samples can be obtained in different ways. In some embodiments, the second cue word is determined as the third cue word in the positive sample, and the second output is corrected according to the negative factor rule to obtain the third output in the positive sample.

[0165] For example, the negative cause rule includes the first negative cause rule, which can be understood as the first prompt word not matching the language of the first output. In this case, the third output is obtained by correcting the second output, which is described in a non-target language, to the content described in the target language.

[0166] In other words, by correcting the second output in the second negative sample, the second output is corrected to meet the expected model output, and then the second prompt word in the second negative sample and the corrected second output are used to form a positive sample.

[0167] In other embodiments, the second prompt word is determined as the third prompt word in the positive sample, the second prompt word is input into the calibration model multiple times, multiple candidate outputs returned by the calibration model are received, and the candidate output that meets the requirements of the positive sample is determined as the third output in the positive sample.

[0168] The calibration model can be a model capable of calibrating training samples to obtain positive samples that meet expectations and user needs. In some possible implementations, the calibration model can be a generative model different from the target model. For example, the calibration model can be a self-developed generative model that has completed the pre-training and post-training phases and is ready for practical use.

[0169] By using the second prompt word in the second negative sample as the model input of the calibration model, the calibration model generates a series of candidate outputs. The candidate output that does not hit the negative cause rule and meets the expectations is selected as the third output, forming a positive sample.

[0170] In pairwise training, methods such as direct preference optimization (DPO) can be used for RLHF.

[0171] In this method, for training samples in the post-training phase of the target model (e.g., a generative model), negative factors are identified in the output of the first negative sample to accurately pinpoint the content that causes it to be negative. The negative sample is then reconstructed so that its output (i.e., the second negative sample) contains only the negative factor content. This optimizes the training samples. When the optimized negative samples are applied to the post-training phase, the probability of generating negative factors can be reduced specifically and precisely. This targeted optimization of the target model improves the efficiency of model fine-tuning, resulting in better inference performance of the fine-tuned target model.

[0172] The above text combined Figure 1 The method for optimizing model training samples provided in the embodiments of this application has been described in detail. The apparatus and equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0173] See Figure 2 The schematic diagram of the device for optimizing model training samples shown is provided. The device 20 includes:

[0174] The acquisition module 201 is used to acquire a first negative sample; wherein, the first negative sample includes: a first prompt word and a first output, and the first output is the output of the target model when the first prompt word is used as the input of the target model;

[0175] The identification module 202 is used to identify negative causes in the first output and determine the negative cause content in the first output, wherein the negative cause content is a part of the content of the first output;

[0176] The optimization module 203 is used to generate a second negative sample based on the negative cause content; wherein the second negative sample includes: a second prompt word and a second output, the second prompt word includes: at least part of the content other than the negative cause content in the first prompt word and the first output, the second output includes the negative cause content, and the second negative sample is used to train the target model.

[0177] In some possible implementations, the identification module 202 is specifically used for:

[0178] The first output is sent to the negative cause identification model, and the negative cause content in the first output returned by the negative cause identification model is received; or...

[0179] According to the set negative cause rules, the first output is identified to determine the negative cause content in the first output; wherein, the negative cause rules are used to indicate the reason why the output is a negative sample.

[0180] In some possible implementations, the identification module 202 is specifically used for:

[0181] According to the set negative cause rules, the first output is identified as a negative cause, and multiple candidate negative cause contents in the first output that match the negative cause rules are determined.

[0182] Determine the starting position of each of the multiple candidate negative cause contents in the first output;

[0183] The candidate negative factor content with the smallest starting position is determined as the negative factor content in the first output.

[0184] In some possible implementations, the starting position of the negative cause content in the first output is the Mth token, the length of the negative cause content is N tokens, and M and N are integers greater than 0; the optimization module 203 is specifically used for:

[0185] Based on the first prompt word and the content in the first output preceding the Mth token, generate a second prompt word; and,

[0186] The second output is generated based on the contents of the Mth to M+N-1th tokens described in the first output.

[0187] In some possible implementations, the negative cause rule includes a first negative cause rule, where the first prompt word is described by the target language, and the recognition module 202 is specifically used for:

[0188] According to the first negative cause rule, the first output is identified as a negative cause, and the content in the first output that is not in the target language is determined as the negative cause content in the first output.

[0189] In some possible implementations, the negative cause rule includes a second negative cause rule, and the identification module 202 is specifically used for:

[0190] According to the second negative cause rule, the first output is identified as a negative cause, and the content that hits the keyword set in the first output is determined as the negative cause content in the first output.

[0191] In some possible implementations, the negative cause rule includes a third negative cause rule, and the identification module 202 is specifically used for:

[0192] According to the third negative cause rule, the first output is identified as a negative cause, and the garbled content in the first output whose length meets the length threshold is determined as the negative cause content in the first output.

[0193] In some possible implementations, the negative cause rule includes a fourth negative cause rule, and the first output includes first knowledge information, which includes at least one of the following: the title of the first knowledge article, indication information for locating the first knowledge article, and the content of the first knowledge article; the identification module 202 is specifically used for:

[0194] According to the fourth negative cause rule, the first output is identified as a negative cause, and at least one of the following in the first output is determined as the negative cause content in the first output: the title of the first knowledge article that is empty or contains an error, the instruction information for locating the first knowledge article that is empty or contains an error, and the content of the first knowledge article that contains a format error.

[0195] In some possible implementations, the device 20 further includes a training module, which is used for:

[0196] Obtain positive samples; wherein, the positive samples include: a third prompt word and a third output;

[0197] The positive sample and the second negative sample are used as training samples to fine-tune the target model.

[0198] In some possible implementations, the training module is specifically used for:

[0199] The second prompt word is identified as the third prompt word in the positive sample;

[0200] According to the negative cause rule, the second output is corrected to obtain the third output in the positive sample; or, the second prompt word is input into the correction model multiple times, multiple candidate outputs returned by the correction model are received, and the candidate output that meets the requirements of the positive sample among the multiple candidate outputs is determined as the third output in the positive sample.

[0201] The apparatus 20 for optimizing model training samples according to the embodiments of this application can correspond to executing the method described in the embodiments of this application, and the above and other operations and / or functions of each module / unit of the apparatus 20 for optimizing model training samples are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0202] This application also provides an electronic device. This electronic device is specifically used to implement, as described above. Figure 2 The function of the device 20 for optimizing model training samples in the illustrated embodiment.

[0203] Figure 3 A structural schematic diagram of an electronic device 300 is provided, such as... Figure 3As shown, the electronic device 300 includes a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0204] Bus 301 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0205] The processor 302 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0206] The communication interface 303 is used for external communication. For example, the communication interface 303 can be used to communicate with a terminal.

[0207] Memory 304 may include volatile memory, such as random access memory (RAM). Memory 304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0208] The memory 304 stores executable code, and the processor 302 executes the executable code to perform the aforementioned method of optimizing the training samples of the model.

[0209] Specifically, in achieving Figure 2 In the case of the illustrated embodiment, and Figure 2 When the modules or units of the apparatus 20 for optimizing model training samples described in the embodiment are implemented by software, the following steps are performed: Figure 2The software or program code required for the functions of each module / unit can be partially or entirely stored in memory 304. Processor 302 executes the program code corresponding to each unit stored in memory 304, and performs the aforementioned method for optimizing model training samples.

[0210] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the method described above for optimizing model training samples applied to the apparatus 20 for optimizing model training samples.

[0211] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0212] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0213] When the computer program product is executed by a computer, the computer performs any of the methods described above for training the optimization model samples. The computer program product may be a software installation package; when any of the methods described above for training the optimization model samples is required, the computer program product may be downloaded and executed on the computer.

[0214] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0215] The units described in the embodiments of this application can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0216] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0217] In the context of embodiments of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0218] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0219] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0220] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0221] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0222] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for optimizing model training samples, characterized in that, The method comprises: obtaining a first negative sample; wherein the first negative sample comprises a first prompt word and a first output, the first output being an output of a target model when the first prompt word is input into the target model; and the first negative sample being a negative sample obtained based on at least one of text data, audio data, picture data or video data; performing negative factor identification on the first output to determine negative factor content in the first output, the negative factor identification being an identification operation on an influencing factor of the first output being a negative sample, and the negative factor content being part of the first output; generating a second negative sample according to the negative factor content; wherein the second negative sample comprises a second prompt word and a second output, the second prompt word comprising at least part of the first prompt word and other content in the first output except the negative factor content, and the second output comprising the negative factor content, the second negative sample being used for training the target model.

2. The method of claim 1, wherein, The negative factor identification on the first output to determine the negative factor content in the first output comprises: sending the first output to a negative factor identification model and receiving negative factor content in the first output returned by the negative factor identification model; or performing negative factor identification on the first output according to a set negative factor rule to determine the negative factor content in the first output; wherein the negative factor rule is used to indicate a reason for the output being a negative sample.

3. The method of claim 2, wherein, The negative factor identification on the first output according to the set negative factor rule to determine the negative factor content in the first output comprises: performing negative factor identification on the first output according to a set negative factor rule to determine a plurality of candidate negative factor contents in the first output that hit the negative factor rule; determining a starting position of each of the plurality of candidate negative factor contents in the first output; determining the candidate negative factor content with the smallest starting position as the negative factor content in the first output.

4. The method of claim 1, wherein, The starting position of the negative factor content in the first output is an Mth token, the length of the negative factor content is N tokens, and the M and the N are integers greater than 0; The generation of the second negative sample according to the negative factor content comprises: generating a second prompt word according to the first prompt word and content in the first output located before the Mth token; and generating a second output according to content of the Mth token to an M+N-1th token in the first output. The negative factor rule comprises a first negative factor rule, the first prompt word is described in a target language, and the negative factor identification on the first output according to the set negative factor rule to determine the negative factor content in the first output comprises:

5. The method of claim 2, wherein, performing negative factor identification on the first output according to the first negative factor rule, and determining content other than the target language in the first output as the negative factor content in the first output. The negative factor rule comprises a second negative factor rule, and the negative factor identification on the first output according to the set negative factor rule to determine the negative factor content in the first output comprises:

6. The method of claim 2, wherein, ​ According to the second negative factor rule, the first output is subjected to negative factor identification, and content in the first output that hits the keyword set is determined as negative factor content in the first output.

7. The method of claim 2, wherein, The negative factor rule includes a third negative factor rule, and the negative factor identification of the first output according to the set negative factor rule to determine the negative factor content in the first output includes: According to the third negative factor rule, the first output is subjected to negative factor identification, and the garbled code content in the first output that meets the length threshold is determined as the negative factor content in the first output.

8. The method of claim 2, wherein, The negative factor rule includes a fourth negative factor rule, and the first output includes first knowledge information, which includes at least one of the following: a title of a first knowledge article, indication information for positioning the first knowledge article, and content of the first knowledge article; The negative factor identification of the first output according to the set negative factor rule to determine the negative factor content in the first output includes: According to the fourth negative factor rule, the first output is subjected to negative factor identification, and at least one of the following in the first output is determined as negative factor content in the first output: the title of the first knowledge article is empty or has an error, the indication information for positioning the first knowledge article is empty or has an error, and the content of the first knowledge article has a format error.

9. The method of any one of claims 2-3, or 5-8, wherein, The method further includes: Obtaining a positive sample; wherein the positive sample includes a third prompt word and a third output; Using the positive sample and the second negative sample as training samples to fine-tune the target model.

10. The method of claim 9, wherein, The method further includes: Determining the second prompt word as the third prompt word in the positive sample; According to the negative factor rule, the second output is corrected to obtain the third output in the positive sample; or the second prompt word is input multiple times into the correction model, and multiple candidate outputs returned by the correction model are received, and a candidate output that meets the positive sample requirement in the multiple candidate outputs is determined as the third output in the positive sample.

11. An apparatus for optimizing model training samples, the apparatus comprising: The device includes: An obtaining module configured to obtain a first negative sample; wherein the first negative sample includes a first prompt word and a first output, the first output is an output of a target model when the first prompt word is input into the target model; and the first negative sample is a negative sample obtained based on at least one of text data, audio data, picture data, or video data; An identification module configured to perform negative factor identification on the first output to determine negative factor content in the first output, the negative factor identification being an identification operation on an influencing factor of the first output as a negative sample, and the negative factor content being part of the content of the first output; An optimization module configured to generate a second negative sample according to the negative factor content; wherein the second negative sample includes a second prompt word and a second output, the second prompt word includes at least part of the first prompt word and other content in the first output except the negative factor content, and the second output includes the negative factor content, the second negative sample being used to train the target model.

12. An electronic device, comprising: The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method of any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer program product comprises computer readable instructions for implementing the method of any one of claims 1-10.

14. A computer program product, characterised in that, The computer program product comprises computer readable instructions for implementing the method of any one of claims 1-10.

Images