Model training method and device

Abstract

One or more embodiments of the invention disclose a model training method and device, and the method comprises the steps: firstly obtaining an intermediate model generated in the iterative training process of a target question and answer model, and an auxiliary model used for generating a preference sample; inputting input data related to the question and answer service into a first model obtained by current iteration in the intermediate model, any second model before the first model and an auxiliary model respectively; respectively obtaining a first output result corresponding to the first model, a second output result corresponding to the second model and a third output result corresponding to the auxiliary model, and then constructing a first preference sample based on the input data, the first output result and the third output result, and constructing a second preference sample based on the input data, the first output result and the second output result, and finally continuing to perform model training on the first model based on the first preference sample, the second preference sample and a preset loss function to obtain a trained target question and answer model.

Description

Technical Field

[0001] This document relates to the field of model processing technology, and in particular to a method and device for model training. Background Art

[0002] At present, with the rapid development of large language models in the field of natural language, as a generative model, the application space in different fields of generating high-quality answers based on prompts through large language models is getting larger and larger. At the same time, as people pay more and more attention to their privacy data, when using the pre-trained general large language model for a specific field, it is usually necessary to further fine-tune the pre-trained large language model to obtain a large language model suitable for a specific field. Therefore, how to efficiently obtain preference samples, and then fine-tune the large language model based on the obtained preference samples to improve the efficiency and accuracy of the large language model used in a specific field is an important issue. Based on this, it is necessary to provide a model training method that can obtain preference samples more efficiently and improve data utilization. Summary of the invention

[0003] On the one hand, one or more embodiments of the present specification provide a model training method, including: obtaining an intermediate model generated during the iterative training of a target question and answer model and an auxiliary model for generating preference samples, wherein for the same input data, the output result of the auxiliary model is better than the output result of the intermediate model; inputting the input data related to the question and answer business into the first model obtained in the current iteration of the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; constructing a first preference sample based on the input data, the first output result, and the third output result, and constructing a second preference sample based on the input data, the first output result, and the second output result; continuing to perform model training on the first model based on the first preference sample, the second preference sample, and a preset loss function to obtain a trained target question and answer model.

[0004] On the other hand, one or more embodiments of the present specification provide a model training method, including: obtaining an intermediate model generated during the iterative training of a target question and answer model; determining a first model obtained in the current iteration of the intermediate model, and a difficult sample in the previous iterative training process corresponding to the first model, and adding the difficult sample to the sample used for model training of the first model, the difficult sample being a sample whose output result has a similarity with a label corresponding to input data related to the question and answer business that is less than a preset similarity, and the output result is a result obtained by inputting the input data into the previous iterative training model corresponding to the first model; continuing model training of the first model based on the difficult sample, the sample used for model training of the first model, and a preset loss function to obtain a trained target question and answer model.

[0005] On the other hand, one or more embodiments of the present specification provide a model training device, including: an intermediate model and auxiliary model acquisition module, which acquires the intermediate model generated during the iterative training of the target question and answer model and the auxiliary model used to generate preference samples, wherein for the same input data, the output result of the auxiliary model is better than the output result of the intermediate model; an output result calculation module, which inputs the input data related to the question and answer business into the first model obtained by the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; a preference sample determination module, which constructs a first preference sample based on the input data, the first output result, and the third output result, and constructs a second preference sample based on the input data, the first output result, and the second output result; a training module, which continues to perform model training on the first model based on the first preference sample, the second preference sample, and a preset loss function to obtain a trained target question and answer model.

[0006] On the other hand, one or more embodiments of the present specification provide another model training device, including: an intermediate model acquisition module, which acquires an intermediate model generated during the iterative training of a target question and answer model; a difficult sample determination module, which determines a first model obtained in the current iteration of the intermediate model, and a difficult sample in the previous iterative training process corresponding to the first model, and adds the difficult sample to the sample used for model training of the first model, wherein the difficult sample is a sample whose output result has a similarity with a label corresponding to input data related to the question and answer business that is less than a preset similarity, and the output result is a result obtained by inputting the input data into the previous iterative training model corresponding to the first model; a model training module, which continues to perform model training on the first model based on the difficult sample, the sample used for model training of the first model, and a preset loss function to obtain a trained target question and answer model.

[0007] On the other hand, one or more embodiments of the present specification provide an electronic device, comprising: a processor; and a memory arranged to store computer-executable instructions, wherein when the executable instructions are executed, the processor is enabled to: obtain an intermediate model generated during the iterative training of a target question-answering model and an auxiliary model for generating preference samples, wherein for the same input data, an output result of the auxiliary model is better than an output result of the intermediate model; input the input data related to the question-answering business into a first model obtained by the current iteration of the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; construct a first preference sample based on the input data, the first output result, and the third output result, and construct a second preference sample based on the input data, the first output result, and the second output result; continue to perform model training on the first model based on the first preference sample, the second preference sample, and a preset loss function to obtain a trained target question-answering model.

[0008] On another aspect, one or more embodiments of the present specification provide a storage medium for storing a computer program, which can be executed by a processor to implement the following process: obtaining an intermediate model generated during the iterative training of a target question and answer model and an auxiliary model for generating preference samples, wherein for the same input data, an output result of the auxiliary model is better than an output result of the intermediate model; inputting the input data related to the question and answer business into the first model obtained in the current iteration of the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; constructing a first preference sample based on the input data, the first output result, and the third output result, and constructing a second preference sample based on the input data, the first output result, and the second output result; continuing to perform model training on the first model based on the first preference sample, the second preference sample, and a preset loss function to obtain a trained target question and answer model.

[0009] On another aspect, one or more embodiments of the present specification further provide a computer program product, including a computer program, which implements the following process when executed by a processor: obtaining an intermediate model generated during the iterative training of a target question and answer model and an auxiliary model for generating preference samples, wherein for the same input data, an output result of the auxiliary model is better than an output result of the intermediate model; inputting the input data related to the question and answer business into the first model obtained in the current iteration of the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; constructing a first preference sample based on the input data, the first output result, and the third output result, and constructing a second preference sample based on the input data, the first output result, and the second output result; continuing to perform model training on the first model based on the first preference sample, the second preference sample, and a preset loss function to obtain a trained target question and answer model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1is a schematic flow chart of a model training method according to an embodiment of this specification; Figure 2 is a schematic diagram of preference samples in the self-comparison iteration phase according to an embodiment of the present specification; Figure 3 is a schematic flow chart of another model training method according to an embodiment of this specification; Figure 4 is a schematic diagram of preference samples in the initialization phase according to an embodiment of the present specification; Figure 5 is a schematic flow chart of another model training method according to an embodiment of this specification; Figure 6 is a schematic block diagram of a model training device according to an embodiment of this specification; Figure 7 is a schematic block diagram of another model training device according to an embodiment of this specification; Figure 8 It is a schematic block diagram of an electronic device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0012] One or more embodiments of this specification provide a model training method and device.

[0013] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0014] like Figure 1 As shown, an embodiment of this specification provides a model training method, and the execution subject of the method can be a terminal device or a server, wherein the terminal device can be a certain terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or an IoT device (specifically such as a smart watch, a car device, etc.). The server can be an independent server, or a server cluster composed of multiple servers, etc. The server can be a background server in the financial field or the online shopping field, or a background server of an application, etc. In this embodiment, the server is used as an example for detailed description. For the execution process of the terminal device, please refer to the following related content, which will not be repeated here. The method can specifically include the following steps: In step S102, an intermediate model generated during iterative training of the target question-answering model and an auxiliary model for generating preferred samples are obtained. For the same input data, the output result of the auxiliary model is better than the output result of the intermediate model.

[0015] The target question-answering model in the embodiment of this specification may be a model constructed by a neural network, or a pre-trained large language model. The large language model refers to a deep neural network model whose model parameters exceed a preset number threshold and / or whose model structure complexity exceeds a preset complexity threshold, such as a model with more than 100 million parameters. Large language models generally include two types: generative large models and discriminative large models (also known as decision-making large models). The target question-answering model in the embodiment of this specification may be a generative large language model. The generative large language model can be trained based on a large amount of text data, can understand and generate coherent and grammatically compliant natural language text, and contains rich world knowledge, bringing great progress in tasks such as text generation, machine translation, and knowledge question-answering. In implementation, the target question-answering model can be used for intelligent question-answering services, content creation services (for example, role-playing services based on different roles, content generation services, etc.), or customer service services, etc. In services such as intelligent question-answering services, content creation services, or customer service services, the target question-answering model can generate output results that meet user preferences based on the acquired user input data.

[0016] The intermediate model is a model generated during the iterative training of the target question-answering model. The auxiliary model is a model used to generate better preference samples. For the same input data, the output of the auxiliary model is better than the output of any intermediate model. In implementation, the auxiliary model can be an open source large language model with better effects and larger parameters, such as the GPT-4 model.

[0017] In step S104, the input data related to the question-and-answer business are respectively input into the first model obtained by the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model.

[0018] Obtain multiple input data related to the question-and-answer business, and input each input data into the first model, the second model, and the auxiliary model, respectively, to obtain the first output result corresponding to the first model, the second output result corresponding to the second model, and the third output result corresponding to the auxiliary model. Among them, the first model is an intermediate model, and is the intermediate model obtained in the current iteration. The first model is trained by inputting preference samples to obtain the trained first model, and the iteration is performed in this way to finally obtain the trained target question-and-answer model. The second model is the model before the first model, including any intermediate model generated during the iterative training process and the target question-and-answer model to be trained (also called the initial model).

[0019] In step S106 , a first preference sample is constructed based on the input data, the first output result and the third output result, and a second preference sample is constructed based on the input data, the first output result and the second output result.

[0020] Among them, the first preference sample and the second preference sample both include three parameters: input data related to the question-answering business (for example: how to learn programming?), better output results (for example: you can learn programming through resources such as M Online University, you can learn programming through steps such as ABC, etc.) and worse output results (for example: programming does not need to be learned, programming is too difficult, etc.). By inputting three different models in step S104, three different output results can be obtained, and by step S106, two preference samples can be constructed based on the three different output results, thereby effectively improving the collection efficiency of preference samples, and is conducive to improving data utilization efficiency, thereby improving the efficiency of model training.

[0021] In implementation, the first preference sample can be constructed based on the output results of the first model of the current iteration and the auxiliary model, and the second preference sample can be constructed based on the output results of the first model of the current iteration and the second model before the first model.

[0022] The preferred samples in the self-comparison iteration phase in the embodiments of this specification can be found in Figure 2 shown. Figure 2 middle, This is input data related to the question-answering business. is the first model of the current iteration, is any second model before the first model, As auxiliary model, , , The output results of the first model, the second model and the auxiliary model are respectively, and the preferred sample 1 is the first preferred sample, and the preferred sample 2 is the second preferred sample. Figure 2It can be seen that the training process of the target question answering model is iterative training. At the Tth iteration (loop), the model , also includes the first model The previous intermediate model The corresponding preference sample set (Right now and the dataset generated by the auxiliary model S), the first model of the current iteration The first preference sample and the second preference sample obtained in step S106 are used as the training model. Model training is performed . Sample input data X and input it into the auxiliary model Get the output result , input the first model of the current iteration Get the output result , input the randomly selected second model Get the output result , the output of the preset auxiliary model is better than the output of the first model, forming a preferred sample (First preference sample ),in, For input data X, the poor output result for , for better output results for At the same time, the output result of the first model is assumed to be better than the output result of the second model, forming a preferred sample (Second preference sample ),in, For input data X, the poor output result for , better output results for Input multiple input data related to the question-answering business, and obtain a first model consisting of multiple first preference samples and second preference samples. Corresponding preference sample set .

[0023] In step S108, the first model is further trained based on the first preference sample, the second preference sample and the preset loss function to obtain a trained target question-answering model.

[0024] Continue to see Figure 2 , and obtain a model consisting of multiple first preference samples and second preference samples After obtaining the corresponding preference sample set, the first model is further trained based on the obtained preference sample set and the preset loss function to obtain the trained target question-answering model.

[0025] In the implementation, a plurality of first preference samples and second preference samples are obtained, which are consistent with the first model. After obtaining the corresponding preference sample set, a reward model can be trained based on the obtained preference sample set, the output result is scored by the reward model, and the target question-answering model is subjected to reinforcement learning training using the reward model to obtain the trained target question-answering model. The target question-answering model can also be subjected to preference optimization training based on the obtained preference sample set to obtain the trained target question-answering model. The target question-answering model can also be subjected to supervised fine-tuning training and preference optimization training based on the obtained preference sample set.

[0026] The embodiment of the present specification provides a model training method, firstly, an intermediate model generated in the iterative training process of the target question-answering model and an auxiliary model for generating a preference sample are obtained, and for the same input data, the output result of the auxiliary model is better than the output result of the intermediate model, and secondly, the input data related to the question-answering business are respectively input into the first model obtained in the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, and the first output result corresponding to the first model, the second output result corresponding to the second model, and the third output result corresponding to the auxiliary model are respectively obtained, and then the first preference sample is constructed based on the input data, the first output result, and the third output result, and the second preference sample is constructed based on the input data, the first output result, and the second output result. Finally, the first model is further trained based on the first preference sample, the second preference sample, and the preset loss function to obtain the trained target question-answering model. In this way, by inputting each of the multiple input data related to the question-answering business into the first model, the second model, and the auxiliary model, three output results are obtained, and two preference samples are constructed based on the three output results, and two preference samples can be constructed based on one input data, thereby effectively improving the collection efficiency of preference samples, which is conducive to fully improving the utilization rate of data resources, and thus improving the efficiency and accuracy of model training. Moreover, in the embodiments of the present specification, the first preference sample and the second preference sample are constructed based on the output results of the current iterative model and any second model, and the preference sample construction method is to use the new model in the current iteration and the historical intermediate model for self-comparison. When collecting preference samples, the model and its output results obtained in the previous model iteration process can be fully utilized, which is beneficial to improving the utilization rate of data resources, and improving the collection efficiency and collection cost of preference samples. Moreover, since the intermediate model is fully utilized, the preference samples used for training are more targeted, which is beneficial to improving the accuracy of the target question-answering model obtained through training.

[0027] like Figure 3As shown, in the embodiment of the present specification, before the above step S108, step S110 may also be included: determining difficult samples in the previous iterative training process corresponding to the first model, and adding the difficult samples to the first preference samples and the second preference samples used for model training of the first model.

[0028] Among them, difficult samples are samples in which the similarity between output results in the same preference sample is less than the preset similarity.

[0029] In implementation, the method of determining difficult samples from the first preference samples and the second preference samples used to train the previous intermediate model of the first model can select difficult samples according to the application scenario, or determine the difficult samples according to the similarity between the two output results in each preference sample.

[0030] Accordingly, the processing of step S108 can be varied. An optional processing method is provided below. For details, please refer to the processing of step S1082 below.

[0031] In step S1082, based on the difficult sample, the first preference sample for training the first model, the second preference sample and the preset loss function, the first model is trained to obtain a trained target question-answering model.

[0032] It can be seen from the above steps S110 and S1082 that in the embodiment of the present specification, in the process of iterative training of the target question and answer model, difficult samples are determined from the first preference samples and the second preference samples used for model training of the previous intermediate model (the previous iterative training model corresponding to the first model), and then the difficult samples are added to the first preference samples and the second preference samples used for model training of the subsequent intermediate model (the first model), and based on the difficult samples, the first preference samples and the second preference samples used for model training of the subsequent intermediate model and the preset loss function, model training is performed on the subsequent intermediate model (the first model) until the trained target question and answer model is obtained.

[0033] By determining the difficult samples in the previous iterative training process corresponding to the first model and adding them to the first preference samples and the second preference samples used for model training of the first model, the difficult samples in the previous iterative training can be fully utilized, effectively reducing the number of new samples required for each round of iteration, improving sample collection efficiency, and helping to improve data utilization. In addition, by adding difficult samples to the first preference samples and the second preference samples used for model training of the first model, the sample imbalance problem can be solved, the accuracy of model training can be improved, and the processing capability and robustness of the trained target question-answering model for complex samples can be improved, thereby improving the accuracy and reliability of the model.

[0034] In a possible implementation, the processing of step S110 may be varied. An optional processing method is provided below. For details, please refer to the processing of the following steps S1102-S1108.

[0035] In step S1102, multiple preference samples in the previous iterative training process corresponding to the first model are determined, and each preference sample is input into the previous iterative training model corresponding to the first model to obtain an output result corresponding to each preference sample.

[0036] In step S1104, the similarity values ​​between the output results corresponding to different preference samples are calculated.

[0037] In implementation, taking the output result as a numerical value as an example, the difference between the two output results in each preference sample of the first preference sample and the second preference sample used for model training of the previous intermediate model of the first model can be calculated.

[0038] In step S1106, the calculated similarity values ​​are sorted, and difficult samples are determined based on the sorting results.

[0039] In implementation, continuing to take the output result as a numerical value as an example, the calculated differences can be sorted in order from high to low, and the difficult samples can be determined based on the sorting results.

[0040] In step S1108 , the difficult samples are added to the first preference samples and the second preference samples used for model training of the first model.

[0041] In implementation, it is possible to For each preferred sample in , calculate the difference between the better output result and the worse output result, and sort the calculated differences in order from high to low, take the first k samples in the sorting as difficult samples, and add them to the first model consisting of multiple first preference samples and second preference samples Corresponding preference sample set , and the sample set is updated. Among them, the similarity between the better output result and the label corresponding to the input data is greater than the similarity between the worse output result and the label corresponding to the input data. That is, the better output result is the output result with a higher similarity between the label corresponding to the input data, and the worse output result is the output result with a lower similarity between the label corresponding to the input data.

[0042] In practice, the formula , calculate the difference between the better output and the worse output The scoring results of the calculated difference are selected, and the top k samples with the highest scores are selected as difficult samples and added to the preferred sample set. Get the preferred sample set , based on the preference sample set The first model is trained continuously using the preset loss function to obtain the trained target question-answering model.

[0043] In the embodiment of this specification, the second model in step S104 may be a target question-answering model, and before step S102, the processing of steps S112-S114 is also included.

[0044] In step S112, the input data related to the question-answering business is input into the target question-answering model and the auxiliary model respectively, two output results corresponding to the input data are obtained, and an initial preference sample is constructed based on the input data and the two output results.

[0045] The target question answering model is the initial model to be trained, and each initial preference sample in the initial preference sample set can be constructed according to each input data and its corresponding two output results (target question answering model and auxiliary model). Figure 4 shown. Figure 4 middle, This is input data related to the question-answering business. is the target question answering model, As auxiliary model, , are the output results of the target question answering model and the auxiliary model respectively. is the initial preference sample. Figure 4 It can be seen that the sample input data X is input into the auxiliary model Get the output result , input the target question answering model Get the output result , the output of the preset auxiliary model is better than the output of the target question-answering model, forming the initial preference sample ,in, For input data X, the poor output result for , better output results for . Input multiple input data related to the question-answering business to obtain the initial preference sample set .

[0046] In step S114, the target question-answering model is iteratively trained based on the initial preference samples to obtain multiple intermediate models.

[0047] In the implementation, the initial preference sample set is obtained Then, based on the initial preference sample set and the preset loss function, the target question answering model is trained. Perform supervised fine-tuning and preference optimization fine-tuning training to obtain the intermediate model after the first iteration training , and so on for the intermediate model Perform model training, thereby iteratively training the target question-answering model to obtain multiple intermediate models.

[0048] In the embodiment of this specification, the processing of the above step S106 can be varied. An optional processing method is provided below. For details, please refer to the processing of the following steps S1062-S1064.

[0049] In step S1062, a first preference sample is constructed based on each input data and its corresponding first output result and third output result. The input data in the first preference sample is each input data, the better output data is the third output result corresponding to each input data, and the worse output data is the first output result corresponding to each input data.

[0050] In step S1064, a second preference sample is constructed based on each input data and its corresponding first output result and second output result. The input data in the second preference sample is each input data, the better output data is the first output result corresponding to each input data, and the worse output data is the second output result corresponding to each input data.

[0051] The first preference sample and the second preference sample constructed through the above steps S1062-S1064 can further improve the construction efficiency and accuracy of the preference samples, thereby improving the efficiency and accuracy of model training.

[0052] In the embodiment of this specification, the processing of the above step S108 can be varied. An optional processing method is provided below. For details, please refer to the processing of the following steps S1082-S1084.

[0053] In step S1082, supervised fine-tuning training is performed on the first model based on a preset first loss function, input data in the first preference sample and its corresponding third output result, and input data in the second preference sample and its corresponding first output result.

[0054] In step S1084, based on the preset second loss function, the first preference sample and the second preference sample, preference optimization training is performed on the first model to obtain a trained target question-answering model.

[0055] In implementation, supervised fine-tuning training only performs model training on samples consisting of input data and better output results (i.e., input data in the first preference sample and its corresponding third output result and input data in the second preference sample and its corresponding first output result). When performing preference optimization training, model training is performed based on the first preference sample and the second preference sample. Supervised fine-tuning model training and preference optimization training can be performed simultaneously or sequentially, and the embodiments of this specification do not limit this.

[0056] In a feasible implementation, supervised fine-tuning model training and preference optimization training are performed simultaneously. The first loss function is the supervised fine-tuning loss function, which can be a cross entropy loss function. The second loss function is the preference optimization loss function, which can be a direct optimization of the preference loss function or an alignment supervised fine-tuning loss function. The loss function calculation formula is ,in, is the supervised fine-tuning loss function, Optimize the loss function for direct preference, is an adjustable weight parameter, The weight relationship between the first loss function and the second loss function can be determined according to the specific application scenario and adjusted flexibly. Take supervised fine-tuning and preference optimization fine-tuning training as an example, the supervised fine-tuning loss function The calculation formula is , is the set of initialized preference samples corresponding to the target question-answering model, is the target question answering model, and the supervised fine-tuning loss function in the next iteration The calculation formula is , you only need to adjust the preferred sample set and the question-answering model of the current iteration.

[0057] In the embodiments of this specification, the target question-answering model is a pre-trained large language model, and the auxiliary models include a GPT-4 (Generative Pre-trained Transformer 4, a deep learning model based on Transformer) model, an LLaMA-3 (Large Language Model Applications 3, a large language model application) model or a GLM-4 (General Language Model 4, a base large model 4).

[0058] like Figure 5As shown, an embodiment of this specification provides another model training method, and the execution subject of this method can be a terminal device or a server, wherein the terminal device can be a certain terminal device such as a mobile phone, a tablet computer, or a computer device such as a laptop or a desktop computer, or an IoT device (specifically such as a smart watch, a car device, etc.). The server can be an independent server, or a server cluster composed of multiple servers, etc. The server can be a background server in the financial field or the online shopping field, or a background server of an application, etc. In this embodiment, the server is used as an example for detailed description. For the execution process of the terminal device, please refer to the following related content, which will not be repeated here. The method can specifically include the following steps: In step S202, an intermediate model generated during the iterative training of the target question-answering model is obtained.

[0059] The method for obtaining the intermediate model in the embodiment of this specification is the same as step S102 and will not be repeated here.

[0060] In step S204, the first model obtained in the current iteration in the intermediate model and the difficult samples in the previous iterative training process corresponding to the first model are determined, and the difficult samples are added to the samples used for model training of the first model.

[0061] The first model is an intermediate model, and is the intermediate model obtained in the current iteration. A difficult sample is a sample whose output result has a similarity with the label corresponding to the input data related to the question-answering business that is less than a preset similarity, that is, the output result corresponding to the difficult sample has a similarity with the label corresponding to the input data that is less than a preset similarity. If the similarity is determined according to the Euclidean distance or cosine similarity algorithm, the distance between the output result corresponding to the difficult sample and the label corresponding to the input data exceeds a preset distance. The output result is the result obtained by inputting the input data into the previous iterative training model corresponding to the first model.

[0062] In step S206, model training of the first model is continued based on the difficult samples, the samples used for model training of the first model, and the preset loss function to obtain a trained target question-answering model.

[0063] After adding the difficult samples to the samples used to train the first model, an updated training sample is obtained. Based on the updated training sample and the preset loss function, the first model is further trained to obtain the trained first model. This process is repeated to obtain the trained target question-answering model.

[0064] In implementation, the preset loss function may adopt a supervised fine-tuning loss function, such as a cross entropy loss function.

[0065] In the embodiment of this specification, the processing of difficult samples in the previous iterative training process corresponding to the first model determined in the above step S204 can be varied. An optional processing method is provided below, and specific reference can be made to the processing of the following steps S2042-S2044.

[0066] In step S2042, the similarity value between the output result corresponding to each sample and the label corresponding to the input data related to the question-answering business is calculated; In step S2044, the calculated similarity values ​​are sorted, and difficult samples are determined based on the sorting results.

[0067] For other parts not described in detail in the embodiments of this specification, reference can be made to the previous embodiments and will not be repeated here.

[0068] The embodiment of this specification provides a model training method, firstly, obtaining the intermediate model generated in the process of iterative training of the target question-answering model, secondly, determining the first model obtained in the current iteration in the intermediate model, and the difficult samples in the previous iterative training process corresponding to the first model, and adding the difficult samples to the samples used for model training of the first model, and finally continuing the model training of the first model based on the difficult samples, the samples used for model training of the first model and the preset loss function, to obtain the trained target question-answering model. In this way, by determining the difficult samples in the previous iterative training process corresponding to the first model and adding them to the samples used for model training of the first model, the difficult samples in the previous iterative training can be fully utilized, the number of samples required for each round of iteration is effectively reduced, the sample collection efficiency is improved, and it is conducive to improving data utilization. Moreover, in the embodiment of this specification, by adding difficult samples to the samples used for model training of the first model, the sample imbalance problem can be solved, the accuracy of model training can be improved, and the processing ability and robustness of the trained target question-answering model for complex samples can be improved, thereby improving the accuracy and reliability of the model.

[0069] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0070] The above is a model training method provided by one or more embodiments of this specification. Based on the same idea, one or more embodiments of this specification also provide a model training device, such as Figure 6 shown.

[0071] The model training device includes: an intermediate model and auxiliary model acquisition module 310, an output result calculation module 320, a preference sample determination module 330 and a training module 340. The intermediate model and auxiliary model acquisition module 310 acquires the intermediate model generated during the iterative training of the target question-answering model and the auxiliary model used to generate the preferred samples. For the same input data, the output result of the auxiliary model is better than the output result of the intermediate model. The output result calculation module 320 inputs the input data related to the question-answering business into the first model obtained by the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; A preference sample determination module 330 constructs a first preference sample based on the input data, the first output result and the third output result, and constructs a second preference sample based on the input data, the first output result and the second output result; The training module 340 continues to perform model training on the first model based on the first preference sample, the second preference sample and the preset loss function to obtain a trained target question-answering model.

[0072] In an embodiment of the present specification, the model training device also includes: a difficult sample determination module, which determines the difficult samples in the previous iterative training process corresponding to the first model, and adds the difficult samples to the first preference samples and the second preference samples used for model training of the first model. The difficult samples are samples whose similarity between output results within the same preference sample is less than a preset similarity.

[0073] Correspondingly, the training module 340 performs model training on the first model based on the difficult sample, the first preference sample for model training of the first model, the second preference sample and the preset loss function to obtain the trained target question-answering model.

[0074] In the embodiment of this specification, the difficult sample determination module includes: A preference sample and output result determination unit determines a plurality of preference samples in a previous iterative training process corresponding to the first model, inputs each preference sample into the previous iterative training model corresponding to the first model, and obtains an output result corresponding to each preference sample; A similarity calculation unit, which calculates the similarity values ​​between the output results corresponding to different preference samples; A sorting unit, which sorts the calculated similarity values ​​and determines difficult samples based on the sorting results; The sample adding unit adds the difficult sample to the first preference sample and the second preference sample used for model training of the first model.

[0075] In the embodiment of this specification, the second model in the output result calculation module 320 is a target question answering model, and the model training device also includes: The initial preference sample set construction module inputs the input data related to the question-answering business into the target question-answering model and the auxiliary model respectively, obtains two output results corresponding to the input data, and constructs the initial preference sample based on the input data and the two output results; The intermediate model acquisition module iteratively trains the target question-answering model based on the initial preference samples to obtain multiple intermediate models.

[0076] In the embodiment of this specification, the preference sample determination module 330 includes: A first preference sample construction unit, constructing a first preference sample based on each input data and its corresponding first output result and third output result, wherein the input data in the first preference sample is each input data, the better output data is the third output result corresponding to each input data, and the worse output data is the first output result corresponding to each input data; The second preference sample construction unit constructs a second preference sample based on each input data and its corresponding first output result and second output result. The input data in the second preference sample is, for each input data, the better output data is the first output result corresponding to each input data, and the worse output data is the second output result corresponding to each input data.

[0077] In this embodiment of the specification, the training module 340 includes: A supervised fine-tuning training unit performs supervised fine-tuning on the first model based on a preset first loss function, input data in the first preferred sample and its corresponding third output result, and input data in the second preferred sample and its corresponding first output result; The preference optimization unit performs preference optimization training on the first model based on a preset second loss function, a first preference sample, and a second preference sample to obtain a trained target question-answering model.

[0078] In the embodiments of the present specification, the first loss function in the supervised fine-tuning unit is a cross entropy loss function, and the second loss function in the preference optimization unit is a direct preference optimization loss function or an aligned supervised fine-tuning loss function.

[0079] The embodiment of the present specification provides a model training device, which first obtains the intermediate model and the auxiliary model used to generate preference samples during the iterative training of the target question-answering model through the intermediate model and auxiliary model acquisition module. For the same input data, the output result of the auxiliary model is better than the output result of the intermediate model. Secondly, the input data related to the question-answering business is input into the first model obtained in the current iteration of the intermediate model, any second model before the first model, and the auxiliary model by the output result calculation module, respectively, to obtain the first output result corresponding to the first model, the second output result corresponding to the second model, and the third output result corresponding to the auxiliary model, respectively. Then, the preference sample determination module constructs the first preference sample based on the input data, the first output result and the third output result, and constructs the second preference sample based on the input data, the first output result and the second output result. Finally, the training module continues to perform model training on the first model based on the first preference sample, the second preference sample and the preset loss function to obtain the trained target question-answering model. In this way, by inputting each of the multiple input data related to the question-answering business into the first model, the second model, and the auxiliary model respectively, three output results are obtained, and two preference samples are constructed based on the three output results. It is possible to construct two preference samples based on one input data, thereby effectively improving the collection efficiency of preference samples, which is conducive to fully improving the utilization rate of data resources, and further improving the efficiency and accuracy of model training. Moreover, in the embodiment of this specification, the first preference sample and the second preference sample are constructed based on the output results of the current iteration model and any second model, and the preference sample construction method of self-comparison between the new model in the current iteration and the historical intermediate model is adopted. When collecting preference samples, the model and its output results obtained in the previous model iteration process can be fully utilized, which is conducive to improving the utilization rate of data resources, and improving the collection efficiency and collection cost of preference samples. Moreover, due to the full use of the intermediate model, the preference samples used for training are more targeted, which is conducive to improving the accuracy of the target question-answering model obtained by training.

[0080] One or more embodiments of this specification also provide another model training device, such as Figure 7 shown.

[0081] The model training device includes: an intermediate model acquisition module 410, which acquires an intermediate model generated during the iterative training of the target question-answering model; The difficult sample determination module 420 determines the first model obtained in the current iteration in the intermediate model, and the difficult samples in the previous iteration training process corresponding to the first model, and adds the difficult samples to the samples used for model training of the first model. The difficult samples are samples whose output results have a similarity with the labels corresponding to the input data related to the question-answering business that is less than a preset similarity, and the output result is the result obtained by inputting the input data into the previous iteration training model corresponding to the first model; The model training module 430 continues to perform model training on the first model based on the difficult samples, the samples used for model training of the first model, and the preset loss function to obtain a trained target question-answering model.

[0082] In the embodiment of this specification, the difficult sample determination module 420 includes: A similarity value calculation unit, which calculates the similarity value between the output result corresponding to each sample and the label corresponding to the input data related to the question-answering business; The sorting unit sorts the calculated similarity values ​​and determines the difficult samples based on the sorting results.

[0083] The embodiment of the present specification provides a model training device, firstly, the intermediate model generated in the process of iterative training of the target question-answering model is obtained by the intermediate model acquisition module, and secondly, the first model obtained in the current iteration in the intermediate model and the difficult samples in the previous iterative training process corresponding to the first model are determined by the difficult sample determination module, and the difficult samples are added to the samples used for model training of the first model, and finally, the model training module continues to perform model training on the first model based on the difficult samples, the samples used for model training of the first model and the preset loss function, so as to obtain the trained target question-answering model. In this way, by determining the difficult samples in the previous iterative training process corresponding to the first model and adding them to the samples used for model training of the first model, the difficult samples in the previous iterative training can be fully utilized, the number of samples required for each round of iteration is effectively reduced, the sample collection efficiency is improved, and it is conducive to improving data utilization. Moreover, in the embodiment of the present specification, by adding the difficult samples to the samples used for model training of the first model, the sample imbalance problem can be solved, the accuracy of model training can be improved, and the processing ability and robustness of the trained target question-answering model for complex samples can be improved, thereby improving the accuracy and reliability of the model.

[0084] Those skilled in the art should understand that the above-mentioned model training device can be used to implement the model training method described above, and the detailed description thereof should be similar to the description of the method part above, and will not be further described here to avoid tediousness.

[0085] Based on the same idea, one or more embodiments of this specification also provide an electronic device, such as Figure 8 As shown. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 501 and memory 502, and the memory 502 may store one or more storage applications or data. Among them, the memory 502 may be a short-term storage or a persistent storage. The application stored in the memory 502 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 501 can be configured to communicate with the memory 502 to execute a series of computer executable instructions in the memory 502 on the electronic device. The electronic device may also include one or more power supplies 503, one or more wired or wireless network interfaces 504, one or more input and output interfaces 505, and one or more keyboards 506.

[0086] Specifically in this embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the electronic device, and the one or more programs are configured to be executed by one or more processors, including the following computer executable instructions: Obtain an intermediate model generated during iterative training of the target question-answering model and an auxiliary model used to generate preferred samples. For the same input data, the output result of the auxiliary model is better than the output result of the intermediate model. Input data related to the question-answering business are respectively input into the first model obtained by the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, to respectively obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; constructing a first preference sample based on the input data, the first output result, and the third output result, and constructing a second preference sample based on the input data, the first output result, and the second output result; The first model is further trained based on the first preference sample, the second preference sample and the preset loss function to obtain a trained target question-answering model.

[0087] One or more embodiments of the present specification provide a storage medium, the storage medium is used to store computer executable instructions, and the executable instructions implement the following process when executed by a processor: Obtain an intermediate model generated during iterative training of the target question-answering model and an auxiliary model used to generate preferred samples. For the same input data, the output result of the auxiliary model is better than the output result of the intermediate model. Input each of the multiple input data related to the question-answering business into the first model obtained by the current iteration, any second model before the first model, and the auxiliary model in the intermediate model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; Construct a first preference sample based on each input data and its corresponding first output result and third output result, and construct a second preference sample based on each input data and its corresponding first output result and second output result; The first model is further trained based on the first preference sample, the second preference sample and the preset loss function to obtain a trained target question-answering model.

[0088] One or more embodiments of the present specification further provide a computer program product, including a computer program, which implements the following process when executed by a processor: Obtain an intermediate model generated during iterative training of the target question-answering model and an auxiliary model used to generate preferred samples. For the same input data, the output result of the auxiliary model is better than the output result of the intermediate model. Input data related to the question-answering business are respectively input into the first model obtained by the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, to respectively obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; constructing a first preference sample based on the input data, the first output result, and the third output result, and constructing a second preference sample based on the input data, the first output result, and the second output result; The first model is further trained based on the first preference sample, the second preference sample and the preset loss function to obtain a trained target question-answering model.

[0089] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0090] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0091] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0092] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0093] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0094] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] One or more embodiments of this specification are described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0099] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0100] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0102] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0104] The above description is only one or more embodiments of this specification and is not intended to limit this document. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of one or more embodiments of this specification.

Claims

1. A model training method, comprising: Obtaining an intermediate model generated during iterative training of the target question-answering model and an auxiliary model for generating preferred samples, wherein for the same input data, an output result of the auxiliary model is better than an output result of the intermediate model; Inputting the input data related to the question-answering business into the first model obtained by the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; constructing a first preference sample based on the input data, the first output result and the third output result, and constructing a second preference sample based on the input data, the first output result and the second output result; Based on the first preference sample, the second preference sample and the preset loss function, the first model is further trained to obtain a trained target question-answering model.

2. According to the method of claim 1, before continuing to train the first model based on the first preference sample, the second preference sample and a preset loss function to obtain the trained target question-answering model, the method further comprises: Determine a difficult sample in a previous iterative training process corresponding to the first model, and add the difficult sample to a first preference sample and a second preference sample used for model training of the first model, wherein the difficult sample is a sample whose similarity between output results in the same preference sample is less than a preset similarity; The step of continuing to train the first model based on the first preference sample, the second preference sample, and a preset loss function to obtain a trained target question-answering model includes: Based on the difficult sample, the first preference sample used for model training of the first model, the second preference sample and the preset loss function, the first model is trained to obtain a trained target question-answering model.

3. The method according to claim 2, wherein determining the difficult samples in the previous iterative training process corresponding to the first model comprises: Determine a plurality of preference samples in a previous iterative training process corresponding to the first model, input each of the preference samples into the previous iterative training model corresponding to the first model, and obtain an output result corresponding to each preference sample; Calculating similarity values ​​between output results corresponding to different preference samples; The calculated similarity values ​​are sorted, and difficult samples are determined based on the sorting results.

4. The method according to claim 1, wherein the second model is a target question answering model, and the method further comprises: Inputting the input data related to the question-answering business into the target question-answering model and the auxiliary model respectively, obtaining two output results corresponding to the input data, and constructing an initial preference sample based on the input data and the two output results; The target question-answering model is iteratively trained based on the initial preference samples to obtain multiple intermediate models.

5. According to the method of claim 4, the target question-answering model is a pre-trained large language model, and the auxiliary model includes a GPT-4 model, an LLaMA-3 model or a GLM-4.

6. The method according to claim 1, further performing model training on the first model based on the first preference sample, the second preference sample and a preset loss function to obtain a trained target question-answering model, comprising: Based on a preset first loss function, the input data in the first preference sample and the corresponding third output result thereof, and the input data in the second preference sample and the corresponding first output result thereof, the first model is fine-tuned in a supervised manner; Based on a preset second loss function, the first preference sample and the second preference sample, preference optimization training is performed on the first model to obtain a trained target question-answering model.

7. According to the method of claim 6, the first loss function is a cross entropy loss function, and the second loss function is a direct preference optimization loss function or an aligned supervised fine-tuning loss function.

8. A model training method, comprising: Obtain the intermediate model generated during the iterative training of the target question-answering model; Determine a first model obtained in the current iteration of the intermediate model, and a difficult sample in a previous iterative training process corresponding to the first model, and add the difficult sample to samples used for model training of the first model, wherein the difficult sample is a sample whose output result has a similarity with a label corresponding to input data related to the question-and-answer business that is less than a preset similarity, and the output result is a result obtained by inputting the input data into the previous iterative training model corresponding to the first model; Based on the difficult samples, the samples used for model training of the first model, and the preset loss function, the first model is further trained to obtain a trained target question-answering model.

9. A model training device, comprising: An intermediate model and auxiliary model acquisition module is used to acquire an intermediate model generated during iterative training of a target question-answering model and an auxiliary model used to generate a preferred sample. For the same input data, the output result of the auxiliary model is better than the output result of the intermediate model. an output result calculation module, which inputs the input data related to the question-answering business into the first model obtained by the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; a preference sample determination module, which constructs a first preference sample based on the input data, the first output result and the third output result, and constructs a second preference sample based on the input data, the first output result and the second output result; The training module continues to perform model training on the first model based on the first preference sample, the second preference sample and a preset loss function to obtain a trained target question-answering model.

10. An electronic device comprising: processor; as well as a memory arranged to store computer executable instructions which, when executed, cause the processor to: Obtaining an intermediate model generated during iterative training of the target question-answering model and an auxiliary model for generating preferred samples, wherein for the same input data, an output result of the auxiliary model is better than an output result of the intermediate model; Inputting the input data related to the question-answering business into the first model obtained by the current iteration in the intermediate model, any second model before the first model, and the auxiliary model, respectively, to obtain a first output result corresponding to the first model, a second output result corresponding to the second model, and a third output result corresponding to the auxiliary model; constructing a first preference sample based on the input data, the first output result and the third output result, and constructing a second preference sample based on the input data, the first output result and the second output result; Based on the first preference sample, the second preference sample and the preset loss function, the first model is further trained to obtain a trained target question-answering model.

Images