Question and answer large model training and question and answer task processing method and device

Abstract

The invention provides a question and answer large model training and question and answer task processing method and device, and relates to the field of artificial intelligence such as deep learning, large models and natural language processing. The method comprises the following steps: acquiring a basic model to be optimized; obtaining target query requests corresponding to the basic model, wherein the number of the target query requests is greater than 1; executing the following first processing: respectively generating first answers corresponding to the target query requests by utilizing the basic model; respectively obtaining an evaluation result of each first answer; and a target training mode suitable for the training is determined from the candidate training modes in response to the evaluation result, and the basic model is trained according to the target training mode.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the training of large question-answering models in the fields of deep learning, large models, and natural language processing, as well as question-answering task processing methods and devices. Background Art

[0002] The large model refers to the Large Language Model (LLM), which is a deep learning model trained with a large amount of text data. It can generate natural language text or understand the meaning of language text, etc. At present, when training a large model, a specified training method is usually used directly, such as supervised fine-tuning (SFT) training and direct preference optimization (DPO) training for the large model in sequence. Summary of the invention

[0003] The present disclosure provides a method and device for training a large question-answering model and processing question-answering tasks.

[0004] A training method for a large question-answering model, comprising:

[0005] Get the basic model to be optimized;

[0006] Obtaining target query requests corresponding to the basic model, where the number of the target query requests is greater than 1;

[0007] Perform the following first processing: use the basic model to generate a first answer corresponding to each target query request; obtain evaluation results of each first answer; in response to determining a target training method suitable for this training from candidate training methods based on the evaluation results, train the basic model according to the target training method.

[0008] A question-answering task processing method, comprising:

[0009] Get pending query requests;

[0010] The answer corresponding to the query request to be processed is generated by using a large question and answer model. The large question and answer model is obtained by training the basic model to be optimized according to a target training method. The target training method is a target training method suitable for this training determined from candidate training methods according to the evaluation results of each first answer. The first answers are the first answers corresponding to each target query request generated respectively by using the basic model. The number of target query requests is greater than 1.

[0011] A training device for a large question-answering model, comprising: a first acquisition module, a second acquisition module, and an optimization processing model;

[0012] The first acquisition module is used to acquire the basic model to be optimized;

[0013] The second acquisition module is used to acquire target query requests corresponding to the basic model, and the number of the target query requests is greater than 1;

[0014] The optimization processing module is used to perform the following first processing: using the basic model to generate a first answer corresponding to each target query request; obtaining an evaluation result of each first answer; in response to determining a target training method suitable for this training from candidate training methods based on the evaluation results, training the basic model according to the target training method.

[0015] A question-answering task processing device comprises: a third acquisition module and a task processing module;

[0016] The third acquisition module is used to acquire the query request to be processed;

[0017] The task processing module is used to generate answers corresponding to the query requests to be processed using the question-answering big model, the question-answering big model is obtained after training the basic model to be optimized according to the target training method, the target training method is a target training method suitable for this training determined from candidate training methods based on the evaluation results of each first answer, the first answers are the first answers corresponding to each target query request generated respectively using the basic model, and the number of target query requests is greater than 1.

[0018] An electronic device, comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0022] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0023] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the method described above when executed by a processor.

[0024] An intelligent agent, comprising: an input module, a processing module and an output module;

[0025] The input module is used to receive query requests to be processed;

[0026] The processing module is used to determine a target task based on the query request to be processed received by the input module, determine a question-answering big model based on the target task, and obtain an answer corresponding to the query request to be processed by calling the question-answering big model to execute the above-mentioned question-answering task processing method;

[0027] The output module is used to output the answer obtained by the processing module.

[0028] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0030] Figure 1 A flowchart of an embodiment of a training method for a large question-answering model disclosed in the present invention;

[0031] Figure 2 A schematic diagram of the overall implementation process of the training method of the question-answering large model described in the present disclosure;

[0032] Figure 3 A flowchart of an embodiment of the question-answering task processing method disclosed in the present invention;

[0033] Figure 4 It is a schematic diagram of the composition structure of an embodiment 400 of a training device for a large question-answering model described in the present disclosure;

[0034] Figure 5 It is a schematic diagram of the composition structure of the embodiment 500 of the question-answering task processing device described in the present disclosure;

[0035] Figure 6 Schematic diagram of the composition structure of the intelligent agent embodiment 600 described in the present disclosure;

[0036] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0038] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0039] Figure 1 Flow chart of an embodiment of the training method of the question-answering large model described in the present disclosure. Figure 1 As shown, the following specific implementation methods are included.

[0040] In step 101, a basic model to be optimized is obtained.

[0041] In step 102, target query requests corresponding to the basic model are obtained, and the number of target query requests is greater than one.

[0042] In step 103, the following first processing is performed: using the basic model to generate the first answer corresponding to each target query request, and obtaining the evaluation results of each first answer, in response to determining the target training method suitable for this training from the candidate training methods according to the evaluation results, the basic model is trained according to the target training method.

[0043] By adopting the scheme described in the above method embodiment, when the basic model needs to be trained, the target training method suitable for this training can be automatically determined from the candidate training methods based on the evaluation results of the first answer generated by the basic model for each target query request, and then the basic model can be trained according to the target training method. The training method used can be flexibly selected according to the actual needs of the model, thereby improving the targeted nature of the training, and then improving the training efficiency and accuracy of the model.

[0044] In some embodiments of the present disclosure, after obtaining the evaluation results of each first answer respectively, in response to not determining the target training method from the candidate training methods according to the evaluation results, the basic model can be determined as the trained target model. In addition, after training the basic model according to the target training method, the latest model can be determined as the basic model, and the target query request corresponding to the basic model can be obtained, and then the first process can be repeatedly performed. Among them, the candidate training methods can be N, and N is a positive integer greater than 1.

[0045] For ease of expression, it is assumed that the basic model to be processed is the question-answering large model at time t, and the corresponding target query requests can be determined, and the basic model can be used to generate the first answer corresponding to each target query request, and then the evaluation results of each first answer can be obtained respectively. Furthermore, if the target training method is not determined from the candidate training methods according to the evaluation results, the basic model can be determined as the trained target model, that is, the final required target model is obtained, and the processing can be terminated. If the target training method suitable for this training is determined from the candidate training methods according to the evaluation results, then the basic model can be trained according to the target training method. Assuming that the trained model is called the question-answering large model at time t+1, the above-mentioned processing for the basic model can be repeated for the question-answering large model at time t+1.

[0046] It can be seen that the solution described in the present disclosure proposes a self-evolution method of the question-answering large model based on adaptive task (different training methods) scheduling, that is, by introducing an adaptive task scheduling mechanism, the question-answering large model can be automatically updated / trained at different time nodes according to the needs of the current task, without the need for human intervention, such as manual specification of training methods, etc., thereby saving manpower and time costs, and improving the targeted nature of training, thereby improving the training efficiency of the model, etc.

[0047] After obtaining the target model, it can be used for actual reasoning applications, such as using the target model to generate corresponding answers to input queries.

[0048] In practical applications, a set of training data may be obtained for the basic model, and the specific number may be determined according to actual needs. Each piece of training data may include: a target query request and a standard answer (response) corresponding to the target query request.

[0049] In the intelligent question-answering scenario, the target query request refers to the question to be answered, and the answer refers to the answer corresponding to the question. Accordingly, the target model trained in the manner described in the present disclosure can be used as an intelligent question-answering model, and the intelligent question-answering model can be used to generate corresponding answers according to the questions input by the user, and accordingly, the accuracy of the generated answers can be improved.

[0050] The above takes the intelligent question-answering scenario as an example, but the solution disclosed in the present invention is not only applicable to the intelligent question-answering scenario, but also to various other scenarios such as information retrieval, data analysis, and recommendation systems, that is, it has wide applicability.

[0051] After generating corresponding first answers for each target query request in the training data, evaluation results of each first answer can be obtained. In some embodiments of the present disclosure, a reward model can be used to generate evaluation results of each first answer.

[0052] The evaluation result usually refers to a reward score, or a preference score, etc. In the traditional processing method, the reward model is only used during DPO training to generate the reward score for updating the model parameters. In the scheme disclosed in the present invention, after using the basic model to generate the first answer corresponding to each target query request, the reward model can be used to generate the evaluation results of each first answer, so as to determine the target training method suitable for this training from N candidate training methods based on the evaluation results, thereby expanding the application scope of the reward model. Moreover, with the help of the evaluation results generated by the reward model, the required target training method can be determined quickly and efficiently, thus laying a good foundation for subsequent model training.

[0053] In addition, in some embodiments of the present disclosure, after obtaining the evaluation results of each first answer respectively, the evaluation results can also be output, and a correction result input for the evaluation results can be obtained, and then a training data set can be generated at least based on the correction result, and the training data set can be used to train the reward model.

[0054] In some embodiments of the present disclosure, in response to determining that there are uncorrected initial results in the evaluation results, a training data set may be generated according to the corrected results and the initial results.

[0055] That is to say, each time after the reward model is used to generate an evaluation result, the evaluation result that is considered inaccurate can be manually corrected. For example, assuming that the number of evaluation results is 30 (the number is only used for example and has no practical significance), and 10 of the evaluation results are manually corrected, then the 10 manually corrected evaluation results and the other 20 uncorrected evaluation results can be combined to construct a training data set corresponding to the reward model, and the training data set can be used to train the reward model. For example, the reward model can be trained in combination with the constructed training data set and the original training data set, thereby achieving continuous optimization of the reward model, and thereby improving the accuracy of subsequently generated evaluation results.

[0056] In some embodiments of the present disclosure, determining a target training method suitable for this training from candidate training methods based on the evaluation results may include: generating target weight parameters corresponding to each candidate training method according to the evaluation results of each first answer, respectively, according to the weight parameter generation method corresponding to each candidate training method; in response to determining that any target weight parameter is greater than a corresponding parameter threshold, determining the candidate training method corresponding to the target weight parameter greater than the parameter threshold as the target training method.

[0057] In some embodiments of the present disclosure, the candidate training methods may include any one of the following: prompting training, SFT training, and DPO training.

[0058] Different candidate training methods may correspond to different weight parameter determination methods, that is, for different candidate training methods, corresponding target weight parameters may be generated by using weight parameter determination methods that match them, thereby improving the accuracy of the generated target weight parameters.

[0059] For example, for Prompting training, the difference between the maximum and minimum values ​​in each evaluation result can be obtained, and the difference can be used as the target weight parameter corresponding to the Prompting training. For SFT training, the mean of each evaluation result can be obtained, and the mean can be used as the target weight parameter corresponding to the SFT training. For DPO training, the absolute value of the difference between each two different evaluation results can be obtained respectively, and the mean of the absolute values ​​of each difference can be obtained, and the mean can be used as the target weight parameter corresponding to the DPO training.

[0060] The above-mentioned target weight parameter determination method is only an example for illustration and is not intended to limit the technical solution of the present disclosure. If other feasible target weight parameter determination methods are adopted, it is also possible.

[0061] In addition, assuming that the parameter threshold corresponding to Prompting training is the first threshold, the parameter threshold corresponding to SFT training is the second threshold, and the parameter threshold corresponding to DPO training is the third threshold, the specific value of each parameter threshold can be determined according to actual needs. Then the target weight parameter corresponding to Prompting training can be compared with the first threshold. If it is determined that the target weight parameter corresponding to Prompting training is greater than the first threshold, then Prompting training can be determined as the target training method. In addition, the weight parameter corresponding to SFT training can be compared with the second threshold. If it is determined that the target weight parameter corresponding to SFT training is greater than the second threshold, then SFT training can be determined as the target training method. Furthermore, the target weight parameter corresponding to DPO training can be compared with the third threshold. If it is determined that the target weight parameter corresponding to DPO training is greater than the third threshold, then DPO training can be determined as the target training method.

[0062] It can be seen that after the above processing, the determined target training method may be 1, multiple, or 0. If it is 0, it means that the target training method has not been determined from the candidate training methods. Accordingly, the current basic model can be determined as the trained target model and the processing is terminated.

[0063] If the determined target training method is not 0, the basic model can be trained according to the target training method.

[0064] In some embodiments of the present disclosure, in response to determining that the number of target training methods is greater than 1, each target training method may be used in sequence to train the basic model in descending order of priority of the target training method.

[0065] Different candidate training methods correspond to different priorities. In addition, the priorities corresponding to different candidate training methods can be determined according to actual needs. For example, the priority of prompting training can be set higher than that of SFT training, and the priority of SFT training can be set higher than that of DPO training.

[0066] In this way, assuming that the determined target training method includes Prompting training, SFT training and DPO training, that is, all candidate training methods are determined as target training methods, then each target training method can be executed in sequence according to the order of Prompting training, SFT training and DPO training. In this way, assuming that the determined target training method includes SFT training and DPO training, then each target training method can be executed in sequence according to the order of SFT training and DPO training.

[0067] By adopting the above processing method, when the number of target training methods is greater than 1, the execution order of each target training method can be reasonably set, thereby improving the execution efficiency of each target training method and correspondingly improving the evolution speed of the model.

[0068] The specific implementation of different target training methods is explained below.

[0069] 1) Prompting training

[0070] In some embodiments of the present disclosure, in response to determining that the target training method is Prompting training, the latest model can be determined as the model to be processed, and the model to be processed can be triggered to update the prompt word (Prompt), and the model to be processed can be triggered to regenerate an answer based on the updated Prompt to obtain a second answer corresponding to each target query request, and then the model parameters of the model to be processed can be updated according to each second answer.

[0071] The latest model may be the basic model or the model trained on the basic model. For example, if the target training method only includes Prompting training, or the target training method also includes other target training methods besides Prompting training, but Prompting training has the highest priority, then the latest model is the basic model. If the target training method also includes other target training methods besides Prompting training, and Prompting training does not have the highest priority, then the latest model is the model trained according to the target training method with a higher priority than Prompting training.

[0072] After the model to be processed updates the Prompt, the corresponding answer can be regenerated according to the updated Prompt and each target query request, that is, the second answer corresponding to each target query request is obtained, wherein, for any target query request, the corresponding first answer and the second answer may be the same or different. As mentioned above, the training data where the target query request is located may also include the corresponding standard answer. Then, further, the model parameters of the model to be processed can be updated in combination with the second answer and the standard answer corresponding to each target query request, that is, the SFT update of the model to be processed can be performed.

[0073] 2) SFT training

[0074] In some embodiments of the present disclosure, in response to determining that the target training method is SFT training, the latest model obtained can be determined as the model to be processed, and the model to be processed can be guided to regenerate answers based on the evaluation results of each first answer to obtain a third answer corresponding to each target query request, and then the model parameters of the model to be processed can be updated based on each third answer.

[0075] Among them, the model to be processed can be guided to generate the third answer corresponding to each target query request according to the evaluation results of each first answer. For example, the model to be processed can be guided to focus on the first answer with a lower evaluation result. After obtaining the third answer corresponding to each target query request, the model parameters of the model to be processed can be updated in combination with the third answer corresponding to each target query request and the standard answer.

[0076] From the above introduction, it can be seen that when performing Prompting training and SFT training, by regenerating answers, it is equivalent to automatically constructing new training data, and the model can be optimized and trained based on the new training data, thereby improving the model's evolution speed and overall performance.

[0077] 3) DPO training

[0078] In response to determining that the target training method is DPO training, the latest model can be determined as the model to be processed, and the model parameters of the model to be processed can be updated according to the evaluation results of each first answer.

[0079] If the target training method is DPO training, the traditional DPO training method can be used to update the model parameters of the model to be processed according to the evaluation results of each first answer.

[0080] After the basic model is trained using each target training method, the latest model can be determined as the basic model, and the relevant processing performed on the basic model in the scheme described in the present disclosure can be repeated. Among them, for the newly obtained basic model (updated basic model), its corresponding target query request can be the same as the target query request corresponding to the basic model before the update, or it can be different, depending on actual needs.

[0081] Combined with the above introduction, Figure 2 Schematic diagram of the overall implementation process of the training method of the question-answering large model described in the present disclosure. Figure 2 As shown, for the basic model, the basic model can be first used to generate the first answer corresponding to each target query request, and the reward model can be used to generate the evaluation results of each first answer. Then, the target training method suitable for this training can be determined from the candidate training methods according to the evaluation results of each first answer. For example, according to the evaluation results of each first answer, the target weight parameters corresponding to each candidate training method can be generated according to the weight parameter generation method corresponding to each candidate training method. In response to determining that the target weight parameter corresponding to any candidate training method is greater than the parameter threshold corresponding to the candidate training method, the candidate training method can be determined as the target training method. Further, the basic model can be optimized and trained according to the target training method, and the latest model can be determined as the basic model. Then, the above process can be repeated.

[0082] In summary, by adopting the scheme described in the present invention, the target training method to be adopted in the next step can be automatically judged and selected with the help of the reward model, that is, the optimal path can be automatically judged, and the model can be optimized and trained according to the target training method. After that, the optimal path judgment and the process of optimizing the model training can be repeated, thereby forming a closed-loop data flow processing, so that the model can adaptively perform multi-stage optimization, thereby improving the training efficiency of the model and the learning effect of the model, and reducing resource waste through automatic path selection, saving computing time and space, etc.

[0083] The above is an explanation of the training method of the question-answering large model. The following is a further explanation of the application method of the question-answering large model.

[0084] Accordingly, Figure 3Flow chart of an embodiment of the question-answering task processing method described in the present disclosure. Figure 3 As shown, the following specific implementation methods are included.

[0085] In step 301, a query request to be processed is obtained.

[0086] In step 302, the question-answering big model is used to generate answers corresponding to the query requests to be processed. The question-answering big model is obtained by training the basic model to be optimized according to the target training method. The target training method is a target training method suitable for this training determined from candidate training methods based on the evaluation results of each first answer. Each first answer is the first answer corresponding to each target query request generated by the basic model respectively. The number of target query requests is greater than 1.

[0087] In the intelligent question-and-answer scenario, the query request to be processed refers to the question to be answered. The question-and-answer big model trained in the manner described in the present invention can be used to generate the answer corresponding to the question. Accordingly, the accuracy of the generated answer can be improved.

[0088] The question may be a question in text form directly input by the user, or may be a question in text form obtained by converting the voice input by the user. In addition, in practical applications, the question may include content in other forms such as pictures in addition to text content, depending on actual needs. Similarly, the answer to the question is usually a text answer, and may also include content in other forms such as pictures if necessary.

[0089] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure. In addition, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0090] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0091] Figure 4 FIG. 4 is a schematic diagram of the structure of the training device embodiment 400 of the question-answering large model described in the present disclosure. Figure 4 As shown, it includes: a first acquisition module 401, a second acquisition module 402 and an optimization processing model 403.

[0092] The first acquisition module 401 is used to acquire the basic model to be optimized.

[0093] The second acquisition module 402 is used to acquire target query requests corresponding to the basic model, and the number of target query requests is greater than one.

[0094] The optimization processing module 403 is used to perform the following first processing: using the basic model to generate a first answer corresponding to each target query request, and obtaining an evaluation result of each first answer, in response to determining a target training method suitable for this training from candidate training methods based on the evaluation results, and training the basic model according to the target training method.

[0095] By adopting the scheme described in the above-mentioned device embodiment, when the basic model needs to be trained, the target training method suitable for this training can be automatically determined from the candidate training methods based on the evaluation results of the first answer generated by the basic model for each target query request, and then the basic model can be trained according to the target training method. The training method to be used can be flexibly selected according to the actual needs of the model, thereby improving the targeted nature of the training, and then improving the training efficiency and accuracy of the model.

[0096] In some embodiments of the present disclosure, after the optimization processing module 403 obtains the evaluation results of each first answer respectively, in response to not determining the target training method from the candidate training methods based on the evaluation results, the basic model can be determined as the trained target model. In addition, after the basic model is trained according to the target training method, the latest model can be determined as the basic model, and the target query request corresponding to the basic model can be obtained, and then the first processing can be repeated.

[0097] For ease of expression, it is assumed that the basic model to be processed is the question-answering large model at time t, and the corresponding target query requests can be determined, and the basic model can be used to generate the first answer corresponding to each target query request, and then the evaluation results of each first answer can be obtained respectively. Furthermore, if the target training method is not determined from the candidate training methods according to the evaluation results, the basic model can be determined as the trained target model, that is, the final required target model is obtained, and the processing can be terminated. If the target training method suitable for this training is determined from the candidate training methods according to the evaluation results, then the basic model can be trained according to the target training method. Assuming that the trained model is called the question-answering large model at time t+1, the above-mentioned processing for the basic model can be repeated for the question-answering large model at time t+1.

[0098] After generating corresponding first answers for each target query request in the training data, the optimization processing module 403 can obtain evaluation results of each first answer. In some embodiments of the present disclosure, the optimization processing module 403 can generate evaluation results of each first answer using a reward model.

[0099] In addition, in some embodiments of the present disclosure, after respectively obtaining the evaluation results of each first answer, the optimization processing module 403 can also output the evaluation results, and can obtain correction results input for the evaluation results, at least generate a training data set based on the correction results, and use the training data set to train the reward model.

[0100] In some embodiments of the present disclosure, in response to determining that there are uncorrected initial results in the evaluation results, the optimization processing module 403 may also generate a training data set according to the corrected results and the initial results.

[0101] In some embodiments of the present disclosure, the optimization processing module 403 determines the target training method suitable for this training from the candidate training methods based on the evaluation results, which may include: generating target weight parameters corresponding to each candidate training method according to the evaluation results of each first answer, respectively according to the weight parameter generation method corresponding to each candidate training method; in response to determining that any target weight parameter is greater than the corresponding parameter threshold, determining the candidate training method corresponding to the target weight parameter greater than the parameter threshold as the target training method.

[0102] The determined target training method may be 1, multiple, or 0. If it is 0, it means that the target training method has not been determined from the candidate training methods. Accordingly, the current basic model can be determined as the trained target model. If the determined target training method is not 0, the basic model can be trained according to the target training method.

[0103] In some embodiments of the present disclosure, in response to determining that the number of target training methods is greater than 1, the optimization processing module 403 may train the basic model using each target training method in order of priority from high to low, and different candidate training methods correspond to different priorities.

[0104] In some embodiments of the present disclosure, the candidate training methods may include any one of the following: Prompting training, SFT training, and DPO training.

[0105] In some embodiments of the present disclosure, in response to determining that the target training method is Prompting training, the optimization processing module 403 may determine the latest model as the model to be processed, and may trigger the model to be processed to perform a Prompt update, and may trigger the model to be processed to regenerate an answer based on the updated Prompt to obtain a second answer corresponding to each target query request, and then update the model parameters of the model to be processed based on each second answer.

[0106] In addition, in some embodiments of the present disclosure, the optimization processing module 403 may determine the latest model as the model to be processed in response to determining that the target training method is SFT training, and may guide the model to be processed to regenerate answers based on the evaluation results of each first answer, to obtain a third answer corresponding to each target query request, and then update the model parameters of the model to be processed based on each third answer.

[0107] Furthermore, in some embodiments of the present disclosure, in response to determining that the target training method is DPO training, the optimization processing module 403 may determine the latest model as the model to be processed, and may update the model parameters of the model to be processed according to the evaluation results of each first answer.

[0108] After the basic model is trained using various target training methods, the latest model can be determined as the basic model, and the relevant processing performed on the basic model in the scheme described in the present disclosure can be repeated.

[0109] Figure 5 FIG. 5 is a schematic diagram of the structure of the question-answering task processing device embodiment 500 described in the present disclosure. Figure 5 As shown, it includes: a third acquisition module 501 and a task processing module 502.

[0110] The third acquisition module 501 is used to acquire the query request to be processed.

[0111] The task processing module 502 is used to generate answers corresponding to the query requests to be processed using the question-answering big model. The question-answering big model is obtained by training the basic model to be optimized according to the target training method. The target training method is a target training method suitable for this training determined from the candidate training methods based on the evaluation results of each first answer. Each first answer is the first answer corresponding to each target query request generated respectively using the basic model. The number of target query requests is greater than 1.

[0112] Figure 6 FIG. 6 is a schematic diagram of the composition structure of the intelligent agent embodiment 600 described in the present disclosure. Figure 6 As shown, it includes: an input module 601, a processing module 602 and an output module 603.

[0113] The input module 601 is used to receive a query request to be processed.

[0114] The processing module 602 is used to determine the target task based on the query request to be processed received by the input module 601, determine the question-answering model based on the target task, and execute the question-answering model by calling the question-answering model. Figure 3 The question-answering task processing method shown obtains the answer corresponding to the query request to be processed.

[0115] The output module 603 is used to output the answer obtained by the processing module 602.

[0116] The specific working processes of the above-mentioned device embodiments can refer to the relevant descriptions in the above-mentioned method embodiments and will not be repeated here.

[0117] The scheme disclosed in the present invention can be applied to the field of artificial intelligence, especially to the fields of deep learning, large models, and natural language processing. Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0118] The target query request and answer in the embodiments described in this disclosure are not targeted at a specific user and do not reflect the personal information of a specific user. In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0119] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0120] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0121] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0122] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the methods described in the present disclosure may be executed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).

[0124] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0126] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0128] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0129] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0131] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A training method for a large question-answering model, comprising: Get the basic model to be optimized; Obtaining target query requests corresponding to the basic model, where the number of the target query requests is greater than 1; Perform the following first processing: using the basic model to generate first answers corresponding to each target query request; Obtaining evaluation results of each of the first answers respectively; In response to determining a target training method suitable for this training from candidate training methods according to the evaluation result, the basic model is trained according to the target training method.

2. The method according to claim 1, after respectively obtaining the evaluation results of each of the first answers, the method further comprises: In response to the target training method not being determined from the candidate training methods according to the evaluation result, the basic model is determined as the trained target model.

3. The method according to claim 1 or 2, wherein: The obtaining of evaluation results of the first answers respectively includes: The evaluation results of each of the first answers are generated respectively by using a reward model.

4. The method according to claim 3, after respectively obtaining the evaluation results of each of the first answers, the method further comprises: outputting the evaluation result; Obtaining a correction result input for the evaluation result; generating a training data set at least according to the correction result; The reward model is trained using the training data set.

5. The method according to claim 4, wherein generating a training data set at least according to the correction result comprises: In response to determining that there are uncorrected initial results in the evaluation results, the training data set is generated according to the corrected results and the initial results.

6. The method according to claim 1 or 2, wherein: The method further comprises: According to the evaluation results, generating target weight parameters corresponding to the candidate training methods according to the weight parameter generation methods corresponding to the candidate training methods; In response to determining that any of the target weight parameters is greater than a corresponding parameter threshold, the candidate training method corresponding to the target weight parameter greater than the parameter threshold is determined as the target training method.

7. The method according to claim 1 or 2, wherein: The training of the basic model according to the target training method includes: In response to determining that the number of the target training methods is greater than 1, the basic model is trained in sequence using each of the target training methods in descending order of priority of the target training methods.

8. The method according to claim 7, wherein: The candidate training methods include any one of the following: prompt word optimization training, supervised fine-tuning training, and direct preference optimization training.

9. The method according to claim 8, wherein: In response to determining that the target training mode is the prompt word optimization training, the training of the basic model according to the target training mode includes: Determine the latest model as the model to be processed; Triggering the model to be processed to update the prompt word, and triggering the model to be processed to regenerate an answer according to the updated prompt word, so as to obtain a second answer corresponding to each target query request; The model parameters of the model to be processed are updated according to each second answer.

10. The method according to claim 8, wherein: In response to determining that the target training mode is the supervised fine-tuning training, training the basic model according to the target training mode includes: Determine the latest model as the model to be processed; Instructing the model to be processed to regenerate answers according to the evaluation results of each first answer, to obtain a third answer corresponding to each target query request; The model parameters of the model to be processed are updated according to each third answer.

11. The method according to claim 8, wherein: In response to determining that the target training mode is the direct preference optimization training, the training of the basic model according to the target training mode includes: The latest model is determined as the model to be processed, and model parameters of the model to be processed are updated according to the evaluation results of each first answer.

12. A method for processing a question-answering task, comprising: Get pending query requests; The answer corresponding to the query request to be processed is generated by using a large question and answer model. The large question and answer model is obtained by training the basic model to be optimized according to a target training method. The target training method is a target training method suitable for this training determined from candidate training methods according to the evaluation results of each first answer. The first answers are the first answers corresponding to each target query request generated respectively by using the basic model. The number of target query requests is greater than 1.

13. A training device for a large question-answering model, comprising: A first acquisition module, a second acquisition module and an optimization processing model; The first acquisition module is used to acquire the basic model to be optimized; The second acquisition module is used to acquire target query requests corresponding to the basic model, and the number of the target query requests is greater than 1; The optimization processing module is used to perform the following first processing: using the basic model to generate first answers corresponding to each target query request; and obtaining evaluation results of each first answer; In response to determining a target training method suitable for this training from candidate training methods according to the evaluation result, the basic model is trained according to the target training method.

14. The device according to claim 13, wherein: The optimization processing module is further used to, after respectively obtaining the evaluation results of each of the first answers, determine the basic model as the trained target model in response to not determining the target training method from the candidate training methods according to the evaluation results.

15. The device according to claim 13 or 14, wherein: The optimization processing module generates the evaluation results of each of the first answers respectively using a reward model.

16. The device according to claim 15, wherein: The optimization processing module is further used to, after obtaining the evaluation results of each of the first answers respectively, output the evaluation results, obtain the correction results input for the evaluation results, generate a training data set at least based on the correction results, and use the training data set to train the reward model.

17. The device according to claim 16, wherein: In response to determining that there are uncorrected initial results in the evaluation results, the optimization processing module generates the training data set according to the corrected results and the initial results.

18. The device according to claim 13 or 14, wherein: The optimization processing module generates target weight parameters corresponding to each of the candidate training methods according to the evaluation results and the weight parameter generation method corresponding to each of the candidate training methods. In response to determining that any of the target weight parameters is greater than the corresponding parameter threshold, the candidate training method corresponding to the target weight parameter greater than the parameter threshold is determined as the target training method.

19. The device according to claim 13 or 14, wherein: In response to determining that the number of the target training methods is greater than 1, the optimization processing module sequentially uses each of the target training methods to train the basic model in descending order of priority of the target training methods.

20. The device according to claim 19, wherein The candidate training methods include any one of the following: prompt word optimization training, supervised fine-tuning training, and direct preference optimization training.

21. The device according to claim 20, wherein: In response to determining that the target training method is the prompt word optimization training, the optimization processing module determines the latest model as the model to be processed, triggers the model to be processed to update the prompt words, and triggers the model to be processed to regenerate answers according to the updated prompt words, obtains the second answer corresponding to each target query request, and updates the model parameters of the model to be processed according to each second answer.

22. The device according to claim 20, wherein: In response to determining that the target training method is the supervised fine-tuning training, the optimization processing module determines the latest model as the model to be processed, guides the model to be processed to regenerate answers based on the evaluation results of each first answer, obtains a third answer corresponding to each target query request, and updates the model parameters of the model to be processed according to each third answer.

23. The device according to claim 20, wherein: In response to determining that the target training method is the direct preference optimization training, the optimization processing module determines the latest model as the model to be processed, and updates the model parameters of the model to be processed according to the evaluation results of each first answer.

24. A question-answering task processing device, comprising: The third acquisition module and task processing module; The third acquisition module is used to acquire the query request to be processed; The task processing module is used to generate answers corresponding to the query requests to be processed using the question-answering big model, the question-answering big model is obtained after training the basic model to be optimized according to the target training method, the target training method is a target training method suitable for this training determined from candidate training methods based on the evaluation results of each first answer, the first answers are the first answers corresponding to each target query request generated respectively using the basic model, and the number of target query requests is greater than 1.

25. An intelligent agent, comprising: Input module, processing module and output module; The input module is used to receive query requests to be processed; The processing module is used to determine a target task based on the query request to be processed received by the input module, determine a question-answering big model based on the target task, and obtain an answer corresponding to the query request to be processed by calling the question-answering big model to execute the question-answering task processing method of claim 12; The output module is used to output the answer obtained by the processing module.

26. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

27. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 12.

28. A computer program product, comprising a computer program / instructions, which implement the method according to any one of claims 1 to 12 when executed by a processor.

Images