Training of a question and answer large model and question and answer task processing method and device

Abstract

The present disclosure provides a method and device for training a large question and answer model and processing a question and answer task, and relates to the fields of deep learning, large models, and natural language processing. The method can include: obtaining a basic model to be optimized; obtaining target query requests corresponding to the basic model, the number of target query requests being greater than 1; performing the following first processing: generating a first answer corresponding to each target query request using the basic model; obtaining an evaluation result of each first answer; and in response to determining a target training method suitable for the current training from candidate training methods according to the evaluation result, training the basic model according to the target training method.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to methods and apparatus for training large question-answering models and processing question-answering tasks in the fields of deep learning, large models, and natural language processing. Background Technology

[0002] Large models, specifically large language models (LLMs), are deep learning models trained on large amounts of text data that can generate natural language text or understand the meaning of language text. Currently, when training large models, pre-defined training methods are typically used, such as sequentially performing supervised fine-tuning (SFT) and direct preference optimization (DPO) training. Summary of the Invention

[0003] This disclosure provides methods and apparatus for training large question-answering models and processing question-answering tasks.

[0004] A method for training a large question-answering model includes:

[0005] Obtain the base model to be optimized;

[0006] Obtain the target query request corresponding to the base model, where the number of target query requests is greater than 1;

[0007] Perform the following first process: generate a first answer corresponding to each target query request using the base model; obtain the evaluation result of each first answer; in response to the evaluation result, determine the target training method suitable for this training from the candidate training methods, and train the base model according to the target training method.

[0008] A question-answering task processing method, comprising:

[0009] Get pending query requests;

[0010] The question-answering model is used to generate the answer corresponding to the query request to be processed. The question-answering model is obtained by training the basic model to be optimized according to the target training method. The target training method is the target training method that is suitable for this training, which is determined from the candidate training methods based on the evaluation results of each first answer. Each first answer is the first answer corresponding to each target query request generated by the basic model. The number of target query requests is greater than 1.

[0011] A training device for a large question-answering model includes: a first acquisition module, a second acquisition module, and an optimization processing model;

[0012] The first acquisition module is used to acquire the basic model to be optimized;

[0013] The second acquisition module is used to acquire the target query requests corresponding to the basic model, wherein the number of target query requests is greater than 1;

[0014] The optimization processing module is configured to perform the following first processing: generate a first answer corresponding to each target query request using the base model; obtain the evaluation result of each first answer; and, in response to the evaluation result, determine a target training method suitable for this training from the candidate training methods, and train the base model according to the target training method.

[0015] A question-and-answer task processing device includes: a third acquisition module and a task processing module;

[0016] The third acquisition module is used to acquire query requests to be processed;

[0017] The task processing module is used to generate answers corresponding to the query requests to be processed using a large question-answering model. The large question-answering model is obtained by training the basic model to be optimized according to the target training method. The target training method is the target training method suitable for this training determined from the candidate training methods based on the evaluation results of each first answer. Each first answer is the first answer corresponding to each target query request generated by the basic model. The number of target query requests is greater than 1.

[0018] An electronic device, comprising:

[0019] At least one processor; and

[0020] A memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0022] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0023] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0024] An intelligent agent includes: an input module, a processing module, and an output module;

[0025] The input module is used to receive query requests to be processed;

[0026] The processing module is used to determine the target task based on the query request to be processed received by the input module, determine the question-and-answer big model based on the target task, and obtain the answer corresponding to the query request to be processed by calling the question-and-answer big model to execute the question-and-answer task processing method described above.

[0027] The output module is used to output the answer obtained by the processing module.

[0028] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0029] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0030] Figure 1 This is a flowchart illustrating an embodiment of the training method for the large question-answering model described in this disclosure;

[0031] Figure 2 This is a schematic diagram illustrating the overall implementation process of the training method for the large question-answering model described in this disclosure;

[0032] Figure 3 This is a flowchart of an embodiment of the question-answering task processing method described in this disclosure;

[0033] Figure 4 This is a schematic diagram of the composition structure of embodiment 400 of the training device for the large question-answering model described in this disclosure;

[0034] Figure 5 This is a schematic diagram of the structural composition of Embodiment 500 of the question-answering task processing device described in this disclosure;

[0035] Figure 6 This is a schematic diagram of the composition structure of the intelligent agent embodiment 600 described in this disclosure;

[0036] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0037] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0038] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0039] Figure 1 This is a flowchart illustrating an embodiment of the training method for the large question-answering model described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0040] In step 101, the basic model to be optimized is obtained.

[0041] In step 102, the target query requests corresponding to the basic model are obtained, and the number of target query requests is greater than 1.

[0042] In step 103, the following first processing is performed: the base model is used to generate the first answer corresponding to each target query request, and the evaluation result of each first answer is obtained. In response, the target training method suitable for this training is determined from the candidate training methods according to the evaluation result, and the base model is trained according to the target training method.

[0043] Using the scheme described in the above method embodiments, when it is necessary to train the basic model, the target training method suitable for this training can be automatically determined from the candidate training methods based on the evaluation results of the first answer generated by the basic model for each target query request. Then, the basic model can be trained according to the target training method. This allows for flexible selection of the training method according to the actual needs of the model, thereby improving the targeting of the training, and thus improving the training efficiency and accuracy of the model.

[0044] In some embodiments of this disclosure, after obtaining the evaluation results of each first answer, in response to the failure to determine the target training method from the candidate training methods based on the evaluation results, the base model can be determined as the trained target model. Furthermore, after training the base model according to the target training method, the latest obtained model can be determined as the base model, and the target query request corresponding to the base model can be obtained. Then, the first process can be repeated. The number of candidate training methods can be N, where N is a positive integer greater than 1.

[0045] For ease of explanation, assume the base model to be processed is the question-and-answer large model at time t. The corresponding target query requests can be identified, and the base model can be used to generate the first answer for each target query request. Then, the evaluation results of each first answer can be obtained. Further, if the target training method is not determined from the candidate training methods based on the evaluation results, the base model can be determined as the trained target model, thus obtaining the final target model, and the processing can end. If the target training method suitable for this training is determined from the candidate training methods based on the evaluation results, then the base model can be trained according to the target training method. Assuming the trained model is called the question-and-answer large model at time t+1, the above processing for the base model can be repeated for the question-and-answer large model at time t+1.

[0046] As can be seen, the scheme described in this disclosure proposes a self-evolutionary approach for a large question-answering model based on adaptive task (different training methods) scheduling. That is, by introducing an adaptive task scheduling mechanism, the large question-answering model can automatically update / train at different time nodes according to the needs of the current task, without manual intervention, such as manually specifying the training method, thereby saving manpower and time costs, improving the relevance of training, and thus improving the training efficiency of the model.

[0047] Once the target model is obtained, it can be used for practical reasoning applications, such as generating corresponding answers based on input queries.

[0048] In practical applications, a set of training data can be obtained for the basic model. The specific amount can be determined according to actual needs. Each training data can include: the target query request and the standard answer (response) corresponding to the target query request.

[0049] In the context of intelligent question answering, the target query request refers to the question to be answered, and the answer refers to the response to the question. Accordingly, the target model trained in accordance with the method described in this disclosure can be used as an intelligent question answering model. The intelligent question answering model can be used to generate corresponding answers based on the user's input question, thereby improving the accuracy of the generated answers.

[0050] The above example uses intelligent question answering, but the solution described in this disclosure is not only applicable to intelligent question answering, but also to other scenarios such as information retrieval, data analysis, and recommendation systems, thus having broad applicability.

[0051] After generating a corresponding first answer for each target query request in the training data, the evaluation result of each first answer can be obtained separately. In some embodiments of this disclosure, a reward model can be used to generate the evaluation result of each first answer separately.

[0052] The evaluation result typically refers to a reward score, or preference score, etc. In traditional methods, the reward model is only used during DPO training to generate the reward score for updating model parameters. However, in the solution described in this disclosure, after generating the first answer for each target query request using the base model, the reward model can be used to generate evaluation results for each first answer. Based on these evaluation results, the target training method suitable for this training can be determined from N candidate training methods, thus expanding the application scope of the reward model. Furthermore, with the evaluation results generated by the reward model, the required target training method can be quickly and efficiently determined, laying a solid foundation for subsequent model training.

[0053] In addition, in some embodiments of this disclosure, after obtaining the evaluation results of each first answer, the evaluation results can be output, and the correction results input for the evaluation results can be obtained. Then, at least the training dataset can be generated based on the correction results, and the reward model can be trained using the training dataset.

[0054] In some embodiments of this disclosure, in response to determining that there are uncorrected initial results in the evaluation results, a training dataset can also be generated based on the corrected results and the initial results.

[0055] In other words, after each evaluation result is generated using the reward model, inaccurate evaluation results can be manually corrected. For example, assuming there are 30 evaluation results (the number is for illustrative purposes only and has no practical significance), and 10 of them are manually corrected, then the 10 manually corrected evaluation results and the other 20 uncorrected evaluation results can be combined to construct the training dataset corresponding to the reward model. The reward model can then be trained using this training dataset, such as by combining the constructed training dataset with the original training dataset. This allows for continuous optimization of the reward model, thereby improving the accuracy of subsequent generated evaluation results.

[0056] In some embodiments of this disclosure, determining the target training method suitable for this training from candidate training methods based on the evaluation results may include: generating target weight parameters corresponding to each candidate training method according to the evaluation results of each first answer and the weight parameter generation method corresponding to each candidate training method; and in response to determining that any target weight parameter is greater than the corresponding parameter threshold, determining the candidate training method corresponding to the target weight parameter that is greater than the parameter threshold as the target training method.

[0057] In some embodiments of this disclosure, candidate training methods may include any of the following: prompting training, SFT training, and DPO training.

[0058] Different candidate training methods can correspond to different weight parameter determination methods. That is, for different candidate training methods, a matching weight parameter determination method can be used to generate the corresponding target weight parameters, thereby improving the accuracy of the generated target weight parameters.

[0059] For example, for Prompting training, the difference between the maximum and minimum values ​​in each evaluation result can be obtained, and the difference can be used as the target weight parameter for Prompting training. For SFT training, the mean of each evaluation result can be obtained, and the mean can be used as the target weight parameter for SFT training. For DPO training, the absolute value of the difference between each two different evaluation results can be obtained, and the mean of the absolute values ​​of each difference can be obtained, and the mean can be used as the target weight parameter for DPO training.

[0060] The above-described methods for determining target weight parameters are merely illustrative examples and are not intended to limit the technical solutions disclosed herein. Other feasible methods for determining target weight parameters are also acceptable.

[0061] Furthermore, assuming the parameter thresholds for Prompting training are the first threshold, SFT training is the second threshold, and DPO training is the third threshold, the specific values ​​of each threshold can be determined according to actual needs. Then, the target weight parameters for Prompting training can be compared with the first threshold. If the target weight parameters for Prompting training are greater than the first threshold, then Prompting training can be selected as the target training method. Similarly, the weight parameters for SFT training can be compared with the second threshold. If the target weight parameters for SFT training are greater than the second threshold, then SFT training can be selected as the target training method. Finally, the target weight parameters for DPO training can be compared with the third threshold. If the target weight parameters for DPO training are greater than the third threshold, then DPO training can be selected as the target training method.

[0062] As can be seen, after the above processing, the number of target training methods determined may be 1, multiple, or 0. If there are 0, it means that no target training method has been determined from the candidate training methods. Accordingly, the current base model can be determined as the target model after training, and the processing ends.

[0063] If there are multiple target training methods, the basic model can be trained according to the target training methods.

[0064] In some embodiments of this disclosure, in response to the determination that the number of target training methods is greater than 1, the base model can be trained sequentially using each target training method in descending order of priority.

[0065] Different candidate training methods correspond to different priorities. Furthermore, the priorities of different candidate training methods can be determined according to actual needs; for example, Prompting training can be given higher priority than SFT training, and SFT training can be given higher priority than DPO training.

[0066] Thus, assuming the determined target training methods include Prompting training, SFT training, and DPO training, that is, all candidate training methods are determined as target training methods, then each target training method can be executed in the order of Prompting training, SFT training, and DPO training. If the determined target training methods include SFT training and DPO training, then each target training method can be executed in the order of SFT training and DPO training.

[0067] By adopting the above processing method, when the number of target training methods is greater than 1, the execution order of each target training method can be reasonably set, thereby improving the execution efficiency of each target training method and correspondingly improving the evolution speed of the model.

[0068] The specific implementation of different target training methods will be explained below.

[0069] 1) Prompting training

[0070] In some embodiments of this disclosure, in response to determining that the target training method is Prompting training, the latest obtained model can be identified as the model to be processed, and the model to be processed can be triggered to update the prompt word, and the model to be processed can be triggered to regenerate the answer according to the updated prompt to obtain the second answer corresponding to each target query request, and then the model parameters of the model to be processed can be updated according to each second answer.

[0071] The latest model may be the base model or a model trained on the base model. For example, if the target training method only includes Prompting training, or if the target training method includes other target training methods besides Prompting training but Prompting training has the highest priority, then the latest model is the base model. If the target training method includes other target training methods besides Prompting training, and Prompting training does not have the highest priority, then the latest model is a model trained according to the target training method with higher priority than Prompting training.

[0072] After the model to be processed updates the Prompt, it can regenerate the corresponding answers based on the updated Prompt and each target query request, that is, obtain the second answer corresponding to each target query request. For any target query request, the corresponding first answer and second answer may be the same or different. As mentioned earlier, the training data where the target query request is located may also include the corresponding standard answer. Therefore, the model parameters of the model to be processed can be updated by combining the second answer and the standard answer corresponding to each target query request, that is, the model to be processed can be updated by SFT.

[0073] 2) SFT training

[0074] In some embodiments of this disclosure, in response to determining that the target training method is SFT training, the latest obtained model can be identified as the model to be processed, and the model to be processed can be guided to regenerate the answer based on the evaluation results of each first answer to obtain the third answer corresponding to each target query request. Then, the model parameters of the model to be processed can be updated based on each third answer.

[0075] Specifically, the evaluation results of each first answer can guide the generation of third answers for each target query request in the model to be processed. For example, the model can focus on first answers with lower evaluation results. After obtaining the third answers for each target query request, the model parameters can be updated by combining the third answers for each target query request with the standard answer.

[0076] As can be seen from the above introduction, during Prompting training and SFT training, by regenerating the answers, new training data is automatically constructed. The model can then be optimized and trained based on the new training data, thereby improving the model's evolution speed and overall performance.

[0077] 3) DPO training

[0078] In response to the determination that the target training method is DPO training, the latest obtained model can be identified as the model to be processed, and the model parameters of the model to be processed can be updated according to the evaluation results of each first response.

[0079] If the target training method is DPO training, simply follow the traditional DPO training method and update the model parameters of the model to be processed based on the evaluation results of each first answer.

[0080] After training the base model using various target training methods, the latest model can be determined as the base model, and the relevant processing performed on the base model in the scheme described in this disclosure can be repeated. Specifically, the target query request corresponding to the newly obtained base model (the updated base model) can be the same as or different from the target query request corresponding to the base model before the update, depending on actual needs.

[0081] Based on the above introduction, Figure 2 This is a schematic diagram illustrating the overall implementation process of the training method for the large question-answering model described in this disclosure. For example... Figure 2 As shown, for the base model, the first answer corresponding to each target query request can be generated using the base model, and the evaluation result of each first answer can be generated using the reward model. Then, based on the evaluation result of each first answer, the target training method suitable for this training can be determined from each candidate training method. For example, based on the evaluation result of each first answer, the target weight parameters corresponding to each candidate training method can be generated according to the weight parameter generation method corresponding to each candidate training method. In response to determining that the target weight parameter corresponding to any candidate training method is greater than the parameter threshold corresponding to the candidate training method, the candidate training method can be determined as the target training method. Furthermore, the base model can be optimized and trained according to the target training method, and the latest model can be determined as the base model. The above process can then be repeated.

[0082] In summary, by employing the scheme described in this disclosure, the reward model can be used to automatically determine and select the target training method for the next step, thereby automatically identifying the optimal path and optimizing the model training according to the target training method. The process of determining the optimal path and optimizing the model training can then be repeated, thus forming a closed-loop data flow processing. This enables the model to adaptively perform multi-stage optimization, thereby improving the model's training efficiency and learning effect. Furthermore, automatic path selection reduces resource waste and saves computation time and space.

[0083] The above explains the training methods for large question-answering models. The following section further explains the application methods of large question-answering models.

[0084] Accordingly, Figure 3This is a flowchart illustrating an embodiment of the question-answering task processing method described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0085] In step 301, the query request to be processed is obtained.

[0086] In step 302, the question-answering big model is used to generate the answer to the query request to be processed. The question-answering big model is obtained by training the basic model to be optimized according to the target training method. The target training method is the target training method that is suitable for this training, which is determined from the candidate training methods based on the evaluation results of each first answer. Each first answer is the first answer corresponding to each target query request generated by the basic model. The number of target query requests is greater than 1.

[0087] In the context of intelligent question answering, the query request to be processed refers to the question to be answered. The question answering model trained in accordance with the method described in this disclosure can be used to generate the answer to the question, thereby improving the accuracy of the generated answer.

[0088] The question can be a text-based question directly entered by the user, or it can be a text-based question obtained by converting the user's voice input. Furthermore, in practical applications, the question may include not only text content but also other forms of content such as images, depending on the specific needs. Similarly, the answer to the question is usually a text-based answer, but may also include other forms of content such as images if necessary.

[0089] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0090] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0091] Figure 4 This is a schematic diagram of the structural composition of embodiment 400 of the training device for the large question-answering model described in this disclosure. Figure 4 As shown, it includes: a first acquisition module 401, a second acquisition module 402, and an optimization processing model 403.

[0092] The first acquisition module 401 is used to acquire the basic model to be optimized.

[0093] The second acquisition module 402 is used to acquire the target query request corresponding to the basic model, and the number of target query requests is greater than 1.

[0094] The optimization processing module 403 is used to perform the following first processing: using the base model to generate the first answer corresponding to each target query request, and obtaining the evaluation result of each first answer respectively; in response, determining the target training method suitable for this training from the candidate training methods according to the evaluation result, and training the base model according to the target training method.

[0095] Using the scheme described in the above device embodiment, when it is necessary to train the basic model, the target training method suitable for this training can be automatically determined from the candidate training methods based on the evaluation results of the first answer generated by the basic model for each target query request. Then, the basic model can be trained according to the target training method. This allows for flexible selection of the training method according to the actual needs of the model, thereby improving the targeting of the training, and thus improving the training efficiency and accuracy of the model.

[0096] In some embodiments of this disclosure, after the optimization processing module 403 obtains the evaluation results of each first answer, in response to the fact that the target training method is not determined from the candidate training methods according to the evaluation results, the basic model can be determined as the target model after training. In addition, after training the basic model according to the target training method, the latest model can be determined as the basic model, and the target query request corresponding to the basic model can be obtained. Then, the first processing can be repeated.

[0097] For ease of explanation, assume the base model to be processed is the question-and-answer large model at time t. The corresponding target query requests can be identified, and the base model can be used to generate the first answer for each target query request. Then, the evaluation results of each first answer can be obtained. Further, if the target training method is not determined from the candidate training methods based on the evaluation results, the base model can be determined as the trained target model, thus obtaining the final target model, and the processing can end. If the target training method suitable for this training is determined from the candidate training methods based on the evaluation results, then the base model can be trained according to the target training method. Assuming the trained model is called the question-and-answer large model at time t+1, the above processing for the base model can be repeated for the question-and-answer large model at time t+1.

[0098] After generating corresponding first answers for each target query request in the training data, the optimization processing module 403 can obtain the evaluation results of each first answer. In some embodiments of this disclosure, the optimization processing module 403 can use a reward model to generate the evaluation results of each first answer.

[0099] In addition, in some embodiments of this disclosure, after the optimization processing module 403 obtains the evaluation results of each first answer, it can also output the evaluation results and obtain the correction results input for the evaluation results. At least a training dataset is generated based on the correction results, and the reward model is trained using the training dataset.

[0100] In some embodiments of this disclosure, the optimization processing module 403 may generate a training dataset based on the corrected result and the initial result in response to determining that there is an uncorrected initial result in the evaluation result.

[0101] In some embodiments of this disclosure, the optimization processing module 403 may determine the target training method suitable for this training from the candidate training methods based on the evaluation results. This may include: generating target weight parameters corresponding to each candidate training method according to the evaluation results of each first answer and the weight parameter generation method corresponding to each candidate training method; and determining the candidate training method corresponding to the target weight parameter that is greater than the corresponding parameter threshold as the target training method in response to determining that any target weight parameter is greater than the parameter threshold.

[0102] The identified target training method may be one, multiple, or zero. If there are zero, it means that no target training method has been identified from the candidate training methods. Accordingly, the current base model can be identified as the target model after training. If there are multiple identified target training methods, the base model can be trained according to the target training method.

[0103] In some embodiments of this disclosure, in response to determining that the number of target training methods is greater than 1, the optimization processing module 403 can sequentially train the base model using each target training method in descending order of priority, with different candidate training methods corresponding to different priorities.

[0104] In some embodiments of this disclosure, candidate training methods may include any of the following: Prompting training, SFT training, and DPO training.

[0105] In some embodiments of this disclosure, in response to determining that the target training method is Prompting training, the optimization processing module 403 can determine the latest obtained model as the model to be processed, and can trigger the model to be processed to update the Prompt, and can trigger the model to be processed to regenerate the answer according to the updated Prompt, so as to obtain the second answer corresponding to each target query request, and then update the model parameters of the model to be processed according to each second answer.

[0106] In addition, in some embodiments of this disclosure, in response to determining that the target training method is SFT training, the optimization processing module 403 can determine the latest obtained model as the model to be processed, and can guide the model to be processed to regenerate the answer according to the evaluation results of each first answer, so as to obtain the third answer corresponding to each target query request, and then update the model parameters of the model to be processed according to each third answer.

[0107] Furthermore, in some embodiments of this disclosure, in response to determining that the target training method is DPO training, the optimization processing module 403 can determine the latest obtained model as the model to be processed, and can update the model parameters of the model to be processed according to the evaluation results of each first answer.

[0108] After training the base model using various target training methods, the latest model can be determined as the base model, and the relevant processing performed on the base model in the scheme described in this disclosure can be repeated.

[0109] Figure 5 This is a schematic diagram of the structural composition of Embodiment 500 of the question-answering task processing device described in this disclosure. Figure 5 As shown, it includes: a third acquisition module 501 and a task processing module 502.

[0110] The third acquisition module 501 is used to acquire query requests to be processed.

[0111] The task processing module 502 is used to generate answers to the query requests to be processed using a large question-answering model. The large question-answering model is obtained by training the basic model to be optimized according to the target training method. The target training method is the target training method that is suitable for this training, which is determined from the candidate training methods based on the evaluation results of each first answer. Each first answer is the first answer corresponding to each target query request generated by the basic model. The number of target query requests is greater than 1.

[0112] Figure 6 This is a schematic diagram of the composition structure of the intelligent agent embodiment 600 described in this disclosure. Figure 6 As shown, it includes: an input module 601, a processing module 602, and an output module 603.

[0113] Input module 601 is used to receive query requests to be processed.

[0114] Processing module 602 is used to determine the target task based on the query request to be processed received by input module 601, determine the question-answering big model based on the target task, and execute the question-answering big model by calling it. Figure 3 The question-and-answer task processing method shown obtains the answer to the query request to be processed.

[0115] Output module 603 is used to output the answer obtained by processing module 602.

[0116] The specific workflow of each of the above device embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0117] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly deep learning, large-scale models, and natural language processing. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0118] The target query requests and answers described in the embodiments of this disclosure are not targeted at any specific user and do not reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0119] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0120] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0121] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0122] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of displays, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0123] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0124] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0126] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0128] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0129] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0130] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0131] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a large question-answering model, comprising: obtaining a base model to be optimized; obtaining target query requests corresponding to the base model, the number of target query requests being greater than 1; performing the following first processing: generating first answers corresponding to each target query request respectively using the base model; obtaining evaluation results of each first answer respectively; generating target weight parameters corresponding to each candidate training mode according to the weight parameters of each candidate training mode according to the evaluation results; in response to determining that any target weight parameter is greater than a corresponding parameter threshold, determining the candidate training mode corresponding to the target weight parameter greater than the parameter threshold as a target training mode; training the base model according to the target training mode; determining the latest obtained model as the base model; and obtaining target query requests corresponding to the base model, and then repeatedly performing the first processing.

2. The method of claim 1, after obtaining the evaluation results of each first answer respectively, the method further comprises: in response to not determining the target training mode from the candidate training modes according to the evaluation results, determining the base model as a target model trained.

3. The method of claim 1 or 2, wherein, obtaining the evaluation results of each first answer respectively comprises: using a reward model to generate the evaluation results of each first answer respectively.

4. The method of claim 3, after obtaining the evaluation results of each first answer respectively, the method further comprises: outputting the evaluation results; obtaining a correction result input for the evaluation results; generating a training data set according to at least the correction result; training the reward model using the training data set.

5. The method of claim 4, generating a training data set according to at least the correction result comprises: in response to determining that there is an initial result that has not been corrected in the evaluation results, generating the training data set according to the correction result and the initial result.

6. The method of claim 1 or 2, wherein, training the base model according to the target training mode comprises: in response to determining that the number of target training modes is greater than 1, sequentially training the base model using each target training mode in descending order of priority of the target training modes.

7. The method of claim 6, wherein: the candidate training mode comprises any of the following: prompt word optimization training, supervised fine-tuning training, and direct preference optimization training.

8. The method of claim 7, wherein, in response to determining that the target training mode is the prompt word optimization training, training the base model according to the target training mode comprises: determining the latest obtained model as a to-be-processed model; triggering the to-be-processed model to update the prompt words, and triggering the to-be-processed model to generate second answers corresponding to each target query request according to the updated prompt words; updating the model parameters of the to-be-processed model according to each second answer.

9. The method of claim 7, wherein, in response to determining that the target training mode is the supervised fine-tuning training, training the base model according to the target training mode comprises: determine the latest obtained model as a to-be-processed model; guide the to-be-processed model to re-generate answers according to evaluation results of each first answer, to obtain third answers corresponding to each target query request; update model parameters of the to-be-processed model according to each third answer.

10. The method of claim 7, wherein, In response to determining that the target training mode is the direct preference optimization training, the training of the base model according to the target training mode includes: determine the latest obtained model as a to-be-processed model, and update model parameters of the to-be-processed model according to evaluation results of each first answer.

11. A method for processing a question and answer task, comprising: obtaining a to-be-processed query request; generating an answer corresponding to the to-be-processed query request by using a question and answer large model, the question and answer large model being trained by the method according to any one of claims 1-10.

12. A training device of a question and answer large model, comprising: a first obtaining module, a second obtaining module, and an optimization processing model; the first obtaining module is configured to obtain a to-be-optimized base model; the second obtaining module is configured to obtain target query requests corresponding to the base model, the number of the target query requests being greater than 1; the optimization processing module is configured to perform the following first processing: generating first answers corresponding to each target query request by using the base model respectively; obtaining evaluation results of each first answer respectively; generating target weight parameters corresponding to each candidate training mode according to a weight parameter generation mode corresponding to each candidate training mode respectively according to each evaluation result; in response to determining that any target weight parameter is greater than a corresponding parameter threshold, determining a candidate training mode corresponding to the target weight parameter greater than the parameter threshold as a target training mode; training the base model according to the target training mode; determining a latest obtained model as the base model; and obtaining target query requests corresponding to the base model, and then repeating the first processing.

13. The apparatus according to claim 12, wherein the optimization processing module is further configured to, after the evaluation results of each first answer are obtained respectively, in response to not determining the target training mode from the candidate training modes according to the evaluation results, determine the base model as a target model of which the training is completed.

14. The apparatus according to claim 12 or 13, wherein the optimization processing module generates the evaluation results of each first answer by using a reward model respectively.

15. The apparatus according to claim 14, wherein the optimization processing module is further configured to, after the evaluation results of each first answer are obtained respectively, output the evaluation results, obtain a correction result input for the evaluation results, generate a training data set according to at least the correction result, and train the reward model by using the training data set.

16. The apparatus according to claim 15, wherein the optimization processing module generates the training data set according to the correction result and an initial result that is not corrected in the evaluation results in response to determining that the initial result exists.

17. The apparatus according to claim 12 or 13, wherein The optimization processing module sequentially trains the base model according to the target training modes in descending order of priority of the target training modes in response to determining that the number of the target training modes is greater than 1.

18. The apparatus of claim 17, wherein, The candidate training mode includes any one of prompt optimization training, supervised fine-tuning training, and direct preference optimization training.

19. The apparatus of claim 18, wherein, The optimization processing module determines the latest obtained model as a to-be-processed model, triggers the to-be-processed model to perform prompt updating, and triggers the to-be-processed model to regenerate answers according to the updated prompts to obtain second answers corresponding to each target query request, and performs model parameter updating on the to-be-processed model according to each second answer in response to determining that the target training mode is the prompt optimization training.

20. The apparatus of claim 18, wherein, The optimization processing module determines the latest obtained model as a to-be-processed model, guides the to-be-processed model to regenerate answers according to the evaluation results of each first answer to obtain third answers corresponding to each target query request, and performs model parameter updating on the to-be-processed model according to each third answer in response to determining that the target training mode is the supervised fine-tuning training.

21. The apparatus of claim 18, wherein, The optimization processing module determines the latest obtained model as a to-be-processed model, and performs model parameter updating on the to-be-processed model according to the evaluation results of each first answer in response to determining that the target training mode is the direct preference optimization training.

22. A question and answer task processing apparatus comprising: The third acquisition module and the task processing module; The third acquisition module is configured to acquire a to-be-processed query request. The task processing module is configured to generate an answer corresponding to the to-be-processed query request by using a question and answer large model, and the question and answer large model is trained by the method in any one of claims 1-10.

23. An agent comprising: The input module, the processing module, and the output module; The input module is configured to receive a to-be-processed query request. The processing module is configured to determine a target task based on the to-be-processed query request received by the input module, determine a question and answer large model based on the target task, execute the question and answer task processing method in claim 11 by calling the question and answer large model, and obtain an answer corresponding to the to-be-processed query request. The output module is configured to output the answer obtained by the processing module.

24. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any one of claims 1-11.

25. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to execute the method in any one of claims 1-11.

26. A computer program product comprising computer programs / instructions which, when executed by a processor, implement the method of any one of claims 1-11.

Images