Question and answer method, model training method, device, equipment, medium and program product

Abstract

This application provides a question-answering method, model training method, apparatus, device, medium, and program product, relating to the field of artificial intelligence, and applicable to the fintech field or other fields, involving the application of large-scale models in information security and fintech. The question-answering method includes: inputting a question to be answered into a first large-scale model to obtain the target answer generated by the first large-scale model; obtaining the first large-scale model by inputting question samples and answer samples from a candidate training set into a second large-scale model to obtain the predicted answer and reasoning process generated by the second large-scale model based on the question samples, wherein the second large-scale model has a larger number of parameters than the first large-scale model; evaluating the predicted answer and answer samples of the question samples using a third large-scale model to obtain an evaluation result; based on the confidence level of the predicted answer indicated by the evaluation result, obtaining N target training sets using the candidate training set, and training the first large-scale model using M training methods.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to the application of large models in information security and fintech, and more specifically to a question-answering method, model training method, device, equipment, medium, and program product. Background Technology

[0002] As the scale of deep learning models used in question answering continues to increase, large models with hundreds of billions of parameters enable question answering methods to perform well in general tasks. However, their massive computational resource requirements and high deployment costs severely limit their application in real-time question answering scenarios in vertical domains. Although techniques for transferring knowledge from large-parameter models to small-parameter models can reduce deployment overhead, existing question answering methods still suffer from poor performance in relevant scenarios. This is because small-parameter models still face limitations in training due to fixed data processing mechanisms and single training strategies: they cannot construct training sets based on the quality classification of predicted answers output by large-parameter models, and they lack reasonable adaptation of training methods for different quality and type data. As a result, small-parameter models struggle to achieve professional inference accuracy in highly rigorous scenarios such as finance. Therefore, there is an urgent need for a new question answering method—one that maintains the advantages of lightweight deployment while ensuring the accuracy and reliability of answers in professional domains through credibility-guided hierarchical dataset construction and multi-strategy adaptation training mechanisms. Summary of the Invention

[0003] In view of the above problems, this application provides a question-answering method, a model training method, an apparatus, a device, a medium, and a program product.

[0004] According to a first aspect of this application, a question-answering method is provided, comprising: inputting an unanswered question sent by a target object into a first large model to obtain a target answer generated by the first large model; wherein the first large model is pre-trained as follows: inputting question samples and answer samples from a candidate training set into a second large model to obtain a predicted answer generated by the second large model based on the question samples and the reasoning process of the predicted answer, wherein the number of parameters of the second large model is greater than the number of parameters of the first large model; evaluating the predicted answer and answer samples of the question samples using a third large model to obtain an evaluation result; based on the credibility of the predicted answer of the question samples indicated by the evaluation result, obtaining N target training sets using the candidate training set, and training the first large model based on M training methods, wherein N and M are integers greater than or equal to 2, and M is less than or equal to N, and the target training sets include question samples with corresponding credibility, the reasoning process of the predicted answer, and answer samples.

[0005] According to an embodiment of this application, obtaining N target training sets using candidate training sets and training a first major model based on M training methods includes: determining the training order for the first major model based on the credibility ranking of the N target training sets; and training the first major model based on the corresponding target training sets using the M training methods based on the training order.

[0006] According to an embodiment of this application, training the first large model based on training ranking using M training methods on their respective target training sets includes: for the m-th training method whose ranking position is not first, training the first large model obtained by training using the (m-1)-th training method on its corresponding target training set using the m-th training method, where m is greater than 1 and less than or equal to M.

[0007] According to an embodiment of this application, based on the credibility of the predicted answer indicated by the evaluation result, N target training sets are obtained based on the candidate training set, including: for question samples whose credibility of the predicted answer is greater than or equal to a first preset value, storing the question sample, its predicted answer, and the reasoning process into the first target training set.

[0008] According to an embodiment of this application, training the first large model based on M training methods includes: fine-tuning the first large model to be trained using a first target training set.

[0009] According to an embodiment of this application, based on the credibility of the predicted answer indicated by the evaluation result, obtaining N target training sets based on the candidate training set includes: for a question sample whose credibility of the predicted answer is less than a first preset value, using a first prompt word to guide a second large model to regenerate the predicted answer and reasoning process of the question sample; using a third large model to evaluate the regenerated predicted answer and answer sample, so as to obtain at least one target training set based on the evaluation result.

[0010] According to an embodiment of this application, the evaluation of the regenerated predicted answers and answer samples using a third model to obtain at least one target training set based on the evaluation results includes: when the number of regenerations is less than or equal to a target value, for question samples whose confidence of the regenerated predicted answers is greater than or equal to a first preset value, obtaining a first training sample based on the question sample and its incorrect predicted answers and incorrect reasoning process, obtaining a second training sample based on the question sample and its correct predicted answers and correct reasoning process; and storing the first training sample and the second training sample into a second target training set.

[0011] According to an embodiment of this application, training a first large model based on M training methods includes: performing direct preference optimization reinforcement learning training on the first large model to be trained using a second target training set; wherein, the loss function value in the reinforcement learning training process is obtained based on a first difference between predicted data and erroneous data, and a second difference between predicted data and correct data, the predicted data includes the predicted answer and reasoning process output by the first large model to be trained, the erroneous data includes the erroneous predicted answer and erroneous reasoning process of the first training sample, and the correct data includes the correct predicted answer and correct reasoning process of the second training sample.

[0012] According to an embodiment of this application, evaluating the regenerated predicted answer and answer sample using a third large model to obtain at least one target training set based on the evaluation result includes: when the number of regenerations is greater than a target value, for a question sample whose confidence of the predicted answer is less than a first preset value, inputting the answer sample of the question sample into the first large model; using a second prompt word to guide the first large model to regenerate the predicted answer and reasoning process based on the answer sample of the question sample; and storing the question sample and its regenerated predicted answer and reasoning process into the third target training set.

[0013] According to an embodiment of this application, training the first large model based on M training methods includes: using a third objective training set to perform group relative policy optimization reinforcement learning training on the first large model to be trained.

[0014] Another aspect of this application provides a model training method, comprising: inputting question samples and answer samples from a candidate training set into a second large model to obtain the predicted answer and reasoning process generated by the second large model based on the question samples, wherein the number of parameters of the second large model is greater than the number of parameters of the first large model to be trained; evaluating the predicted answer and answer samples of the question samples using a third large model to obtain an evaluation result; and obtaining N target training sets based on the confidence level of the predicted answer indicated by the evaluation result, and training the first large model using M training methods, wherein N and M are integers greater than or equal to 2, and M is less than or equal to N, and the target training sets include question samples, predicted answers, and reasoning processes corresponding to the confidence level.

[0015] Another aspect of this application provides a question-answering device, comprising: a target answer generation module, used to input an unanswered question sent by a target object into a first large model to obtain a target answer generated by the first large model; and a first large model pre-training device, comprising: a predicted answer and reasoning process generation module, used to input question samples and answer samples from a candidate training set into a second large model to obtain a predicted answer and reasoning process generated by the second large model based on the question samples, wherein the number of parameters of the second large model is greater than the number of parameters of the first large model; an evaluation module, used to evaluate the predicted answer and answer samples of the question samples using a third large model to obtain an evaluation result; and a target training set generation module, used to obtain N target training sets based on the confidence level of the predicted answer indicated by the evaluation result using the candidate training set, and to train the first large model based on M training methods, wherein N and M are integers greater than or equal to 2, and M is less than or equal to N, and the target training sets include question samples, predicted answers, and reasoning processes corresponding to the confidence levels.

[0016] Another aspect of this application provides a model training apparatus, comprising: a prediction answer and reasoning process generation module, used to input question samples and answer samples from a candidate training set into a second large model to obtain prediction answers and reasoning processes generated by the second large model based on the question samples, wherein the number of parameters of the second large model is greater than the number of parameters of the first large model to be trained; an evaluation module, used to evaluate the prediction answers and answer samples of the question samples using a third large model to obtain evaluation results; and a target training set generation module, used to obtain N target training sets based on the confidence level of the prediction answers indicated by the evaluation results using the candidate training set, and to train the first large model based on M training methods, wherein N and M are integers greater than or equal to 2, and M is less than or equal to N, and the target training sets include question samples, prediction answers, and reasoning processes corresponding to the confidence levels.

[0017] Another aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0018] Another aspect of this application provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0019] Another aspect of this application provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0020] The performance improvement of the question-answering method in this application is achieved by using a third model to evaluate the predicted answers and answer samples of the second model for question samples, thereby obtaining an evaluation result on the credibility of the indicated predicted answers. The candidate training set is divided into multiple target training sets. Each target training set contains question samples, predicted answers, and reasoning processes at its corresponding credibility level. For these target training sets, multiple adaptive training methods are used to train the first model. This fully utilizes the characteristics of training data at different credibility levels, selecting matching training strategies for each target training set, thus efficiently transferring the professional knowledge and reasoning ability of the second model in question answering. Ultimately, when responding to the unanswered questions of the target object, the first model can generate accurate target answers and reasoning processes, significantly improving the accuracy and reliability of question answering. Attached Figure Description

[0021] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0022] Figure 1 The illustrations depict application scenarios of question-answering methods, model training methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0023] Figure 2 A flowchart illustrating a question-and-answer method according to an embodiment of this application is shown schematically;

[0024] Figure 3 This schematically illustrates a flowchart of determining the order of the target training set and training a first large model based on the order according to an embodiment of this application;

[0025] Figure 4 An example flowchart of a model training method according to an embodiment of this application is illustrated schematically;

[0026] Figure 5 This schematic diagram illustrates a structural block diagram of a question-and-answer device according to an embodiment of the present application;

[0027] Figure 6 This schematic diagram illustrates the structure of a model training apparatus according to an embodiment of the present application;

[0028] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a question-answering method and a model training method according to an embodiment of this application. Detailed Implementation

[0029] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0032] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0033] Figure 1 The illustrations depict application scenarios of question-answering methods, model training methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0034] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the question-answering method and model training method provided in this application embodiment can generally be executed by server 105. Correspondingly, the question-answering device and model training device provided in this application embodiment can generally be located in server 105. The question-answering method and model training method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the question-answering device and model training device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] The following will be based on Figure 1 The described scene, through Figures 2-3 The question-and-answer method according to the embodiments of this application will be described in detail.

[0041] Figure 2 A flowchart illustrating a question-and-answer method according to an embodiment of this application is shown schematically.

[0042] like Figure 2 As shown, the flowchart of the question-answering method in this embodiment includes operations S210 to S240, wherein operations S210 to S230 are the training phase and operation S240 is the execution phase.

[0043] In operation S210, the question samples and answer samples in the candidate training set are input into the second large model to obtain the predicted answer and reasoning process generated by the second large model based on the question samples. The number of parameters in the second large model is greater than the number of parameters in the first large model.

[0044] In the embodiments of this application, the large models involved (such as the first, second, or third large models) can be general-purpose large language models with a large number of parameters, such as models with hundreds of millions, billions, tens of billions, or hundreds of billions of parameters. Large models can also be expert large models with complete knowledge systems and sophisticated complex logical reasoning capabilities in various vertical fields such as finance and healthcare; this application does not impose any limitations. For example, the first large model to be trained may adopt a lightweight architecture with fewer parameters than the second large model, significantly reducing deployment hardware requirements and improving training efficiency while effectively saving computational overhead. The relationship between the two can be that the second large model acts as a teacher model, transferring knowledge within a specific domain to the first large model. The powerful teacher model automatically generates intermediate reasoning processes and answers, forming question and answer samples that are then passed to the first large model as a student model, thereby extracting high-quality training samples. In this way, reliable thinking examples can be provided to the model without manual annotation of the reasoning process, providing a reliable data foundation for subsequent model fine-tuning and reinforcement learning, and improving the reasoning ability and accuracy of the first large model. Its core idea is to transfer "knowledge" (such as output distribution, intermediate features, or decision logic) into a lightweight student model, thereby achieving model capability distillation for specific professional fields such as finance, healthcare, education, and law. It focuses on extracting core knowledge of vertical fields from large models to build small and specialized models.

[0045] The question samples can be pre-prepared professional questions within a specific field. Taking the financial field as an example, in financial question-and-answer application scenarios, a financial question-and-answer corpus is collected, in the form of a candidate training set consisting of "question samples + answer samples." The data collection and processing process involves the following steps:

[0046] Dimensional Design: The financial sector covers banking business (account management, credit products, etc.), securities market (trading mechanisms, derivatives pricing, etc.), insurance actuarial science (risk models / solvency, etc.), regulatory compliance (anti-money laundering, etc.), etc.

[0047] Data collection methods: Utilizing technologies such as document parsing, natural language processing entity recognition, and knowledge graph construction to extract relevant regulatory documents, scalable business report language data, and other financial professional examination question banks.

[0048] Data format: Please refer to the JSON (JavaScript Object Notation) data format below.

[0049] {

[0050] "qid":"000001",

[0051] "category":"insurance",

[0052] "question":"How to calculate the basis point value of a CDS (Credit Default Swap)?"

[0053] "answer":"CDS basis point value = notional principal × duration × 0.0001"

[0054] }

[0055] Cleaning rules: Remove duplicate question-answer pairs, for example, by deduplicating based on qid (Question Identifier); filter non-text data (such as chart-based questions); standardize financial terminology, etc.

[0056] Compliance Detection: A financial sentiment analysis model is used to detect financially sensitive words in question-and-answer pairs. A risk level (positive, negative, and neutral) is output, and question-and-answer pairs with a negative risk level are filtered out.

[0057] In operation S220, the third major model is used to evaluate the predicted answers and answer samples of the question samples to obtain the evaluation results.

[0058] The third model can be used to compare the predicted answers and the actual answers output by the second model for the input question sample, and to assess the reliability of the predicted answers.

[0059] In operation S230, based on the credibility of the predicted answer indicated by the evaluation result, N target training sets are obtained using the candidate training sets. The first model is trained using M training methods, where N and M are integers greater than or equal to 2, and M is less than or equal to N. The target training sets include question samples, predicted answers, and reasoning processes corresponding to the credibility.

[0060] In the embodiments of this application, based on the confidence level of the second model's predicted answer indicated by the evaluation results, at least two target training sets can be obtained by dividing the question samples and corresponding answer samples within the candidate training set. The training methods corresponding to the different target training sets can be different or the same, but more than two training methods are used overall. For example, at least one target training set has a different training method than the others. The training methods can include Supervised Fine-Tuning (SFT), Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), etc.

[0061] After the training phase, the execution phase begins. In operation S240, the unanswered question sent by the target object is input into the first large model, and the target answer generated by the first large model is obtained.

[0062] Through credibility assessment and multi-strategy training mechanisms, the first model possesses professional capabilities similar to the second model within a specific domain. The target user inputs the question to be answered into the trained first model and obtains the target answer generated by the first model, achieving accurate question answering within a specific domain and enhancing the accuracy and reliability of the question answering method in generating answers.

[0063] The following is combined Figure 3 The credibility of the predicted answer based on the evaluation result indication provided in the embodiments of this application is introduced by using candidate training sets to obtain N target training sets and training the first large model based on M training methods.

[0064] Figure 3 The flowchart illustrating the process of determining the order of the target training set and training a first large model based on the order is shown in the embodiment of this application.

[0065] like Figure 3 As shown, this embodiment includes operations S310 to S320.

[0066] In operation S310, based on the credibility ranking of N target training sets, the training ranking for the first large model is determined based on M training methods.

[0067] In the embodiments of this application, the credibility of N target training sets can be sorted in descending order, and a training method corresponding to one or more target training sets can be determined according to actual training needs. The number M of training methods corresponding to all N target training sets is greater than or equal to 2.

[0068] In operation S320, based on training ranking, the first large model is trained using M training methods on their respective target training sets.

[0069] The training order is determined by ranking the credibility of the target training set, optimizing the fit with the training strategy, ensuring efficient and progressive training, and improving model performance.

[0070] In the embodiments of this application, the training methods determined by the N target training sets based on credibility ranking are determined hierarchically according to the credibility and data format of the target data, and each can correspond to the most suitable training method. At this time, based on the training ranking, training the first large model using M training methods on their respective target training sets includes: for the m-th training method that is not in the first position of the ranking, training the first large model obtained by training the (m-1)-th training method using the m-th training method on its corresponding target training set, where m is greater than 1 and less than or equal to M. This forms an iterative training mechanism, achieving progressive optimization of model capabilities through the ordered execution of training methods corresponding to data credibility. The iterative training mechanism, where non-first-position training methods depend on the previous training result, achieves gradual optimization of model capabilities, reduces resource waste, and enhances model stability.

[0071] Based on operations S210~S240 and S310~S320, and based on the credibility of the predicted answer indicated by the evaluation result, N target training sets are obtained based on the candidate training sets. These include: for question samples whose predicted answer credibility is greater than or equal to a first preset value, the question sample, its predicted answer, and the reasoning process are stored in the first target training set. The first target training set is constructed by selecting high-credibility data to provide high-quality basic data for subsequent training.

[0072] Training the first major model using M training methods includes: fine-tuning the first major model to be trained using the first target training set. Supervised fine-tuning using the first target training set allows for rapid semantic alignment and reduces basic inference errors.

[0073] Based on operations S210~S240 and S310~S320, and based on the credibility of the predicted answer indicated by the evaluation result, N target training sets are obtained based on the candidate training set. These include: for question samples where the credibility of the predicted answer is less than a first preset value, using a first prompt word to guide the second model to regenerate the predicted answer and reasoning process for that question sample; and using a third model to evaluate the regenerated predicted answer and answer sample, obtaining at least one target training set based on the evaluation result. This triggers a reflection mechanism for low-credibility data, regenerating the predicted answer, improving data utilization, and correcting original erroneous samples.

[0074] The third model is used to evaluate the regenerated predicted answers and answer samples to obtain at least one target training set based on the evaluation results. This includes: for question samples where the number of regenerations is less than or equal to the target value, and the credibility of the regenerated predicted answers is greater than or equal to a first preset value, a first training sample is obtained based on the question sample, its incorrect predicted answer, and incorrect reasoning process; a second training sample is obtained based on the question sample, its correct predicted answer, and correct reasoning process; the first and second training samples are stored in the second target training set. Preference data pairs (incorrect samples vs. correct samples) are constructed based on the reflection results to provide differentiated training materials for reinforcement learning.

[0075] Training the first major model using M training methods includes: Direct preference optimization reinforcement learning training of the first major model using a second target training set; where the loss function value in the reinforcement learning training process is obtained based on the first difference between predicted data and incorrect data, and the second difference between predicted data and correct data. Predicted data includes the predicted answers and reasoning processes output by the first major model; incorrect data includes the incorrect predicted answers and incorrect reasoning processes of the first training samples; and correct data includes the correct predicted answers and correct reasoning processes of the second training samples. Reinforcement learning through direct preference optimization utilizes the loss function to strengthen the model's preference for correct answers, thereby improving decision-making accuracy.

[0076] The third model is used to evaluate the regenerated predicted answers and answer samples. Based on the evaluation results, at least one target training set is obtained. This includes: when the number of regenerations exceeds the target value, for question samples whose predicted answer confidence is less than a first preset value, the answer sample of that question sample is input into the first model. This guides the first model in the reasoning process for generating question samples that have failed to be corrected multiple times, transforming low-confidence data into suitable training samples, overcoming the performance bottleneck of the first model, and effectively improving data utilization.

[0077] The second prompt word guides the first model to regenerate the predicted answer and reasoning process based on the answer sample of the question sample. This question sample, along with the regenerated predicted answer and reasoning process, is then stored in the third objective training set. The first model to be trained is then trained using group relative policy optimization reinforcement learning on the third objective training set. By employing group relative policy optimization reinforcement learning to train the third objective training set, the stability and reliability of the model in handling complex problems are enhanced through group reward calculation and policy update constraints.

[0078] Figure 4 This schematically illustrates an example flowchart of a model training method according to an embodiment of this application.

[0079] like Figure 4 As shown in the embodiments of this application, the predicted answer and the answer sample generated by the second model for the question sample are input into the third model. A consistency comparison is triggered by a specific prompt word, that is, a semantic consistency judgment is performed. In a specific implementation, the prompt word of the third model can be designed as "Please evaluate whether the following two answers are consistent. The answers to be evaluated are [answer sample] and [predicted answer]. The output is [yes] or [no]. The consistency evaluation result can be converted into a credibility indicator: [yes] - [high credibility], [no] - [low credibility]".

[0080] Here, high-confidence questions, reasoning processes, and predicted answers are added to the supervised fine-tuning training set. This first target training set is used for full-scale fine-tuning of the first large model. The loss function is designed as follows:

[0081]

[0082] in The standard cross-entropy loss is used for basic semantic alignment to optimize the model's general ability to generate answers; The loss of factual accuracy is used to calculate the probability distribution predicted by the model. Compared with the standard distribution of the training set KL divergence: Weight .

[0083] The learning rate is set to 2e-5, using a cosine decay method. An early stopping condition is also set to ensure that the learning stops when the validation set loss decreases by less than 0.1% for three consecutive rounds, thus reducing computational cost.

[0084] The low-confidence question samples are then input into the second model for t rounds of reflective reasoning, where t is a preset positive integer greater than or equal to 1. During this process, the third model continues to be used for answer evaluation.

[0085] If, within t iterations, the second largest model has already output the correct predicted answer as determined by the third largest model, then record (the question sample, the correct reasoning process in that iteration, and the correct predicted answer) as the preferred sample y in the preference data. w (The problem sample, the previous incorrect reasoning process, and the incorrectly predicted answer) serve as the inferior sample y in the preference data. l , will this pair of data (y w ,y l Add direct preference optimization to the training set;

[0086] Based on the supervised fine-tuning model, the direct preference optimization training set is used as the input set for the second objective model for direct preference optimization reinforcement learning training. The loss function design is as follows:

[0087]

[0088] Where σ is the sigmoid function and β is the control hyperparameter. Output the advantage value of the current model relative to the reference model, with weights β∈[0,1].

[0089] To avoid overfitting to biased data and reducing model performance, the training epochs should be set to less than 3.

[0090] If the second-largest model fails to output the correct predicted answer within t iterations, a prompt is used to force it to generate a reasoning process based on the given answer. The prompt can be designed as "Given question: [question sample] and answer: [answer sample], please output the reasoning process." Subsequently, the question sample, the forced reasoning process generated by the model, and the answer sample are stored as a triple [question sample, forced reasoning process, answer sample], and added to the group relative policy optimization training set D. grpo .

[0091] For training set D grpo Manual review and annotation are performed, focusing on correcting logical breaks in forced generation reasoning and removing samples that conflict with answer samples (such as formula errors) to ensure high data reliability. After processing, based on the model trained by reinforcement learning through direct preference optimization, the training set D is optimized using the group relative policy. grpo That is, GRPO reinforcement learning training is performed on the third objective training set.

[0092] The loss function is shown below:

[0093]

[0094] Where G represents the number of samples in a training batch. To optimize the current training model, To optimize the previous model, The KL divergence is calculated to constrain the policy update magnitude and prevent it from deviating too far from the current policy. Weights .

[0095] In order to target the The group relative reward loss term for each sample is used to maximize the expected reward, defined by group preference data, obtained by the model-generated response relative to the reference response. The specific calculation steps are as follows:

[0096] 1. Input a sample question Input question sample Up to the current model to be trained and optimized Generate multiple outputs within the group ;

[0097] 2. Obtain the reward value R(o): Input each output into the reward model R to obtain the reward;

[0098] 3. Calculate the average reward and standard deviation ;

[0099] 4. Calculate the relative reward for the group: For each sample within the group... Calculate their group relative reward ,in For constant terms, avoid having a denominator of zero:

[0100]

[0101] 5. Calculate the loss function: ,in This indicates that the current model is for the input Output Probability.

[0102] Setting the number of training epochs to less than 3 avoids overfitting to biased data and reducing model performance.

[0103] Finally, the model was trained using training datasets with three levels of credibility and three training methods that corresponded to the sequence, enabling the first model to process question-and-answer requests from the target object in real time.

[0104] It should be noted that the evaluation process performed by the third model described above outputs a "consistency" judgment result (yes / no), which is a specific quantitative implementation of the "credibility of the predicted answer." In this embodiment, a "consistency" judgment result can be equated to "the credibility of the predicted answer is greater than or equal to a first preset value"; a "disconsistency" judgment result can be equated to "the credibility of the predicted answer is less than a first preset value." In other words, the "first preset value" in this embodiment corresponds to the confidence threshold at which the evaluation model can confirm "consistency."

[0105] Of course, the "first preset value" can also be a pre-set specific numerical threshold (e.g., 0.8, 0.9, etc.). In practical applications, the semantic similarity score or confidence score between the predicted answer and the standard answer can be output by the third major model, and the credibility level can be divided by comparing it with the numerical threshold. This application does not limit the specific implementation of the "first preset value," and the above-mentioned binary division method based on consistency judgment is an efficient and reliable preferred solution.

[0106] Embodiments of this application also relate to a model training method, which involves inputting question samples and answer samples from a candidate training set into a second large model to obtain the predicted answers and reasoning processes generated by the second large model based on the question samples. The number of parameters in the second large model is greater than the number of parameters in the first large model to be trained. A third large model is then used to evaluate the predicted answers and answer samples of the question samples to obtain evaluation results. A complete model training framework is provided to achieve high-precision reasoning of lightweight models in professional scenarios.

[0107] Based on the above question-and-answer method, this application also provides a question-and-answer device. The following will combine... Figure 5 The device is described in detail.

[0108] Figure 5 A schematic block diagram of a question-and-answer device according to an embodiment of this application is shown.

[0109] like Figure 5 As shown, the question-answering device 500 of this embodiment includes a target answer generation module 510, a predicted answer and reasoning process generation module 520, an evaluation module 530, and a target training set generation module 540.

[0110] The target answer generation module 510 is used to input the unanswered question sent by the target object into the first large model and obtain the target answer generated by the first large model. In one embodiment, the target answer generation module 510 can be used to perform the operation S240 described above, which will not be repeated here.

[0111] The prediction answer and reasoning process generation module 520 is used to input question samples and answer samples from the candidate training set into the second large model to obtain the prediction answer and reasoning process generated by the second large model based on the question samples. The number of parameters in the second large model is greater than the number of parameters in the first large model. In one embodiment, the prediction answer and reasoning process generation module 520 can be used to perform the operation S210 described above, which will not be repeated here.

[0112] The evaluation module 530 is used to evaluate the predicted answers and answer samples of the question samples using the third model to obtain evaluation results. In one embodiment, the evaluation module 530 can be used to perform the operation S220 described above, which will not be repeated here.

[0113] The target training set generation module 540 is used to obtain N target training sets based on the credibility of the predicted answer indicated by the evaluation result, and to train the first model based on M training methods, where N and M are integers greater than or equal to 2, and M is less than or equal to N. The target training set includes question samples, predicted answers, and reasoning processes corresponding to the credibility. In one embodiment, the target training set generation module 540 can be used to perform the operation S230 described above, which will not be repeated here.

[0114] According to embodiments of this application, any multiple modules among the target answer generation module 510, the predicted answer and reasoning process generation module 520, the evaluation module 530, and the target training set generation module 540 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the target answer generation module 510, the predicted answer and reasoning process generation module 520, the evaluation module 530, and the target training set generation module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the target answer generation module 510, the predicted answer and reasoning process generation module 520, the evaluation module 530, and the target training set generation module 540 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0115] Figure 6A schematic block diagram of a model training apparatus according to an embodiment of this application is shown.

[0116] like Figure 6 As shown, the model training device 600 of this embodiment includes a prediction answer and reasoning process generation module 610, an evaluation module 620, and a target training set generation module 630.

[0117] The prediction answer and reasoning process generation module 610 is used to input question samples and answer samples from the candidate training set into a second large model to obtain the prediction answer and reasoning process generated by the second large model based on the question samples. The number of parameters in the second large model is greater than the number of parameters in the first large model to be trained. In one embodiment, the prediction answer and reasoning process generation module 610 can be used to perform the operation S310 described above, which will not be repeated here.

[0118] The evaluation module 620 is used to evaluate the predicted answers and answer samples of the question samples using the third model, and obtain the evaluation results. In one embodiment, the evaluation module 620 can be used to perform the operation S320 described above, which will not be repeated here.

[0119] The target training set generation module 630 is used to obtain N target training sets based on the credibility of the predicted answer indicated by the evaluation result, and to train the first model based on M training methods, where N and M are integers greater than or equal to 2, and M is less than or equal to N. The target training set includes question samples, predicted answers, and reasoning processes corresponding to the credibility. In one embodiment, the target training set generation module 630 can be used to perform the operation S240 described above, which will not be repeated here.

[0120] According to embodiments of this application, any multiple modules among the prediction answer and reasoning process generation module 610, evaluation module 620, and target training set generation module 630 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the prediction answer and reasoning process generation module 610, evaluation module 620, and target training set generation module 630 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the prediction answer and reasoning process generation module 610, evaluation module 620, and target training set generation module 630 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0121] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a question-answering method and a model training method according to an embodiment of this application.

[0122] like Figure 7 As shown, an electronic device 700 according to an embodiment of this application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0123] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0124] According to embodiments of this application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0125] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0126] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0127] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the question-and-answer method provided in the embodiments of this application.

[0128] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0129] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0130] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0131] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A question-and-answer method, characterized in that, include: Input the unanswered question sent by the target object into the first large model, and obtain the target answer generated by the first large model; The first large model was pre-trained as follows: Input the question samples and answer samples from the candidate training set into the second large model to obtain the predicted answer and reasoning process generated by the second large model based on the question samples. The number of parameters of the second large model is greater than that of the first large model. The predicted answers and answer samples of the question samples are evaluated using the third major model to obtain the evaluation results; Based on the credibility of the predicted answer indicated by the evaluation results, N target training sets are obtained using the candidate training sets, and the first large model is trained using M training methods, where N and M are integers greater than or equal to 2, and M is less than or equal to N. The target training sets include question samples, predicted answers, and reasoning processes corresponding to the credibility.

2. The method according to claim 1, characterized in that, The step of obtaining N target training sets using the candidate training sets and training the first large model based on M training methods includes: Based on the credibility ranking of the N target training sets, the training ranking for the first large model based on the M training methods is determined; Based on the training ranking, the first large model is trained using the M training methods on their respective target training sets.

3. The method according to claim 2, characterized in that, The step of training the first large model based on the training ranking and using the M training methods on their respective target training sets includes: for the m-th training method whose ranking position is not the first position... Using the m-th training method based on its corresponding target training set, train the first large model obtained by training using the (m-1)-th training method, where m is greater than 1 and less than or equal to M.

4. The method according to any one of claims 1 to 3, characterized in that, The credibility of the predicted answer based on the evaluation result, and the N target training sets obtained based on the candidate training sets, include: For question samples whose predicted answer confidence is greater than or equal to a first preset value, the question sample, its predicted answer, and reasoning process are stored in the first target training set.

5. The method according to claim 4, characterized in that, The training of the first large model based on M training methods includes: The first target training set is used to fine-tune the first large model to be trained.

6. The method according to any one of claims 1 to 3, characterized in that, The credibility of the predicted answer based on the evaluation result, and the N target training sets obtained based on the candidate training sets, include: For question samples where the confidence level of the predicted answer is less than the first preset value, the second large model is guided by the first prompt word to regenerate the predicted answer and reasoning process for the question sample. The regenerated predicted answers and answer samples are evaluated using the third major model to obtain at least one target training set based on the evaluation results.

7. The method according to claim 6, characterized in that, The process of evaluating the regenerated predicted answers and answer samples using the third major model to obtain at least one target training set based on the evaluation results includes: If the number of regenerations is less than or equal to the target value, for a question sample whose confidence of the regenerated predicted answer is greater than or equal to the first preset value, a first training sample is obtained based on the question sample, its incorrect predicted answer, and incorrect reasoning process, and a second training sample is obtained based on the question sample, its correct predicted answer, and correct reasoning process. The first training sample and the second training sample are stored in the second target training set.

8. The method according to claim 7, characterized in that, The training of the first large model based on M training methods includes: The first large model to be trained is trained using direct preference optimization reinforcement learning with the second target training set. The loss function value in the reinforcement learning training process is obtained based on a first difference between the predicted data and the incorrect data, and a second difference between the predicted data and the correct data. The predicted data includes the predicted answer and reasoning process output by the first large model to be trained. The incorrect data includes the incorrect predicted answer and incorrect reasoning process of the first training sample. The correct data includes the correct predicted answer and correct reasoning process of the second training sample.

9. The method according to claim 6, characterized in that, The process of evaluating the regenerated predicted answers and answer samples using the third major model to obtain at least one target training set based on the evaluation results includes: If the number of regenerations exceeds the target value, for question samples whose predicted answer confidence is less than the first preset value, the answer sample of that question sample is input into the first large model; The second prompt word guides the first large model to regenerate the predicted answer and reasoning process based on the question sample; The question sample, along with its regenerated predicted answer and reasoning process, is stored in the third target training set.

10. The method according to claim 9, characterized in that, The training of the first large model based on M training methods includes: The first large model to be trained is subjected to group relative policy optimization reinforcement learning training using the third objective training set.

11. A model training method, characterized in that, include: Input the question samples and answer samples from the candidate training set into the second large model to obtain the predicted answer and reasoning process generated by the second large model based on the question samples. The number of parameters of the second large model is greater than the number of parameters of the first large model to be trained. The predicted answers and answer samples of the question samples are evaluated using the third major model to obtain the evaluation results; Based on the credibility of the predicted answer indicated by the evaluation results, N target training sets are obtained using the candidate training sets, and the first large model is trained using M training methods, where N and M are integers greater than or equal to 2, and M is less than or equal to N. The target training sets include question samples, predicted answers, and reasoning processes corresponding to the credibility.

12. A question-and-answer device, characterized in that, include: The target answer generation module is used to input the unanswered question sent by the target object into the first large model and obtain the target answer generated by the first large model; And the first major model pre-training device, including: The prediction answer and reasoning process generation module is used to input question samples and answer samples from the candidate training set into the second large model to obtain the prediction answer and reasoning process generated by the second large model based on the question samples. The number of parameters of the second large model is greater than that of the first large model. The evaluation module is used to evaluate the predicted answers and answer samples of the question samples using the third major model, and obtain the evaluation results; The target training set generation module is used to obtain N target training sets based on the credibility of the predicted answer indicated by the evaluation result, using the candidate training set, and to train the first large model based on M training methods, where N and M are integers greater than or equal to 2, and M is less than or equal to N. The target training set includes question samples, predicted answers, and reasoning processes corresponding to the credibility.

13. A model training device, characterized in that, include: The prediction answer and reasoning process generation module is used to input the question samples and answer samples in the candidate training set into the second large model to obtain the prediction answer and reasoning process generated by the second large model based on the question samples. The number of parameters of the second large model is greater than the number of parameters of the first large model to be trained. The evaluation module is used to evaluate the predicted answers and answer samples of the question samples using the third major model, and obtain the evaluation results; The target training set generation module is used to obtain N target training sets based on the credibility of the predicted answer indicated by the evaluation result, using the candidate training set, and to train the first large model based on M training methods, where N and M are integers greater than or equal to 2, and M is less than or equal to N. The target training set includes question samples, predicted answers, and reasoning processes corresponding to the credibility.

14. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 11.

15. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 11.

Images