Model training method, question and answer method, device, equipment, medium and product

Abstract

The embodiment of the invention discloses a model training method, a question and answer method and device, model training equipment, a computer storage medium and a computer program product. The model training method comprises the steps of obtaining a plurality of first training samples and hidden variables corresponding to each first training sample; each first training sample comprises a query sample; combining each first training sample with the corresponding hidden variable to obtain a second training sample; processing the query sample in each first training sample by using a strategy model to obtain a first answer set; processing a query sample in each second training sample by using the strategy model to obtain a second answer set; and based on the plurality of first answer sets and the plurality of second answer sets, training the strategy model to obtain a target strategy model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, a question-answering method, an apparatus, a device, a computer storage medium, and a computer program product. Background Technology

[0002] In the e-commerce field, intelligent assistants are often used to handle complex tasks, such as multi-turn logical question answering and price calculation. To improve their reasoning ability and question-answering service quality, a large language model (LLM) is usually used as the reasoning core of the intelligent assistant, and reinforcement learning algorithms are used to post-train it to enhance the model's ability to handle complex tasks.

[0003] In related technologies, reinforcement learning algorithms such as Proximal Policy Optimization (PPO) are commonly used to train models. However, these algorithms rely too heavily on a single inference path during model training, leading to policy collapse and affecting the reliability of the model's output. Summary of the Invention

[0004] This application provides a model training method, a question-answering method, an apparatus, a model training device, a computer storage medium, and a computer program product.

[0005] The technical solution of this application is implemented as follows: A model training method, the method comprising: Obtain multiple first training samples and the latent variables corresponding to each first training sample; each first training sample includes a query sample. Each of the first training samples is combined with its corresponding latent variable to obtain a second training sample; The query samples in each of the first training samples are processed using a strategy model to obtain a first set of answers; The strategy model is used to process the query samples in each second training sample to obtain a second set of answers; The target policy model is obtained by training the policy model based on multiple sets of first answers and multiple sets of second answers.

[0006] A question-and-answer method, the method comprising: Retrieve target query information; The target query information is processed using a target strategy model to obtain a target answer set; The target strategy model is trained using the method provided by the aforementioned technical solution.

[0007] A model training device, the device comprising: The first acquisition module is used to acquire multiple first training samples and the latent variables corresponding to each first training sample; each first training sample includes a query sample. The combination module is used to combine each first training sample with the corresponding latent variable to obtain a second training sample; The first processing module is used to process the query samples in each of the first training samples using a strategy model to obtain a first answer set; and to process the query samples in each of the second training samples using the strategy model to obtain a second answer set. The training module is used to train the policy model based on multiple first answer sets and multiple second answer sets to obtain a target policy model. A question-and-answer device, the device comprising: The second acquisition module is used to acquire the target query information; The second processing module is used to process the target query information using the target strategy model to obtain the target answer set; The target strategy model is trained using the method provided by the aforementioned technical solution.

[0008] A model training device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the model training method or question-answering method provided by the aforementioned technical solution.

[0009] A computer storage medium storing a computer program; the computer program, when executed by a processor, can implement the model training method or question-answering method provided by the aforementioned technical solution.

[0010] A computer program product includes a computer program that, when executed by a processor, implements the model training method or question-answering method provided by one or more of the aforementioned technical solutions.

[0011] Because it is possible to obtain multiple first training samples and the latent variables corresponding to each first training sample; each first training sample includes a query sample; each first training sample is combined with the corresponding latent variable to obtain a second training sample; the query sample in each first training sample is processed using a policy model to obtain a first answer set; the query sample in each second training sample is processed using the policy model to obtain a second answer set; based on multiple first answer sets and multiple second answer sets, the policy model is trained to obtain a target policy model.

[0012] As can be seen, in this embodiment of the application, by introducing latent variables and combining them with the first training samples to generate the second training samples, the diversity of training samples can be improved; by using the first answer set and the second answer set together for model training, the model can be guided to learn more diverse reasoning paths, avoiding the policy collapse problem caused by single path dependence and improving the reliability of the model output results; in addition, the policy generated by latent variable conditionalization can enhance the model's exploration ability, achieve a balance between exploration and convergence during the optimization process, and improve the robustness of the model. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a model training method in an embodiment of this application. Figure 2 A flowchart illustrating another model training method provided in an embodiment of this application; Figure 3 A flowchart illustrating a question-and-answer method provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a question-and-answer device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application. Detailed Implementation

[0014] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present application and are not intended to limit the present application. Furthermore, the embodiments provided below are some embodiments for implementing the present application, and not all embodiments for implementing the present application. Unless otherwise specified, the technical solutions described in the embodiments of the present application can be implemented in any combination. It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element. It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence number of each process does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0015] Unless otherwise specified, any step in the embodiments of this application performed by the model training device may be executed by the processor of the model training device. It is also worth noting that the embodiments of this application do not limit the order in which the model training device performs the following steps. Furthermore, the methods used to process the data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the model training device; that is, when the model training device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0016] In some embodiments of this application, the model training method can be implemented using a processor in the model training device. The processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, and microprocessor.

[0017] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. Where applicable, user personal information has been subjected to de-identification and / or anonymization and / or encryption technical processing.

[0018] In this embodiment, the model training method can be applied to the reinforcement learning optimization scenario of the base model of the target domain intelligent assistant; here, the target domain can be the e-commerce domain, or other domains such as the financial domain, without specific limitations, and the following will use the e-commerce domain as an example for explanation; wherein, the base model refers to a general domain large model with general language understanding and generation capabilities, which is the basic platform for subsequent fine-tuning and application; in the case that the target domain is the e-commerce domain, the base model is also called the policy model.

[0019] In this embodiment of the application, the model training method can be executed by a model training device, which can be configured in a model training equipment; wherein, the model training equipment can be a server, a computing terminal or a distributed computing system, etc., without specific limitations.

[0020] Figure 1 This is a flowchart illustrating a model training method in an embodiment of this application, as shown below. Figure 1 As shown, the method may include the following steps: Step 100: Obtain multiple first training samples and the latent variables corresponding to each first training sample.

[0021] In this embodiment, the multiple first training samples are sample data used to train the policy model; here, the policy model can be any open-source large language model, and its type is not specifically limited here. For example, it can be a Qwen series or Llama series model.

[0022] For example, multiple first training samples can be obtained from the historical Q&A database and customer service records of the e-commerce platform; wherein each first training sample may include a query sample.

[0023] For example, the query sample can be a question that the user actually entered when interacting with the intelligent assistant during a historical time period, that is, the question that the user actually asked the intelligent assistant; for example, for intelligent assistants in the e-commerce field, the query sample can be "What is the final price of the product under multiple discounts" or "How to choose the most suitable coupon combination", etc.

[0024] In this embodiment of the application, each first training sample corresponds to a latent variable; wherein, the latent variable may be a random variable sampled from a prior distribution; for example, the prior distribution may be a discrete uniform distribution or a Gaussian distribution with equal probability distribution, and no specific limitation is made here.

[0025] Here, there are no specific restrictions on the form of latent variables. For example, latent variables can be in numerical form, such as any number from 0 to 9, or they can be in the form of more specific prompts, such as calculation-based answers, recommendation-based answers, or explanation-based answers, to guide the model to generate answers of a specific style.

[0026] For example, embodiments of this application can support dynamic adjustment of latent variables under different user intentions, styles or task types to achieve diversified training sample generation.

[0027] Understandably, the setting of latent variables plays a role in regulating the reasoning process of the policy model, enabling the policy model to generate different reasoning paths under different latent variable conditions, which helps to improve the model's exploration capabilities.

[0028] Step 101: Combine each first training sample with its corresponding latent variable to obtain the second training sample.

[0029] In this embodiment of the application, for each of the multiple first training samples, the first training sample is combined with the corresponding latent variable to obtain the second training sample.

[0030] It should be noted that combining the first training sample with the corresponding latent variable means combining the query sample in the first training sample with the corresponding latent variable to obtain the query sample in the second training sample; that is, the query sample in the second training sample is obtained by adding latent variables to the query sample in the first training sample, while the rest remains unchanged.

[0031] Here, there are no specific restrictions on the combination of the first training sample and the latent variable. For example, the combination can be to directly append the latent variable to the query sample of the first training sample, or it can be any other combination.

[0032] For example, as can be seen from the above, the latent variable can be in numerical form or in the form of a prompt. It should be noted that when the latent variable is in numerical form, it can be encoded into a vector representation and then appended to the query sample of the first training sample to obtain the query sample of the second training sample. When the latent variable is in the form of a prompt, the latent variable in the form of a prompt can be directly appended to the query sample of the first training sample to obtain the query sample of the second training sample.

[0033] Understandably, attaching latent variables to the query samples in the first training sample ensures that the policy model can generate diverse inference paths based on the different latent variables when processing the same query sample. For example, for the query sample "What is the final price of the product under multiple discounts?", the policy model might generate an inference path that calculates the minimum purchase amount first and then the discount, or it might generate an inference path that calculates the member price first and then applies the coupon. Therefore, introducing latent variables helps improve the exploration capability of the policy model, prevents the policy model from becoming overly dependent on a single path, and effectively improves the model's robustness and generalization ability.

[0034] Step 102: Process the query samples in each first training sample using the policy model to obtain the first answer set; process the query samples in each second training sample using the policy model to obtain the second answer set.

[0035] For example, the strategy model can be a base model used by intelligent assistants in the e-commerce field that has not been optimized by reinforcement learning, and has basic question-answering and logical reasoning capabilities.

[0036] In this embodiment, if each first training sample is input into the policy model, the policy model will process the query samples in the first training samples based on its internal parameters to generate a set of responses, i.e., a first answer set. The first answer set may include multiple predicted answers, which may be similar or different. Here, predicted answers are also referred to as predicted inference paths, predicted inference trajectories, predicted inference results, or predicted answers, etc.

[0037] In this embodiment, if each second training sample is input into the policy model, the policy model will process the query samples in the second training samples based on its internal parameters to generate a set of responses, i.e., the second answer set. Here, the second answer set also includes multiple predicted answers. The difference between the second and first answer sets is that the predicted answers in the second answer set are generated by introducing latent variables based on the first training samples.

[0038] Understandably, by introducing latent variables, the strategy model can generate different predicted answers based on different latent variables when faced with the same query sample. For example, for the same query sample "how to choose the most suitable coupon combination", when the latent variable is an explanatory answer, the strategy model may analyze various discount rules in detail and provide a detailed calculation process; while when the latent variable is a recommendation answer, the strategy model may give a concise recommendation result.

[0039] As can be seen, in this embodiment of the application, the introduction of latent variables helps the policy model to explore more possible reasoning paths, thereby enhancing the policy model's ability to cope with complex tasks.

[0040] Step 103: Based on multiple sets of first answers and multiple sets of second answers, train the policy model to obtain the target policy model.

[0041] In this embodiment of the application, after obtaining multiple first answer sets and multiple second answer sets according to the above steps, the multiple first answer sets and multiple second answer sets can be used to train the policy model to obtain the target policy model.

[0042] In some embodiments, training a policy model to obtain a target policy model based on multiple first answer sets and multiple second answer sets may include: determining a policy loss function and a mutual information loss function based on multiple first answer sets and multiple second answer sets; constructing a target loss function for the policy model based on the policy loss function and the mutual information loss function; and training the policy model using the target loss function to obtain the target policy model.

[0043] In this embodiment of the application, the policy loss function and the mutual information loss function can be determined respectively based on the first answer set corresponding to multiple first training samples and the second answer set corresponding to multiple second training samples, that is, multiple first answer sets and multiple second answer sets.

[0044] For example, policy loss functions typically aim to maximize rewards. Mutual information loss functions, on the other hand, are regularization terms introduced to enhance exploration diversity; they quantify the mutual information between the inference trajectory and latent variables.

[0045] In this embodiment, structured and stable exploration can be achieved by maximizing the mutual information between the inference path and the latent variables, which helps to overcome the problems of policy collapse and entropy regularization instability during model training. By simultaneously considering the policy loss function and the mutual information loss function, the model's exploration and convergence can be balanced, ensuring that the model has sufficient diversity while maintaining high accuracy on each inference path.

[0046] In some embodiments, determining a policy loss function based on multiple first answer sets and multiple second answer sets may include: processing answer samples in each first answer set and the first training samples corresponding to the first answer set using a reward function to obtain a first reward set; processing answer samples in each second answer set and the second training samples corresponding to the second answer set using a reward function to obtain a second reward set; and determining a policy loss function based on the reward values ​​in the multiple first reward sets and the multiple second reward sets.

[0047] In this embodiment of the application, the first answer set includes multiple predicted answers, and the first reward set includes the reward value of each predicted answer in the first answer set; similarly, the second answer set includes multiple predicted answers, and the second reward set includes the reward value of each predicted answer in the second answer set.

[0048] It should be noted that each first training sample includes not only the query sample but also the answer sample corresponding to the query sample; here, the answer sample refers to the standard answer that matches the query sample.

[0049] For example, the reward function is used to score multiple predicted answers generated by the policy model; that is, each predicted answer is assigned a reward value, which reflects the degree of matching between the predicted answer and the answer sample corresponding to the query sample.

[0050] In this embodiment of the application, a reward function can be used to process the answer samples in the first training samples corresponding to each first answer set to obtain a first reward set.

[0051] For example, for each first answer set, the first answer set (multiple predicted answers) and the answer samples (standard answers) in the first training samples corresponding to the first answer set can be input into the reward function. The reward function can evaluate the quality difference between each predicted answer in the first answer set and the answer sample (standard answer), and generate a corresponding reward value for each predicted answer in the first answer set, thereby obtaining the first reward set.

[0052] In this embodiment of the application, a reward function can also be used to process the answer samples in the second training samples corresponding to each second answer set and the second answer set to obtain a second reward set.

[0053] Similarly, for each second training sample, the second answer set (multiple predicted answers) and the answer samples (standard answers) in the second training samples corresponding to the second answer set can be input into the reward function. The reward function can evaluate the quality difference between each predicted answer in the second answer set and the answer sample (standard answer), and generate a corresponding reward value for each predicted answer in the second answer set, thereby obtaining the second reward set.

[0054] In this embodiment of the application, after obtaining multiple first reward sets and multiple second reward sets, the policy loss function can be determined based on the reward values ​​in the multiple first reward sets and the reward values ​​in the multiple second reward sets.

[0055] For example, the process of determining the policy loss function based on the reward values ​​in multiple first reward sets and multiple second reward sets can be as follows: determine the advantage value of each predicted answer in the corresponding first answer set based on the reward value in each first reward set; determine the advantage value of each predicted answer in the corresponding second answer set based on the reward value in each second reward set; and determine the policy loss function based on the advantage values ​​of each predicted answer in the multiple first answer sets and the advantage values ​​of each predicted answer in the multiple second answer sets.

[0056] Here, no specific limitation is made on the method of determining the advantage value. For example, the within-group standardization method can be used to determine the advantage value, or other methods can be used.

[0057] For example, assuming that the odds value is determined using the within-group standardization method, it can be determined using formula (1): (1) in, Indicates the first The odds value of each predicted answer Indicates the first The reward value for each predicted answer, This represents the average reward value within the group, that is, the average reward value of all predicted answers in the first answer set. This represents the standard deviation of the within-group reward value. For example, using within-group standardization to determine the advantage value can avoid training an additional value network and remove systematic bias.

[0058] In this embodiment of the application, after obtaining the advantage value of each predicted answer in each first answer set and the advantage value of each predicted answer in each second answer set according to the above formula (1), the strategy loss function can be determined based on the advantage values ​​of each predicted answer in multiple first answer sets and the advantage values ​​of each predicted answer in multiple second answer sets. It can be expressed by formula (2): (2) in, For the expectation operator, The policy ratio represents the new policy. Compared to the old strategy The probability ratio when generating the i-th predicted answer, old strategy These are the model parameters before the update, and the new strategy. These are the updated model parameters; For the clipping function, It is a hyperparameter that defines the clipping range. It is usually set to a small value, such as 0.1 or 0.2, etc., and no specific limit is made here.

[0059] As can be seen, in this embodiment of the application, the reward function is used to quantify and score each predicted answer output by the strategy model, and a strategy loss function is constructed based on the scoring results. This helps the model adjust its own parameters in real time according to the reward, thereby better adapting to the needs of complex tasks.

[0060] In this embodiment of the application, in addition to determining the policy loss function, a mutual information loss function is also determined. The process of determining the mutual information loss function is illustrated below.

[0061] It should be noted that the main purpose of introducing the mutual information loss function is to maximize the mutual information between the inference trajectory and the latent variables, thereby guiding the policy model to conduct structured and stable exploration and overcoming the policy collapse and entropy regularization instability problems that exist in traditional policy models during training. For ease of understanding, the technical principles behind introducing the mutual information loss function will be explained below.

[0062] In this embodiment of the application, the strategy The exploration process is designed as a conditionalization of a latent variable z. z can be a value derived from a prior distribution. The discrete variables sampled in the middle are used to adjust the randomness of the policy. The core idea of ​​this application is to maximize the inference trajectory. With the hidden variable z in a given initial state The mutual information under the following conditions can be represented by expression (3): (3) Maximize mutual information This means that the policy model is incentivized to learn a conditional policy. This allows different latent variables z to produce significantly different inference trajectories. This effectively decomposes the single-strategy exploration task into learning a diverse set of sub-strategies, each anchored by a specific z-value. According to information theory principles, mutual information can be decomposed through the relationship between marginal entropy and conditional entropy, as shown in equation (4): (4) in, Indicates a given initial state Under the conditions, the reasoning trajectory marginal entropy Indicates a given initial state Given the latent variable z, the inference trajectory The conditional entropy. Understandably, marginal entropy This indicates that the strategy is as follows, without considering the latent variable z. Generate reasoning trajectory Given the overall uncertainty, maximizing this factor will prompt the strategy This generates more diverse reasoning trajectories and avoids getting trapped in local optima. Conditional entropy. This represents the sub-policy given a specific latent variable z. Output inference trajectory Minimizing this factor will enhance each sub-strategy, thus reducing uncertainty. The certainty, that is, the high confidence convergence to a specific behavioral pattern.

[0063] Furthermore, maximizing mutual information The objective is transformed into the optimization objective of the strategy model. As shown in formula (5): (5) It can be seen that the above formula (5) cleverly reconciles the contradiction between exploration and convergence. Specifically, it minimizes the conditional entropy. This ensures that specific sub-policies converge to the solution space with high confidence, avoiding policy collapse. Furthermore, since this term is a "minimization" objective, it does not need to introduce conflict into the optimization process like traditional entropy maximization; it achieves this by maximizing marginal entropy. This ensures sufficient diversity across the entire strategy set, enabling structured exploration. Below, we will provide an illustrative explanation of the process for determining the mutual information loss function based on the aforementioned technical principles.

[0064] In some embodiments, determining the mutual information loss function based on multiple first answer sets and multiple second answer sets may include: processing each first answer set using a policy model to obtain the marginal entropy of each predicted answer in the first answer set; processing each second answer set using a policy model to obtain the conditional entropy of each predicted answer in the second answer set; and determining the mutual information loss function based on the marginal entropy of each predicted answer in the multiple first answer sets and the conditional entropy of each predicted answer in the multiple second answer sets.

[0065] For example, marginal entropy is used to measure the uncertainty of the policy output; in this embodiment of the application, the policy model can be used to process each first answer set to obtain the marginal entropy of each predicted answer in the first answer set; as can be seen from the above, by maximizing the marginal entropy of the predicted answer, the exploration breadth of the policy model can be ensured.

[0066] For example, conditional entropy is used to measure the uncertainty of policy output given a specific latent variable; in this embodiment of the application, a policy model can be used to process each second set of answers to obtain the conditional entropy of each predicted answer in the second set of answers; as can be seen from the above, minimizing the conditional entropy of the predicted answer can improve the answer confidence of the policy model.

[0067] Furthermore, after obtaining the marginal entropy of each predicted answer in multiple first answer sets and the conditional entropy of each predicted answer in multiple second answer sets, the mutual information loss function can be determined based on these two pieces of information. As shown in formula (5) above, it will not be repeated here.

[0068] For example, the marginal entropy and conditional entropy used to construct the mutual information loss function can be estimated by Monte Carlo approximation or by other means, without being specifically limited here.

[0069] In this embodiment of the application, determining the mutual information loss function based on the marginal entropy of each predicted answer in multiple first answer sets and the conditional entropy of each predicted answer in multiple second answer sets may include: smoothing the marginal entropy of each predicted answer in each first answer set to obtain a smoothed marginal entropy; and determining the mutual information loss function based on the smoothed marginal entropy of each predicted answer in multiple first answer sets and the conditional entropy of each predicted answer in multiple second answer sets.

[0070] For example, smoothing is an operation used to stabilize numerical computation, aiming to mitigate the logarithmic singularity problem caused by traditional entropy maximization methods. Understandably, marginal entropy reflects the overall uncertainty of the policy output; if used directly for training without smoothing, it may lead to gradient explosion or numerical overflow. Therefore, after obtaining the marginal entropy of each predicted answer in the first answer set, smoothing can be applied to the marginal entropy of each predicted answer in the first answer set.

[0071] Here, no specific limitation is made on the method of smoothing. For example, smoothing can be performed by introducing weighting coefficients, or by other methods.

[0072] For example, if a weighting coefficient is introduced... Smoothing the original marginal entropy yields the corresponding mutual information loss function. It can be shown in formula (6): (6) Understandably, smoothing not only preserves the exploration level information represented by marginal entropy, but also effectively solves the numerical instability problem that may occur during the optimization process.

[0073] In some embodiments, determining the mutual information loss function based on the smoothed marginal entropy of each predicted answer in a plurality of first answer sets and the conditional entropy of each predicted answer in a plurality of second answer sets may include: subtracting the smoothed marginal entropy of each predicted answer in a plurality of first answer sets and the conditional entropy of each predicted answer in a plurality of second answer sets to obtain an entropy difference; and determining the mutual information loss function based on the target regularization coefficient and the entropy difference.

[0074] For example, the smoothed marginal entropy reflects the overall uncertainty of the policy without considering latent variables, while the conditional entropy represents the uncertainty of the policy output given specific latent variables; by subtracting the two, the entropy difference can reflect the degree of influence of latent variables on the policy output.

[0075] For example, the target regularization coefficient is a hyperparameter that can be a fixed value or dynamically adjusted according to the training process to adapt to the training needs at different stages. For instance, in the early stages of training, the target regularization coefficient can be set to a larger value, which helps to improve the exploration ability of the policy model; while in the later stages of training, the system gradually reduces the target regularization coefficient to promote further optimization of the policy model on the existing inference path, thereby improving the accuracy and stability of inference.

[0076] It should be noted that the target regularization coefficient is used to control the weighting of the entropy difference in the target loss function. The target regularization coefficient determines the degree to which the policy model focuses on policy diversity during the optimization process; the larger the target regularization coefficient, the more the policy model tends to retain diverse inference paths, and vice versa, it focuses more on the convergence and accuracy of each inference path.

[0077] For example, after obtaining the target regularization coefficient Then, the target regularization coefficient can be used. The mutual information loss function is determined by the entropy difference, as shown in formula (7): (7) in, Marginal entropy, corresponding to the above ; For conditional entropy, corresponding to the above ; This represents the entropy difference.

[0078] Understandably, by combining the target regularization coefficient and the entropy difference to construct a mutual information loss function, the model can achieve both diversified exploration and high-confidence convergence during training, avoiding the numerical instability problem caused by the traditional entropy maximization method.

[0079] In this embodiment of the application, constructing the target loss function of the policy model based on the policy loss function and the mutual information loss function may include: summing the policy loss function and the mutual information loss function to obtain the target loss function of the policy model.

[0080] For example, after obtaining the policy loss function and mutual information loss function Then, the two can be summed to obtain the target loss function of the policy model. As shown in formula (8): (8) In essence, the policy loss function is a loss function used during reinforcement learning training to measure the difference between the current policy and the desired policy of the policy model. The policy loss function typically reflects the model's output response, i.e., the difference between the predicted answer and the standard answer.

[0081] For example, the policy loss function is mainly derived from the Group Relative Policy Optimization (GRPO) method, which evaluates the merits of different inference trajectories by calculating the intra-group advantage, thereby guiding the update direction of the policy model.

[0082] For example, as described above, the mutual information loss function is an information-theory-related regularization term used to measure the correlation between latent variables and inference trajectories. By maximizing mutual information, the policy model can be encouraged to generate diverse inference paths under different latent variable conditions, while ensuring that each inference path has a high confidence level. The mutual information loss function consists of two parts: maximizing marginal entropy and minimizing conditional entropy. The maximization of marginal entropy promotes the model to explore more possibilities, while the minimization of conditional entropy ensures convergence during model training.

[0083] As can be seen from the above formula (8), the objective loss function is the overall objective function obtained by summing the policy loss function and the mutual information loss function, which is used to guide the updating of policy model parameters. It can be seen that the objective loss function not only considers the model performance (through the policy loss function), but also introduces a structured exploration mechanism (through the mutual information loss function), thereby achieving a balance between diversity and accuracy.

[0084] Furthermore, after obtaining the target loss function of the policy model... Then, the parameters of the policy model can be updated using the gradient ascent method shown in expression (9). The training continues until the convergence condition is met, at which point the trained target policy model is obtained.

[0085] (9) in, The learning rate controls the update step size; The gradients, including the policy gradient and the regularization gradient, are calculated automatically using differentiation.

[0086] Here, the convergence condition can be that the model performance no longer improves significantly, or that the preset maximum number of training epochs is reached. In practical applications, the training samples can be divided into multiple batches, and the gradient can be calculated and the model parameters updated independently for each batch.

[0087] It should be noted that the embodiments of this application do not impose specific limitations on the settings of the various parameters involved in the policy model. For example, the settings of the various parameters can be: learning rate. 1×10 -6 The batch size is 128, the number of predicted answers in the answer set is 16, the latent variable space size is 10, and the target regularization coefficient is... The weighting coefficient is 0.005. It is 0.5.

[0088] As can be seen, in this embodiment of the application, by introducing latent variables and mutual information regularization mechanism, the needs of exploration and convergence can be met at the same time, realizing interpretable, stable and efficient diversified exploration training, which is suitable for reinforcement learning optimization scenarios of intelligent assistants in the e-commerce field.

[0089] To better illustrate the purpose of this application, further examples are provided based on the above embodiments.

[0090] Figure 2 A flowchart illustrating another model training method provided in this application embodiment is shown below. Figure 2 As shown, the method includes the following steps: First, the policy model receives two sets of inputs: the query sample q of the first training sample, and the query sample of the second training sample, which is the combination of the query sample q and the latent variable z. The strategy model outputs two sets of responses based on the two sets of inputs, namely the first set of responses. And the second set of answers Understandably, to ensure the effectiveness of model training, both the first and second training samples are multiple. When there are many training samples, training can be done in batches.

[0091] Then, the reward function is applied to the first set of answers. The corresponding answer samples are processed to obtain the first answer set. The reward value for each predicted answer in the first set; simultaneously, the reward function is applied to the second set of answers. The corresponding answer samples are processed to obtain the second answer set. The reward value for each predicted answer is determined; then the advantage value for each predicted answer is determined based on the reward value; and finally, the strategy loss function is determined based on the advantage value.

[0092] Next, the first set of answers And the second set of answers The marginal entropy and conditional entropy are respectively input into the policy model to obtain the corresponding marginal entropy and conditional entropy; the mutual information loss function is obtained based on the marginal entropy and conditional entropy.

[0093] Finally, the policy loss function and the mutual information loss function are summed to obtain the target loss function of the policy model; the policy model is then trained using the target loss function. The specific training process has been described in the above embodiments and will not be repeated here to avoid repetition.

[0094] Understandably, the policy loss function comes from the GRPO loss, and the mutual information loss function is used to measure the degree of correlation between the latent variable z and the inference trajectory. The latent variable z guides the model to generate different inference trajectories. For example, z=1 guides the model to calculate the discount before the full reduction, and z=2 guides the model to calculate the member price first. Using the target loss function to train the policy model can simultaneously ensure the diversity of inference paths and the accuracy of each inference path.

[0095] Figure 3 This is a flowchart illustrating a question-and-answer method provided in an embodiment of this application, such as... Figure 3 As shown, the method includes the following steps: Step 300: Obtain the target query information.

[0096] Here, the target query information can be the query information entered by the target user who is currently interacting with the intelligent assistant through the terminal; this query information can be understood as the question information that the target user wants to ask.

[0097] Here, the terminal can be a mobile phone, tablet computer, IoT device, smart wearable device, etc., without specific limitations.

[0098] For example, after obtaining the target query information, some standardization processing can be performed before it is input into the model to facilitate subsequent model processing, such as removing special characters and unifying the encoding format, etc., without specific limitations here.

[0099] Step 301: Process the query information using the target strategy model to obtain the target answer set.

[0100] Here, the target answer set can include multiple answer information corresponding to the target query information.

[0101] For example, after obtaining the target set of answers, the answer information can be directly fed back to the target user, or the target set of answers can be further filtered according to set conditions to obtain answer information that better meets the user's needs and then fed back to the target user.

[0102] Correspondingly, the target set of answers or answer information that better meets the user's needs can be sent to the target user's terminal, so that the target user can obtain the answer information.

[0103] In this embodiment of the application, the target strategy model is obtained according to the above steps 100 to 103; it will not be repeated here; by using the target strategy model to process the query information, the corresponding answer information can be obtained.

[0104] As can be seen from the embodiments of this application, if the target strategy model is applied to a target domain, such as an intelligent assistant in the e-commerce field, the target strategy model will accurately respond to each user's input question, thereby improving the user's interactive experience.

[0105] Based on the foregoing embodiments, this application also provides a model training device, such as... Figure 4 As shown, the model training device may include: a first acquisition module 400, a combination module 401, a first processing module 402, and a training module 403, wherein: The first acquisition module 400 is used to acquire multiple first training samples and the latent variables corresponding to each first training sample; each first training sample includes a query sample. The combination module 401 is used to combine each first training sample with the corresponding latent variable to obtain a second training sample; The first processing module 402 is used to process the query samples in each first training sample using a strategy model to obtain a first answer set; and to process the query samples in each second training sample using a strategy model to obtain a second answer set. Training module 403 is used to train the policy model based on multiple first answer sets and multiple second answer sets to obtain the target policy model.

[0106] In some embodiments, the training module 403 is further configured to: Based on multiple sets of first responses and multiple sets of second responses, the policy loss function and the mutual information loss function are determined. Based on the policy loss function and the mutual information loss function, construct the target loss function of the policy model; The target policy model is obtained by training the policy model using the target loss function.

[0107] In some embodiments, each first training sample further includes an answer sample corresponding to the query sample, and the training module 403 is further configured to: The reward function is used to process the answer samples in the first training sample corresponding to each first answer set to obtain the first reward set; the first answer set includes multiple predicted answers, and the first reward set includes the reward value of each predicted answer in the first answer set; The reward function is used to process the answer samples in the second training samples corresponding to each second answer set and the second answer set to obtain the second reward set; the second answer set includes multiple predicted answers, and the second reward set includes the reward value of each predicted answer in the second answer set; The policy loss function is determined based on the reward values ​​in multiple first reward sets and multiple second reward sets.

[0108] In some embodiments, the training module 403 is further configured to: The policy model is used to process each first set of answers to obtain the marginal entropy of each predicted answer in the first set of answers; The policy model is used to process each second set of answers to obtain the conditional entropy of each predicted answer in the second set of answers; The mutual information loss function is determined based on the marginal entropy of each predicted answer in multiple first answer sets and the conditional entropy of each predicted answer in multiple second answer sets.

[0109] In some embodiments, the training module 403 is further configured to: The marginal entropy of each predicted answer in each first answer set is smoothed to obtain the smoothed marginal entropy; The mutual information loss function is determined based on the smoothed marginal entropy of each predicted answer in multiple first answer sets and the conditional entropy of each predicted answer in multiple second answer sets.

[0110] In some embodiments, the training module 403 is further configured to: The entropy difference is obtained by subtracting the smoothed marginal entropy of each predicted answer in multiple first answer sets from the conditional entropy of each predicted answer in multiple second answer sets. The mutual information loss function is determined based on the target regularization coefficient and the entropy difference.

[0111] In some embodiments, the training module 403 is further configured to: The objective loss function of the policy model is obtained by summing the policy loss function and the mutual information loss function.

[0112] Figure 5 This is a schematic diagram of the composition structure of a question-and-answer device according to an embodiment of this application, as shown below. Figure 5 As shown, the device includes: a second acquisition module 501 and a second processing module 502, wherein: The second acquisition module 501 is used to acquire target query information; The second processing module 502 is used to process the target query information using the target strategy model to obtain the target answer set.

[0113] The target strategy model is trained using the method provided by the aforementioned technical solution.

[0114] In practical applications, the first acquisition module 400, the combination module 401, the first processing module 402 and the training module 403, the second acquisition module 501 and the second processing module 502 can all be implemented by a processor located in the model training device. The processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller and microprocessor.

[0115] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0116] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] Specifically, the computer program instructions corresponding to a model training method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer program instructions corresponding to a model training method in the storage media are read or executed by a model training device, any of the model training methods in the aforementioned embodiments can be implemented.

[0118] Based on the same technical concept as the foregoing embodiments, see Figure 6 The illustration shows a model training device 600 provided in an embodiment of this application, which may include: a memory 601 and a processor 602; wherein, the memory 601 is used to store computer programs and data; the processor 602 is used to execute the computer programs stored in the memory to implement any of the model training methods or question-answering methods in the foregoing embodiments.

[0119] In practical applications, the aforementioned memory 601 can be volatile memory, such as RAM; or non-volatile memory, such as ROM, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, providing instructions and data to the processor 602. The aforementioned processor 602 can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, or microprocessor. It is understood that for different model training devices, the electronic device used to implement the above processor function can also be other, and this application embodiment does not specifically limit it.

[0120] This application provides a computer program product, including a computer program that, when executed by a processor, implements any of the model training methods or question-answering methods described in the foregoing embodiments.

[0121] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. Specific implementations can be referred to the descriptions of the above method embodiments, and for brevity, will not be repeated here. The descriptions of the various embodiments above tend to emphasize the differences between them; their similarities or commonalities can be referred to each other, and for brevity, will not be repeated here.

[0122] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict. The features disclosed in the various product embodiments provided in this application can be arbitrarily combined to obtain new product embodiments without conflict. The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined to obtain new method embodiments or device embodiments without conflict.

[0123] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0124] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable model training apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable model training apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable model training device, causing a series of operational steps to be executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a plurality of first training samples and corresponding hidden variables of each first training sample; each first training sample comprises a query sample; combining each first training sample and the corresponding hidden variable to obtain a second training sample; processing the query sample in each first training sample using a strategy model to obtain a first answer set; processing the query sample in each second training sample using the strategy model to obtain a second answer set; training the strategy model based on a plurality of first answer sets and a plurality of second answer sets to obtain a target strategy model.

2. The method of claim 1, wherein, The method comprises: determining a strategy loss function and a mutual information loss function based on a plurality of first answer sets and a plurality of second answer sets; constructing a target loss function of the strategy model based on the strategy loss function and the mutual information loss function; training the strategy model using the target loss function to obtain the target strategy model.

3. The method of claim 2, wherein, Each first training sample further comprises a corresponding answer sample of the query sample, and determining a strategy loss function based on a plurality of first answer sets and a plurality of second answer sets comprises: processing each first answer set and the answer sample in the first training sample corresponding to the first answer set using a reward function to obtain a first reward set; the first answer set comprises a plurality of predicted answers, and the first reward set comprises a reward value of each predicted answer in the first answer set; processing each second answer set and the answer sample in the second training sample corresponding to the second answer set using the reward function to obtain a second reward set; the second answer set comprises a plurality of predicted answers, and the second reward set comprises a reward value of each predicted answer in the second answer set; determining the strategy loss function based on reward values in a plurality of first reward sets and reward values in a plurality of second reward sets.

4. The method of claim 2, wherein, Determining a mutual information loss function based on a plurality of first answer sets and a plurality of second answer sets comprises: processing each first answer set using the strategy model to obtain a marginal entropy of each predicted answer in the first answer set; processing each second answer set using the strategy model to obtain a conditional entropy of each predicted answer in the second answer set; determining the mutual information loss function based on the marginal entropy of each predicted answer in a plurality of first answer sets and the conditional entropy of each predicted answer in a plurality of second answer sets.

5. The method of claim 4, wherein, Determining the mutual information loss function based on the marginal entropy of each predicted answer in a plurality of first answer sets and the conditional entropy of each predicted answer in a plurality of second answer sets comprises: smoothing the marginal entropy of each predicted answer in each first answer set to obtain a smoothed marginal entropy; Determine the mutual information loss function based on the smoothed marginal entropy of each predicted answer in the plurality of first answer sets and the conditional entropy of each predicted answer in the plurality of second answer sets.

6. The method of claim 5, wherein, The determination of the mutual information loss function based on the smoothed marginal entropy of each predicted answer in the plurality of first answer sets and the conditional entropy of each predicted answer in the plurality of second answer sets comprises: Differencing the smoothed marginal entropy of each predicted answer in the plurality of first answer sets and the conditional entropy of each predicted answer in the plurality of second answer sets to obtain an entropy difference value; Determine the mutual information loss function based on the target regularization coefficient and the entropy difference value.

7. The method according to any one of claims 2 to 6, characterized in that, The construction of the target loss function of the policy model based on the policy loss function and the mutual information loss function comprises: Summing the policy loss function and the mutual information loss function to obtain the target loss function of the policy model.

8. A question and answer method characterized by, The method comprises: Obtaining target query information; Processing the target query information using a target policy model to obtain a target answer set; The target policy model is trained according to the method of any one of claims 1 to 7.

9. A model training apparatus, comprising: The device comprises: A first obtaining module configured to obtain a plurality of first training samples and an implicit variable corresponding to each first training sample; each first training sample comprises a query sample; A combination module configured to combine each first training sample with the corresponding implicit variable to obtain a second training sample; A first processing module configured to process the query sample in each first training sample using a policy model to obtain a first answer set, and process the query sample in each second training sample using the policy model to obtain a second answer set; A training module configured to train the policy model based on the plurality of first answer sets and the plurality of second answer sets to obtain a target policy model.

10. A question answering apparatus characterized by comprising: The device comprises: A second obtaining module configured to obtain target query information; A second processing module configured to process the target query information using a target policy model to obtain a target answer set; The target policy model is trained according to the method of any one of claims 1 to 7.

11. A model training device comprising: The model training device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the method of any one of claims 1 to 8.

12. A computer storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method of any one of claims 1 to 8.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 8.

Images