Task processing model training method, role playing model training method, and task processing method

Abstract

The embodiment of the specification provides a task processing model training method, a role playing model training method and a task processing method, wherein the task processing model training method comprises the following steps: acquiring a plurality of sample reply contents based on sample dialogue data by using a task processing model; performing comparative analysis on the plurality of sample reply contents to obtain reply indexes corresponding to the plurality of sample reply contents respectively, wherein the reply index is used to measure the quality of the corresponding sample reply content; and training the task processing model according to the reply index to obtain a trained task processing model. By simultaneously generating and comparing a plurality of sample reply contents, a reply index with higher distinguishability and stability is generated, the judgment deviation caused by the ambiguous standard when scoring a single sample is avoided, a more objective and accurate basis is provided for the training process, and the efficiency and accuracy of the task processing model training process are significantly improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the field of artificial intelligence technology, and in particular to a task processing model training method, a role-playing model training method and a task processing method. BACKGROUND

[0002] With the continuous progress of artificial intelligence technology, reinforcement learning, as an important method to realize autonomous decision-making of an agent, has been widely applied in natural language processing, dialogue systems, text generation and other fields. In reinforcement learning, quality evaluation can be performed on the text generated by the task processing model to provide learning signals for the training process of the task processing model.

[0003] Currently, the traditional evaluation modeling method usually relies on large models to evaluate a single sample, resulting in unstable evaluation results and seriously affecting the efficiency and accuracy of task processing model training. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a task processing model training method. One or more embodiments of the present specification also relate to a role-playing model training method, a task processing method, a request processing method based on a task processing model, a task platform, a task processing model training device, a role-playing model training device, a task processing device, a request processing device based on a task processing model, a computing device, an electronic device, a computer-readable storage medium and a computer program product, to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a task processing model training method is provided, comprising:

[0006] obtaining a plurality of sample reply contents based on sample dialogue data by using a task processing model;

[0007] performing comparative analysis on the plurality of sample reply contents to obtain reply indicators respectively corresponding to the plurality of sample reply contents, wherein the reply indicators are used to measure the quality of the corresponding sample reply contents;

[0008] training the task processing model according to the reply indicators to obtain a trained task processing model.

[0009] The task processing model training method provided by one embodiment of the present specification comprises: obtaining a plurality of sample reply contents based on sample dialogue data by using a task processing model; performing comparative analysis on the plurality of sample reply contents to obtain reply indexes corresponding to the plurality of sample reply contents respectively, wherein the reply indexes are used to measure the quality of the corresponding sample reply contents; and training the task processing model according to the reply indexes to obtain a trained task processing model. By generating and comparing a plurality of sample reply contents at the same time, the advantages and disadvantages of the replies can be accurately identified in the relative comparison, so as to generate a reply index with higher discrimination and stability, avoid the judgment deviation caused by the fuzzy standard when scoring a single sample, provide a more objective and accurate basis for the training process, and significantly improve the efficiency and accuracy of the task processing model training process. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 FIG. 1 is a flowchart of a task processing model training method according to one embodiment of the present specification;

[0011] Figure 2 FIG. 2 is a processing process diagram of a task processing model training method according to one embodiment of the present specification;

[0012] Figure 3 FIG. 3 is a flowchart of a role-playing model training method according to one embodiment of the present specification;

[0013] Figure 4 FIG. 4 is a flowchart of a task processing method according to one embodiment of the present specification;

[0014] Figure 5 FIG. 5 is an architectural diagram of a task processing system according to one embodiment of the present specification;

[0015] Figure 6 FIG. 6 is a flowchart of a request processing method based on a task processing model according to one embodiment of the present specification;

[0016] Figure 7 FIG. 7 is a structural diagram of a task platform according to one embodiment of the present specification;

[0017] Figure 8 FIG. 8 is a structural diagram of a task processing model training device according to one embodiment of the present specification;

[0018] Figure 9 FIG. 9 is a structural diagram of a role-playing model training device according to one embodiment of the present specification;

[0019] Figure 10 FIG. 10 is a structural diagram of a task processing device according to one embodiment of the present specification;

[0020] Figure 11 FIG. 1 is a structural schematic diagram of a request processing apparatus based on a task processing model according to an embodiment of the present specification;

[0021] Figure 12 FIG. 2 is a structural block diagram of a computing device according to an embodiment of the present specification;

[0022] Figure 13 FIG. 3 is a structural block diagram of an electronic device according to an embodiment of the present specification. DETAILED DESCRIPTION

[0023] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the drawings, and it is understood that the scope of the present specification is not limited to the details of the description.

[0024] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0025] It is to be understood that although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, a first item could be termed a second item, and, similarly, a second item could be termed a first item without departing from the scope of one or more embodiments of the present specification. As used herein, the term "if' can be construed to mean "when" or "in response to determining" or "in response to a determination" or "in response to the fact that...."

[0026] In addition, it should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present specification are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or rejection.

[0027] In one or more embodiments of the present specification, a large model refers to a deep learning model with a large number of model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, thousands of billions or even tens of billions of model parameters. The large model can also be called a foundation model. Through large-scale unlabeled corpus pre-training of the large model, a pre-trained model with hundreds of millions of parameters is output. Such a model can adapt to a wide range of downstream tasks, and the model has good generalization ability. For example, large language models (LLM, Large Language Model), multi-modal pre-training models, etc.

[0028] In actual application, the large model only needs a small amount of samples to fine-tune the pre-trained model and can be applied to different tasks. The large model can be widely applied to natural language processing (NLP, Natural Language Processing) and computer vision fields. Specifically, it can be applied to computer vision field tasks such as visual question answering (VQA, Visual Question Answering), image captioning (IC, Image Caption), image generation, and natural language processing field tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0029] First, the nomenclature involved in one or more embodiments of the present specification is explained.

[0030] Supervised fine-tuning (SFT, Supervised Fine-Tuning): It is a further training method based on pre-trained models. In this method, the model is trained on a dataset containing input and expected output pairs so that the model can learn to generate answers closer to human level. Supervised fine-tuning is usually used to adapt the model to data in a specific task or field, thereby improving the model's performance on these tasks.

[0031] Reinforcement Learning (RL): A machine learning paradigm where the core idea is to let an agent learn how to take better actions in a given situation by continuously interacting with the environment to maximize some cumulative reward signal. In the context of LLM training, reinforcement learning can often be used to fine-tune the LLM based on an evaluation model (also known as a reward model) to generate text that better aligns with human preferences.

[0032] Group Relatively Policy Optimization (GRPO): A reinforcement learning algorithm that uses normalized rewards for multiple responses to calculate the advantage value for each response and then updates the model parameters.

[0033] Proximal Policy Optimization (PPO): A reinforcement learning algorithm that optimizes policy through advantage function estimation and KL divergence constraints.

[0034] Kullback-Leibler Divergence (KL Divergence): A measure of the difference between two probability distributions. Specifically, KL divergence measures the amount of information lost when one probability distribution is used to approximate another.

[0035] Reward Hacking: A model may manipulate the evaluation model or scoring mechanism by generating lengthy, repetitive, or irrelevant but seemingly "reasonable" content to inflate its evaluation scores, rather than genuinely improving output quality.

[0036] Clip Function: A function commonly used in programming, mathematics, and deep learning to limit numerical values within a specified range. If a number exceeds the set maximum or minimum value, it will be "clipped" to the boundary value.

[0037] Comparative Policy Optimization (CPO): A method used in reinforcement learning to improve policies by comparing the performance of multiple candidate actions or policies. This method is particularly useful for tasks that require selecting the best option from a set of possible actions, such as multi-turn dialogue management in dialogue systems or item recommendation in recommendation systems.

[0038] The optimization process of LLM usually includes two stages: the first stage is to perform SFT on role dialogue corpus; and the second stage is to perform reinforcement learning by using an evaluation model. Taking an open task such as role playing as an example, due to the unclear evaluation standard, it is difficult to build an evaluation model, and effective reinforcement learning is difficult to perform. In addition, in the traditional evaluation model scheme, a score is usually calculated separately for each model reply. This way often faces the following problems in open tasks, making it difficult to achieve effective reinforcement learning training: ambiguous evaluation standard: for open task responses, it is difficult to adopt explicit and consistent scoring rules. Unstable scoring: the evaluator based on single-sample scoring is sensitive to prompt changes, and often generates unstable and discriminative scores. In certain cases, the score may also collapse, with most outputs falling into a narrow scoring range. Error amplification: for reinforcement learning algorithms that use group normalization (such as the GRPO algorithm), the rewards are normalized during optimization. When the model generates content containing factual errors or logical contradictions, the single-sample score may give a high score due to local fluency, thereby amplifying errors and misleading the policy learning process.

[0039] To solve the problems of reward ambiguity and unstable scoring in open tasks such as role playing, an embodiment of the present specification proposes a reinforcement learning training scheme based on CPO. Specifically, a task processing model is used to obtain a plurality of sample reply contents based on sample dialogue data; the plurality of sample reply contents are compared and analyzed to obtain a plurality of reply indicators corresponding to the plurality of sample reply contents respectively, wherein the reply indicators are used to measure the quality of the corresponding sample reply contents; and the task processing model is trained according to the reply indicators to obtain a trained task processing model. This scheme changes the traditional single-sample evaluation to comparative sample evaluation. Through comparative evaluation, the ambiguity of the evaluation standard can be reduced, the response quality can be more accurately evaluated, and the accuracy of the reply indicators can be improved. At the same time, the fluctuation of the reply indicators can be reduced, making the reply indicators more discriminative and stable, and reducing the error amplification problem caused by single-sample evaluation.

[0040] In the present specification, a task processing model training method is provided. The present specification also relates to a role playing model training method, a task processing method, a request processing method based on a task processing model, a task platform, a task processing model training device, a role playing model training device, a task processing device, a request processing device based on a task processing model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product. Each is described in detail in the following embodiments.

[0041] Reference is made to Figure 1 , Figure 1A flowchart of a task processing model training method provided by one embodiment of the present specification is shown, which specifically includes the following steps:

[0042] Step 102: Obtain multiple sample reply contents based on the sample dialogue data by using the task processing model.

[0043] It should be noted that the task processing model refers to a machine learning model used to complete a specific task (such as role-playing, dialogue reply generation, question and answer, etc.). The task processing model can be a general pre-trained large model, or a domain large model fine-tuned on a specific task. The task processing model has the ability to receive input (such as user dialogue history or questions) and generate corresponding output (such as reply content). During training, the model parameters of the task processing model can be adjusted through reinforcement learning and other methods to improve task performance.

[0044] Sample dialogue data refers to a set of real or constructed dialogue data used to train the task processing model. The sample dialogue data can be question and answer data with dialogue goals, or chat data without dialogue goals. The sample dialogue data includes user input (such as questions, instructions), and can also include dialogue role settings and dialogue context. The sample dialogue data can be data of different modalities, such as text, speech, video, image, etc. These data are provided as input to the task processing model to generate corresponding sample reply contents. The number of sample dialogue data can be multiple, and in each training iteration of the task processing model, a batch of data can be extracted for updating the model parameters. After multiple iterations, reinforcement learning training is completed.

[0045] Multiple sample reply contents refer to multiple different replies generated by the task processing model for a piece of sample dialogue data through multiple samplings (such as using different generation strategies or random sampling). The multiple sample reply contents may differ in expression, information integrity or quality, and can provide diverse comparison objects for subsequent comparative analysis.

[0046] In actual application, there are multiple ways to obtain multiple sample reply contents based on sample dialogue data by using the task processing model, which are specifically selected according to actual conditions, and the present specification does not make any limitation on this. In one possible implementation manner of the present specification, the sample dialogue data sent by the user through the client can be received, the sample dialogue data is input into the task processing model, and multiple sample reply contents are obtained. In another possible implementation manner of the present specification, multiple sample reply contents obtained by the task processing model from the sample dialogue data can be read from other data acquisition devices or databases.

[0047] Step 104: Comparative analysis is performed on the plurality of sample reply contents to obtain a plurality of reply indicators corresponding to the plurality of sample reply contents respectively, wherein the reply indicator is used to measure the quality of the corresponding sample reply content.

[0048] It should be noted that comparative analysis refers to the process of comparing and evaluating each other among the plurality of sample reply contents. The comparative analysis process is usually based on the preset quality evaluation dimensions (such as relevance, fluency, factual accuracy, safety, etc.), identifies the advantages and disadvantages between each sample reply content, and thus assigns a corresponding reply indicator to each sample reply content.

[0049] The reply indicator refers to the relative advantage or disadvantage judgment or score used to measure the quality of a sample reply content. The reply indicator is used to reflect the advantages and disadvantages of the sample reply content relative to other sample reply contents, and can be used as a reward signal for subsequent task processing model training. The reply indicator can be a single-dimensional (such as fluency) evaluation result, or a weighted result of evaluating multiple dimensions. The reply indicator is usually generated based on a comparative analysis strategy or a content comparison model.

[0050] In practical applications, there are many ways to compare and analyze the plurality of sample reply contents to obtain a plurality of reply indicators corresponding to the plurality of sample reply contents, which are selected according to actual conditions. In one possible implementation of the present specification, for any one sample reply content, the sample reply content can be compared with all sample reply contents other than the sample reply content, for example, assuming that there are three sample reply contents, which are sample reply content A, sample reply content B, and sample reply content C. For sample reply content A, sample reply content A is compared with sample reply content B and sample reply content C to obtain the reply indicator of sample reply content. In another possible implementation of the present specification, for any one sample reply content, at least one sample reply content can be selected (randomly selected or selected based on similarity and the like) from the sample reply content other than the sample reply content and compared with the sample reply content, for example, assuming that there are three sample reply contents, which are sample reply content A, sample reply content B, and sample reply content C. For sample reply content A, sample reply content C is selected from sample reply content B and sample reply content C, and sample reply content A and sample reply content C are compared to obtain the reply indicator of sample reply content.

[0051] In an optional embodiment of the present specification, the above comparative analysis of the plurality of sample reply contents to obtain a plurality of reply indicators corresponding to the plurality of sample reply contents can include the following steps:

[0052] screening at least one sample comparison content from the second sample reply content, wherein the first sample reply content is any one of the plurality of sample reply contents, and the second sample reply content is the sample reply content other than the first sample reply content in the plurality of sample reply contents;

[0053] comparing the first sample reply content with the sample comparison content to obtain a reply index corresponding to the first sample reply content.

[0054] It should be noted that the first sample reply content refers to one sample reply content selected from the plurality of sample reply contents, which can be used as the main object of the current evaluation. The second sample reply content refers to the remaining reply set other than the first sample reply content in the plurality of sample reply contents, which is used to provide a comparison reference for the first sample reply content.

[0055] The sample comparison content refers to one or more sample reply contents selected from the second sample reply content for quality comparison with the first sample reply content. The sample comparison content can be randomly selected, or it can be a sample reply content with comparability selected based on theme, semantic or structural similarity.

[0056] In actual application, there are various ways to compare the first sample reply content with the sample comparison content to obtain the reply index corresponding to the first sample reply content, which is selected according to actual conditions, and the embodiments of the present specification do not make any limitation on this. In one possible implementation of the present specification, the reply index can be obtained by directly comparing the first sample reply content with the sample comparison content. In another possible implementation of the present specification, in order to improve the quality of the reply index, sample dialogue data can be introduced, and the association between the sample dialogue data and the first sample reply content and the sample comparison content is considered in the comparison process to obtain the reply index.

[0057] The scheme of the embodiments of the present specification dynamically selects sample comparison content for each sample reply content and performs relative quality judgment, which realizes more fine and stable reply index modeling. The reply index based on pair or group comparison can effectively reduce subjective bias, enhance distinguishability, and especially identify the reply with " seemingly reasonable but with slight defects", so as to guide the task processing model to more accurately optimize the generation strategy in reinforcement learning and improve the stability and reliability of the output quality.

[0058] In an optional embodiment of the present specification, the comparison of the first sample reply content with the sample comparison content to obtain the reply index corresponding to the first sample reply content can include the following steps:

[0059] Based on the sample dialogue data, the first sample reply content is compared with the sample comparison content to obtain a reply index corresponding to the first sample reply content.

[0060] In actual applications, there are various ways to obtain the reply index corresponding to the first sample reply content based on the sample dialogue data, by comparing the first sample reply content with the sample comparison content, which are selected according to actual conditions, and the embodiments of the present specification do not make any limitation on this. In a possible implementation manner of the present specification, the content comparison strategy and the sample dialogue data can be used to compare the first sample reply content with the sample comparison content to obtain the reply index corresponding to the first sample reply content. In another possible implementation manner of the present specification, the sample dialogue data, the first sample reply content and the sample comparison content can be input into a content comparison model, and the content comparison model can be used to compare the first sample reply content with the sample comparison content based on the sample dialogue data to obtain the reply index corresponding to the first sample reply content.

[0061] By applying the scheme of the embodiments of the present specification, the context intention can be understood by combining the sample dialogue data, so that the comparison and analysis between the first sample reply content and the sample comparison content can be compared horizontally under the same input context, the generation of the reply index is more objective and accurate, and the training efficiency of the task processing model and the reliability of the generated result are effectively improved.

[0062] In an optional embodiment of the present specification, the comparison between the first sample reply content and the sample comparison content to obtain the reply index corresponding to the first sample reply content can include the following steps:

[0063] The comparison prompt information, the first sample reply content and the sample comparison content are input into the content comparison model to obtain the reply index corresponding to the first sample reply content, wherein the comparison prompt information is used to guide the content comparison model to compare the first sample reply content with the sample comparison content to obtain the reply index.

[0064] It should be noted that the comparison prompt information refers to a pre-designed instructive text (Prompt) used to explicitly guide the content comparison model to perform a specific comparison task, for example, “Please compare the advantages and disadvantages of the following two replies in terms of factual accuracy and expression clarity”. The comparison prompt information can standardize the comparison and analysis standard to ensure the consistency and interpretability of the comparison process.

[0065] The content comparison model refers to a model used to evaluate and compare different content quality. The content comparison model can receive the comparison prompt information, the first sample reply content and the sample comparison content as input, and output the reply index of the first sample reply content. The content comparison model can be a pre-trained large model, or a discriminant model trained based on the training sample reply content, the training comparison content and the training reply index.

[0066] By applying the scheme of the embodiments of the present specification, the content comparison model simultaneously compares and analyzes multiple sample reply contents, and gives a reply indicator, thereby improving the stability and consistency of the reply indicator. By introducing structured "comparison prompt information" to guide the content comparison model to perform standardized and controllable quality evaluation, the evaluation dimension can be effectively focused, subjective bias can be reduced, reliable and interpretable reply indicators can be generated in complex contexts, and the controllability of the task processing model training process can be improved.

[0067] In an optional embodiment of the present specification, the above comparison of the first sample reply content and the sample comparison content to obtain the reply indicator corresponding to the first sample reply content can include the following steps:

[0068] Based on the content comparison strategy, the first sample reply content and the sample comparison content are compared to obtain the reply indicator corresponding to the first sample reply content, wherein the content comparison strategy is constructed based on at least one quality evaluation dimension.

[0069] It should be noted that the quality evaluation dimension refers to a specific standard or aspect for measuring the quality of the reply, such as relevance, fluency, factual accuracy, safety, semantic accuracy, logical coherence, language naturalness, whether it contains key information points, etc.

[0070] The content comparison strategy refers to a method or rule system for comparing the first sample reply content and the sample comparison content, which is usually constructed based on one or more quality evaluation dimensions to determine how to quantify or determine the difference between the two. The content comparison strategy can be a natural language form of comparison strategy based on quality evaluation dimensions, or an automated comparison code based on quality evaluation dimensions, which is selected according to actual conditions, and the embodiments of the present specification do not make any limitation.

[0071] By applying the scheme of the embodiments of the present specification, the first sample reply content and the sample comparison content are compared item by item using the content comparison strategy, thereby realizing multi-dimensional and systematic quality evaluation of the first sample reply content.

[0072] Step 106: training the task processing model according to the reply indicator to obtain a trained task processing model.

[0073] It should be noted that training refers to the process of updating the internal parameters of the task processing model through optimization algorithms (such as GRPO, PPO, etc.) according to the reward signal feedback by the reply indicator. The purpose of training is to make the task processing model generate higher quality replies, i.e., obtain higher quality content.

[0074] The trained task processing model refers to a task processing model trained based on the reply indicators obtained through comparative analysis. The trained task processing model has stronger generation capability and can generate reply content that meets the expected quality standard for the input dialogue data. The trained task processing model can be used for actual deployment and application.

[0075] By simultaneously generating and comparing multiple sample reply contents, the scheme of the embodiments of the present specification can accurately identify the differences between the replies in the relative comparison, thereby generating reply indicators with higher discrimination and stability, avoiding the judgment deviation caused by ambiguous standards when scoring a single sample, providing a more objective and accurate basis for the training process, and significantly improving the efficiency and accuracy of the task processing model training process.

[0076] In an optional embodiment of the present specification, the training process of the task processing model is described by taking the GRPO algorithm as an example. That is, the training of the task processing model according to the reply indicators to obtain the trained task processing model can include the following steps:

[0077] According to the reply indicators, a strategy loss is calculated, wherein the strategy loss is used to measure the difference between the model performance of the task processing model and the expected model performance.

[0078] According to the multiple sample reply contents, a model difference indicator is calculated, wherein the model difference indicator is used to measure the difference between the task processing model and the historical task processing model, and the historical task processing model is a model version of the task processing model at a previous time point.

[0079] According to the strategy loss and the model difference indicator, the task processing model is trained to obtain the trained task processing model.

[0080] It should be noted that the strategy loss (Policy Loss) refers to the loss function value calculated based on the reply indicators in the reinforcement learning optimization process. The strategy loss can be used to measure the gap between the model performance (current behavior / policy) of the current task processing model and the expected model performance (expected behavior / policy), and encourage the model to take better actions. The larger the strategy loss, the more the quality of the current output of the task processing model deviates from the expected target. The strategy loss can be calculated based on the pruning mechanism.

[0081] The model difference indicator can be used to measure the difference between the output behavior / policy or parameter distribution of the task processing model and the historical task processing model. The common form of the model difference indicator is KL divergence, so the model difference indicator can also be called KL divergence penalty. Through the model difference indicator, the "step" or degree of change of the model strategy adjustment can be reflected, preventing the parameter update from being too large and causing unstable training or catastrophic forgetting.

[0082] In practical applications, the task processing model can be trained using the following formulas (1) to (5) based on the policy loss and model difference index to obtain the trained task processing model:

[0083] (1)

[0084] in, This represents sample dialogue data; The model parameters are: Task processing model; This indicates that the sample dialogue data is input into the task processing model, and the obtained... Sample response content; This represents the response content of the i-th sample.

[0085] (2)

[0086] in, This represents a content comparison model; Indicates will The model compares the input content of each sample response with the obtained G response metrics. This represents the response content of the i-th sample. The corresponding i-th response indicator.

[0087] (3)

[0088] in, This represents the advantage value of the t-th token in the response of the i-th sample; This represents the average of the G response indicators; This represents the standard deviation of the G response indicators.

[0089] (4)

[0090] in, Indicates the strategy loss; This means taking the minimum value of the two expressions as... This is to prevent the task processing model from overestimating or underestimating the probability of certain actions, which could lead to unstable training. Indicates a given and the output token sequence In this case, task processing model Output The probability of; Indicates a given and In this case, historical task processing model Output The probability of; and represents the probability change of the task processing model selecting a specific output relative to the historical task processing model; represents a clipping operation that limits the strategy ratio change range of the task processing model and the historical task processing model; and represents a preset threshold for controlling the maximum and minimum ratio changes allowed. The clipped result ensures that the strategies of the task processing model and the historical task processing model do not differ too much, thereby maintaining the stability of the reinforcement learning process.

[0091] (5)

[0092] wherein, represents a function for training the task processing model, aiming to maximize the expected value; represents the expected value of and , which means that the input and all output cases need to be considered to ensure that the model performs well in various cases; represents the average processing of all sample reply contents, ensuring that each sample reply content contributes equally to the final result; represents the total strategy loss; represents the model difference indicator, also known as the KL divergence penalty, which aims to prevent excessive deviation between the task processing model and the historical task processing model, thereby maintaining the stability of the training process; is a hyperparameter for controlling the weight of the KL divergence term; represents the KL divergence of the task processing model relative to the historical task processing model; represents the content length of .

[0093] By applying the scheme of the embodiments of the present specification, efficient and stable model optimization is achieved by training in combination with the strategy loss and the model difference indicator. The strategy loss can ensure that the task processing model learns towards higher reply quality, while the model difference indicator constrains the update amplitude of the task processing model, avoiding drastic fluctuations or performance decline of the strategy due to reward signal noise. Through the synergistic effect of the strategy loss and the model difference indicator, the robustness and convergence of the model training process are improved, ultimately obtaining a trained task processing model with high generation quality, stable behavior, and strong generalization ability.

[0094] In an optional embodiment of the present specification, in order to avoid the reward manipulation problem caused by the too long response (sample reply content) of the task processing model, the present specification introduces a response length soft penalty mechanism to constrain the reply indicator based on the content length of the sample reply content, that is, before the above training of the task processing model based on the reply indicator to obtain the trained task processing model, the following steps can be further included:

[0095] constraining the reply indicator based on the content length of the sample reply content to obtain a target reply indicator;

[0096] training the task processing model based on the reply indicator to obtain the trained task processing model can include the following steps:

[0097] training the task processing model based on the target reply indicator to obtain the trained task processing model.

[0098] It should be noted that the content length refers to the text length of each sample reply content generated by the task processing model. The content length is usually calculated in terms of the number of characters or words. The content length can reflect the detail and complexity of the sample reply content.

[0099] Constraining the reply indicator means imposing a limit condition on the reply indicator based on the content length of the sample reply content, ensuring that the reply indicator not only considers the quality of the sample reply content but also takes into account the content length factor to avoid reward manipulation problems.

[0100] The target reply indicator refers to the reply indicator adjusted by the length constraint, aiming to balance the quality and length of the sample reply content to provide a more comprehensive and reasonable optimization direction for the task processing model.

[0101] In actual application, there are various ways to constrain the reply indicator based on the content length of the sample reply content to obtain the target reply indicator, which is selected according to actual conditions, and the present specification does not make any limitation on this. In a possible implementation manner of the present specification, when the length of the sample reply content exceeds the preset length, the reply indicator can be directly reduced by a certain percentage or set to a lower value to obtain the target reply indicator, thereby encouraging the task processing model to generate concise and clear answers. In another possible implementation manner of the present specification, a penalty term related to the reply length can be introduced, which can be a linear, exponential or other form of function that increases with the increase of the reply length. Then, the penalty term is subtracted from the reply indicator to obtain the target reply indicator adjusted by the length, thereby flexibly balancing the relationship between the reply quality and the length.

[0102] Further, the implementation manner of "training the task processing model according to the target reply indicator to obtain the trained task processing model" can refer to the above-mentioned manner of "training the task processing model according to the reply indicator to obtain the trained task processing model". For example, the formula (3) is modified as the target reply indicator, and the description of the embodiments of the present specification will not be repeated. The modification of the formula (3) as the target reply indicator will not be repeated in the description of the embodiments of the present specification.

[0103] By introducing the constraint mechanism for the content length to adjust the reply indicator, the application of the scheme of the embodiments of the present specification avoids the reward manipulation problem, so that the training process can not only improve the overall quality of the reply, but also effectively control the length of the reply, and finally obtain a trained task processing model that can accurately respond to user needs and maintain good user experience.

[0104] In an optional embodiment of the present specification, the above-mentioned step of "constraining the reply indicator according to the content length of the sample reply content to obtain the target reply indicator" can include the following steps:

[0105] According to the content length of the sample reply content, determining a plurality of content indicators respectively corresponding to the plurality of sample reply contents;

[0106] Fusing the content indicators and the reply indicator to obtain an intermediate reply indicator;

[0107] Standardizing the intermediate reply indicator to obtain the target reply indicator.

[0108] It should be noted that the content indicator is a length penalty value calculated according to the length of the sample reply content. For example, a sample reply content that is too short or too long may obtain a higher content indicator, while a sample reply content in an ideal length interval obtains a lower score. The intermediate reply indicator is an intermediate score result obtained by fusing (such as weighted summation, multiplication) the content indicator and the reply indicator.

[0109] The standardization processing refers to mapping the intermediate reply indicators to a unified numerical range (such as [0, 1]) through a certain mathematical transformation (such as minimum-maximum standardization, zero-mean standardization). Through the standardization processing, the influence of the scale of the reply indicators can be eliminated. For example, in some states, the sample reply content generated by the task processing model is generally of high quality, and the reply indicators are concentrated in [8, 9]. In other states, the sample reply content generated is of poor quality, and the reply indicators are concentrated in [2, 3]. If the reply indicators are directly used as advantages without standardization, the task processing model will be more inclined to update those states with "high reply indicators", and the actions with better relative performance will be ignored. After standardization, the target reply indicators are standardized to have a mean of 0 and a standard deviation of 1, so that different states are comparable. In reinforcement learning, the direction and size of parameter update are affected by the advantage value. If the advantage value fluctuates too much (such as +100 and -50), the parameter update will be explosive or oscillatory. Through standardization, the parameter update can be smoother, which helps the task processing model to converge quickly.

[0110] In practical applications, the target reply indicators can be calculated by the following formula (6):

[0111] (6)

[0112] wherein, represents the target reply indicator, represents the content indicator, represents the intermediate reply indicator.

[0113] By introducing the content indicator based on the content length and fusing it with the reply indicator to generate the intermediate reply indicator, the application of the scheme of the embodiments of the present specification realizes the joint modeling of the reply quality and length, and then the target reply indicator of the unified scale is obtained through the standardization processing, which effectively avoids the scoring deviation caused by the length difference, so as to guide the task processing model to generate the output meeting the expected length while ensuring the high quality of the reply, and improve the user experience and system controllability.

[0114] In an optional embodiment of the present specification, the above determining the content indicators corresponding to the plurality of sample reply contents respectively according to the content length of the sample reply contents can include the following steps:

[0115] obtaining a first length parameter and a second length parameter, wherein the first length parameter is greater than the second length parameter;

[0116] determining the content indicators corresponding to the plurality of sample reply contents respectively according to the first length parameter, the second length parameter and the content length of the sample reply contents.

[0117] It should be noted that the first length parameter refers to the maximum allowed length of the sample response content (i.e., the upper limit constraint). Sample responses exceeding the first length parameter will be considered too long and penalized (e.g., -1 point). The second length parameter refers to the length of the "soft penalty interval," which defines a transition region from "no penalty" to "strong penalty." This is used to avoid the task processing model being suddenly and severely penalized when approaching the length limit, thereby improving training stability. The first and second length parameters are selected based on the actual situation, and this specification does not impose any limitations on them in the embodiments.

[0118] In practical applications, the content index can be calculated using the following formula (7):

[0119] (7)

[0120] Among them, when the content length Greater than the first length parameter At that time, content metrics -1; when the content length is... Less than or equal to the first length parameter Second length parameter Length difference At that time, content metrics =0; when the response length is 0 Greater than or equal to But less than or equal to At that time, according to , as well as Calculate content indicators .

[0121] By applying the solutions in the embodiments of this specification, and by introducing a first length parameter and a second length parameter, combined with the content length of the sample response, the adaptability of the sample response content in terms of length can be flexibly evaluated, thereby effectively controlling the length of the model response and avoiding the generation of overly lengthy answers. Furthermore, by setting a reasonable transition area, the task processing model is given a certain degree of flexibility, preventing a significant drop in response metrics due to slightly exceeding the length limit.

[0122] See Figure 2 , Figure 2A processing process schematic diagram of a task processing model training method provided by an embodiment of the present specification is shown. The task processing model training process includes: obtaining a query, inputting the query into a task processing model, and obtaining a plurality of replies (reply 1 to reply G); using a content comparison model, comparing and analyzing the plurality of replies to obtain a plurality of reward values (reward value 1 to reward value G); calculating a plurality of advantage values (advantage value 1 to advantage value G) according to the plurality of reward values; calculating a strategy loss according to the plurality of advantage values; using an auxiliary model, calculating a KL divergence penalty according to the plurality of sample reply contents; training the task processing model according to the strategy loss and the KL divergence penalty to obtain a trained task processing model.

[0123] It is worth noting that, Figure 2 The snowflake symbol in the above formula indicates that the model parameters of the content comparison model and the auxiliary model are fixed and unchanged during the training process of the task processing model, and the spark indicates that the model parameters of the task processing model are adjusted. The auxiliary model refers to a benchmark model, and the probability distribution output by the benchmark model can be used as a comparison basis to compare the probability distribution generated by the task processing model to calculate the KL divergence between them.

[0124] Figure 2 A traditional reward value modeling scheme and a reward value modeling scheme based on comparison strategy optimization proposed by an embodiment of the present specification are also shown. In the traditional reward value modeling scheme, a reward model is used to calculate the reward value of each reply separately, such as reply 1 corresponding to reward value 1 being 0.6, reply 2 corresponding to reward value 2 being 0.6, and reply G corresponding to reward value G being 0.7. In the reward value modeling scheme based on comparison strategy optimization proposed by an embodiment of the present specification, the content comparison model is used to compare a plurality of replies to obtain the reward value of each reply, such as reply 1 corresponding to reward value 1 being 0.54, reply 2 corresponding to reward value 2 being 0.62, and reply G corresponding to reward value G being 0.69.

[0125] The following describes the task processing model training method provided by the present specification in combination with the accompanying Figure 3 The task processing model training method provided by the present specification is further described by taking the application of the task processing model training method in a role-playing scene as an example. Wherein, Figure 3 A flowchart of a role-playing model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps:

[0126] Step 302: obtaining a plurality of sample reply contents based on sample dialogue data using a role-playing model.

[0127] Step 304: comparing and analyzing the plurality of sample reply contents to obtain a plurality of reply indicators corresponding to the plurality of sample reply contents respectively, wherein the reply indicator is used to measure the quality of the corresponding sample reply content.

[0128] Step 306: Train the role-playing model based on the response metrics to obtain the trained role-playing model.

[0129] It's important to note that role-playing models allow users to create virtual characters with personalities, memories, and behavioral logic tailored to their needs. These virtual characters can be used in a variety of scenarios, including games, emotional support, virtual assistants, education and training, and psychological counseling. The sample dialogue data input into the role-playing model includes, but is not limited to, user information, character information, and dialogue context. User information refers to background or state information about the user, used to help the role-playing model better understand the context of the dialogue and personalized needs. Character information defines the role the role-playing model should play and its related attributes. It determines the model's behavior, tone, knowledge boundaries, and personality settings. Dialogue context refers to the historical content of the current dialogue, including multi-turn dialogue records between the user and the character.

[0130] In practical applications, the implementation methods of steps 302 to 306 are the same as those of steps 102 to 106, and will not be described again in the embodiments of this specification.

[0131] By applying the solution in the embodiments of this specification, and by simultaneously generating and comparing multiple sample responses, the differences in quality between responses can be accurately identified in relative comparisons. This results in the generation of more discriminative and stable response indicators, avoiding judgment bias caused by standard ambiguity when scoring single samples. This provides a more objective and accurate basis for the training process, significantly improving the efficiency and accuracy of the role-playing model training process.

[0132] See Figure 4 , Figure 4 This specification shows a flowchart of a task processing method according to an embodiment, which specifically includes the following steps:

[0133] Step 402: Obtain the task data for the target task.

[0134] Step 404: Input the task data into the task processing model to obtain the task processing result of the target task. The task processing model is trained based on the task processing model training method.

[0135] It should be noted that the target task refers to the specific task to be completed, such as question answering, reasoning, classification, generation, and dialogue. The target task can be a task in different scenarios, such as a multi-turn dialogue task in a role-playing scenario or a consultation task in a customer service scenario. Task data refers to the input data for the target task, used to feed into the task processing model to generate the task processing result. Task data includes at least one of the following: text, images, video, audio, and tables.

[0136] The task processing model refers to a deep learning model capable of generating a task processing result based on input task data. The task processing model includes but is not limited to a language model, a classification model, an inference model, etc. Since the task processing model is a model trained by the task processing model training method, the task processing model has high stability and precision, and can generate a task processing result with high accuracy. The task processing result refers to an output result generated by the task processing model according to input task data, and is a solution or prediction of the target task by the task processing model.

[0137] By applying the scheme of the embodiments of the present specification, the task data of the target task is obtained and input into the task processing model trained by the task processing model training method, so that a high-quality task processing result can be obtained.

[0138] Considering that the model parameter amount of the task processing model is relatively large and the operation resources of the client are limited, the task processing method provided by the embodiments of the present specification can be applied to a task processing system as shown in Figure 5 , but is not limited thereto. Referring to Figure 5 , Figure 5 An architecture diagram of a task processing system provided by one embodiment of the present specification is shown, and the task processing system can include a client 502 and a server 504.

[0139] The client 502 is configured to send task data of a target task to the server 504.

[0140] The server 504 is configured to input the task data into a task processing model to obtain a task processing result of the target task, wherein the task processing model is trained based on a task processing model training method; and send the task processing result to the client 502.

[0141] The client 502 is further configured to receive the task processing result sent by the server 504.

[0142] As shown in Figure 5 , the task processing model is deployed in the server 504, and the server 504 can be connected to one or more clients 502 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client 502 can include but is not limited to a smartphone, a tablet computer, a notebook computer, a palm computer, a personal computer (PC, Personal Computer), a smart home device, a vehicle-mounted device, etc. The client 502 can also interact with the user through a user graphical interface to realize the calling of the task processing model, and thus realize the task processing method provided by the embodiments of the present specification.

[0143] It should be noted that the task processing method provided in the embodiments of the present specification is generally executed by the server, but in other embodiments of the present specification, the client can also have similar functions as the server to execute the task processing method provided in the embodiments of the present specification, in the case that the running resources of the client can meet the deployment and running conditions of the task processing model. In other embodiments, the task processing method provided in the embodiments of the present specification can also be executed by the client and the server together.

[0144] Referring to Figure 6 , Figure 6 A flowchart of a request processing method based on a task processing model is shown, which is provided in an embodiment of the present specification, and the method is applied to a task platform, and specifically includes the following steps:

[0145] Step 602: receiving a model request sent by a terminal device.

[0146] Step 604: determining a target task processing model from a plurality of task processing models based on the model request, wherein the plurality of task processing models are obtained by training based on a task processing model training method.

[0147] It should be noted that the plurality of task processing models can be task processing models with different model specification parameters and adapted to different scenarios. Since the task processing model is a model obtained by training based on the task processing model training method, it has high stability and precision. The target task processing model is a task processing model suitable for a target scenario. The target scenario can be different scenarios, such as a role-playing scenario, a multi-round emotional dialogue scenario, etc. The model request includes at least one of a scenario identifier of the target scenario, scenario input data of the target scenario, and a model specification parameter.

[0148] In actual application, there are various ways to determine the target task processing model from the plurality of task processing models based on the model request, which are specifically selected according to actual conditions, and the embodiments of the present specification do not make any limitation on this. In the first possible implementation manner of the present specification, the corresponding target task processing model can be searched from the task processing models included in the first model library based on the scenario identifier included in the model request; in the second possible implementation manner of the present specification, the target task processing model can be obtained by training based on the scenario input data included in the model request; and in the third possible implementation manner of the present specification, the corresponding target task processing model can be searched from the task processing models included in the second model library based on the model specification parameter included in the model request.

[0149] Exemplarily, at least one task processing model pre-trained can be searched from the first model library based on the scene identification of the target scene, then an intermediate task processing model can be obtained by screening from the at least one task processing model based on the model specification parameter, and then the intermediate task processing model obtained by screening can be trained based on the scene input data of the target scene to obtain a target task processing model suitable for the user demand.

[0150] By applying the scheme of the embodiments of the present specification, the target task processing model is obtained according to the user demand, the personalized model service is realized, the user is provided with an efficient, flexible and easy-to-use model service method, and the user experience is improved.

[0151] In an optional embodiment of the present specification, the above determining the target task processing model from the plurality of task processing models based on the model request can include the following steps:

[0152] In the case where the model request includes the scene identification of the target scene, the target task processing model suitable for the target scene can be searched from the first model library based on the scene identification, wherein the first model library stores a plurality of task processing models suitable for different scenes;

[0153] In the case where the model request includes the scene input data of the target scene, a to-be-trained task processing model suitable for the target scene can be determined from the plurality of task processing models, and the to-be-trained task processing model can be trained based on the scene input data to obtain the target task processing model;

[0154] In the case where the model request includes the model specification parameter, the target task processing model corresponding to the model specification parameter can be searched from the second model library, wherein the second model library stores a plurality of task processing models with different model specification parameters.

[0155] It should be noted that the scene identification refers to a unique or specific label used to distinguish different scenes. The first model library is a database for storing and managing various pre-trained deep learning models. The plurality of task processing models suitable for different scenes cover different application scenarios and demands. The first model library allows the user to select a suitable task processing model according to his own demand, or directly call a suitable task processing model to execute a target task through an application programming interface. The plurality of task processing models suitable for different scenes are a plurality of models suitable for different scenes stored in the first model library, and each task processing model is optimized for a specific application environment. For example, a target task processing model suitable for a role-playing scene can be searched from the first model library based on the scene identification "role-playing" of the target scene.

[0156] Scenarios adapted by each task processing model are different, for example, task processing model one is adapted to scenario one and scenario two, and task processing model two is adapted to scenario two and scenario three. The task processing model to be trained refers to a model in the plurality of task processing models that is adapted to the target scenario but the model effect can be further optimized. If the target scenario is scenario one, the task processing model to be trained is task processing model one adapted to scenario one. The task processing model to be trained can be adapted not only to the target scenario but also to other scenarios, and is a general task processing model that can be adapted to different scenarios. The target task can be executed by using the task processing model to be trained, but the effect can not be very good. At this time, the task processing model to be trained can be optimized based on the scene input data of the target scenario, for example, the task processing model to be trained is optimized based on the scene input data of the role-playing scenario, and a target task processing model adapted to the role-playing scenario can be obtained. The scene input data of the target scenario can be understood as a sample set (sample dialogue data and a plurality of sample reply contents corresponding thereto) in the target scenario used to train the task processing model.

[0157] Model specification parameters refer to various parameters that define the structure and behavior of a model. These parameters can be roughly divided into two categories: model parameters (learnable parameters) and hyperparameters. Model parameters refer to parameters that are automatically adjusted during the model training process through the backpropagation algorithm, including but not limited to weight matrices (weights) and bias terms (biases). For example, in a simple fully connected layer, the weight matrix is a two-dimensional tensor that connects the input layer and the output layer of the neurons; the bias term is a one-dimensional vector that provides an additional offset value for each output neuron. Hyperparameters refer to parameters set before starting model training, which are used to control the learning process and architecture of the model. Hyperparameters include but are not limited to learning rate (Learning Rate), number of neurons per layer (Number of Neurons per Layer), which are selected according to actual conditions.

[0158] By applying the scheme of the embodiments of the present specification, the target task processing model adapted to the corresponding scenario is accurately found based on the scene demand through the scene identification, so that the processing process of the target task is more accurate and more suitable for the target scenario; the general task processing model to be trained is further trained based on the scene input data based on the scene demand, and the target task processing model adapted to the target scenario is obtained, so that the target task processing model is more suitable for the target scenario, thereby improving the user experience and the processing quality of the target task; based on the model specification parameters, the corresponding target task processing model can be accurately found, ensuring efficient and stable operation of the target task processing model, and improving the user experience.

[0159] In one optional embodiment of this specification, after determining the target task processing model from multiple task processing models based on the model request, the following steps may be further included:

[0160] Deploy the target task processing model, and build a task processing interface based on the target task processing model so that the terminal device can schedule the target task processing model to execute the target task through the task processing interface.

[0161] It should be noted that the task processing interface is an interactive programming interface for terminal devices to schedule target task processing models to process target tasks, and it is usually provided in the form of an application programming interface (API). Through the task processing interface, users can input task data for the target task to be processed.

[0162] In practical applications, there are various ways to deploy the target task processing model, and the specific method should be chosen based on the actual situation. This specification does not impose any limitations on this approach. One possible implementation of this specification is to deploy the target task processing model on cloud-side devices using infrastructure provided by a cloud service provider. Another possible implementation of this specification is to deploy the target task processing model on edge devices using a lightweight framework. For example, the target task processing model can be deployed on a distributed system, and a task processing interface can be built based on the target task processing model and provided to terminal devices, enabling the terminal devices to schedule the target task processing model to execute the target task.

[0163] By applying the solutions in the embodiments of this specification, deploying the target task processing model, and constructing the task processing interface, the terminal device can efficiently call the target task processing model, thereby improving the processing quality and response speed of the target task.

[0164] See Figure 7 , Figure 7 A schematic diagram of a task platform according to an embodiment of this specification is shown. The task platform 700 includes a request interface 702 and a response unit 704;

[0165] Request interface 702 is used to receive a model request sent by a terminal device, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0166] The response unit 704 is used to determine the target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on the task processing model training method.

[0167] In one optional embodiment of this specification, the task platform further includes a task processing interface, which is constructed based on the target task processing model.

[0168] a task processing interface for the terminal device to schedule execution of the target task.

[0169] The above is a schematic scheme of the task platform of the embodiment. It should be noted that the technical scheme of the task platform belongs to the same concept as the technical scheme of the request processing method based on the task processing model described above, and the details of the technical scheme of the task platform that are not described in detail can be referred to the description of the technical scheme of the request processing method based on the task processing model described above.

[0170] Corresponding to the task processing model training method embodiment described above, the present specification also provides a task processing model training device embodiment, Figure 8 A structural schematic diagram of a task processing model training device provided by one embodiment of the present specification is shown. As Figure 8 shown, the device includes:

[0171] The first acquisition module 802 is configured to acquire a plurality of sample reply contents based on the sample dialogue data by using the task processing model;

[0172] The first analysis module 804 is configured to perform comparative analysis on the plurality of sample reply contents to obtain a reply index corresponding to each of the plurality of sample reply contents, wherein the reply index is used to measure the quality of the corresponding sample reply content.

[0173] The first training module 806 is configured to train the task processing model according to the reply index to obtain a trained task processing model.

[0174] Optionally, the first analysis module 804 is further configured to, for the first sample reply content, screen at least one sample comparison content from the second sample reply content, wherein the first sample reply content is any one of the plurality of sample reply contents, and the second sample reply content is a sample reply content other than the first sample reply content among the plurality of sample reply contents; compare the first sample reply content with the sample comparison content to obtain the reply index corresponding to the first sample reply content.

[0175] Optionally, the first analysis module 804 is further configured to compare the first sample reply content with the sample comparison content based on the sample dialogue data to obtain the reply index corresponding to the first sample reply content.

[0176] Optionally, the first analysis module 804 is further configured to input the comparison prompt information, the first sample reply content and the sample comparison content into a content comparison model to obtain the reply index corresponding to the first sample reply content, wherein the comparison prompt information is used to guide the content comparison model to compare the first sample reply content with the sample comparison content to obtain the reply index.

[0177] Optionally, the first analysis module 804 is further configured to compare the first sample reply content with sample comparison content based on a content comparison strategy to obtain a reply indicator corresponding to the first sample reply content, where the content comparison strategy is constructed based on at least one quality evaluation dimension.

[0178] Optionally, the first training module 806 is further configured to calculate a strategy loss according to the reply indicator, where the strategy loss is used to measure a difference between a model performance of the task processing model and an expected model performance; calculate a model difference indicator according to the plurality of sample reply contents, where the model difference indicator is used to measure a difference between the task processing model and a historical task processing model, the historical task processing model being a model version of the task processing model at a previous time point; and train the task processing model according to the strategy loss and the model difference indicator to obtain the trained task processing model.

[0179] Optionally, the apparatus further includes a constraint module configured to constrain the reply indicator according to a content length of the sample reply content to obtain a target reply indicator; and the first training module 806 is further configured to train the task processing model according to the target reply indicator to obtain the trained task processing model.

[0180] Optionally, the constraint module is further configured to determine a content indicator corresponding to each of the plurality of sample reply contents according to the content length of the sample reply content; fuse the content indicator and the reply indicator to obtain an intermediate reply indicator; and perform standardization processing on the intermediate reply indicator to obtain the target reply indicator.

[0181] Optionally, the constraint module is further configured to obtain a first length parameter and a second length parameter, where the first length parameter is greater than the second length parameter; and determine the content indicator corresponding to each of the plurality of sample reply contents according to the first length parameter, the second length parameter, and the content length of the sample reply content.

[0182] By simultaneously generating and comparing a plurality of sample reply contents, the scheme of the embodiments of the present specification can accurately identify the differences between the replies in a relative comparison, thereby generating a reply indicator with higher discrimination and stability, avoiding the judgment deviation caused by ambiguous standards when scoring a single sample, providing a more objective and accurate basis for the training process, and significantly improving the efficiency and accuracy of the task processing model training process.

[0183] The above is a schematic scheme of the task processing model training device of the embodiment. It should be noted that the technical scheme of the task processing model training device belongs to the same concept as the technical scheme of the task processing model training method described above. Details of the technical scheme of the task processing model training device that are not described in detail can be referred to the description of the technical scheme of the task processing model training method described above.

[0184] Corresponding to the role-playing model training method embodiment described above, the present specification also provides a role-playing model training device embodiment, Figure 9 A structural schematic diagram of a role-playing model training device provided by an embodiment of the present specification is shown. As shown in Figure 9 The device includes:

[0185] The second acquisition module 902 is configured to acquire a plurality of sample reply contents based on the sample dialogue data by using the role-playing model;

[0186] The second analysis module 904 is configured to perform comparative analysis on the plurality of sample reply contents to obtain reply indicators corresponding to the plurality of sample reply contents respectively, wherein the reply indicators are used to measure the quality of the corresponding sample reply contents;

[0187] The second training module 906 is configured to train the role-playing model according to the reply indicators to obtain a trained role-playing model.

[0188] By simultaneously generating and comparing a plurality of sample reply contents, the embodiments of the present specification can accurately identify the advantages and disadvantages of the replies in the relative comparison, thereby generating a reply indicator with higher discrimination and stability, avoiding the judgment deviation caused by the ambiguous standard when scoring a single sample, providing a more objective and accurate basis for the training process, and significantly improving the efficiency and accuracy of the role-playing model training process.

[0189] The above is a schematic scheme of the role-playing model training device of the embodiment. It should be noted that the technical scheme of the role-playing model training device belongs to the same concept as the technical scheme of the role-playing model training method described above. Details of the technical scheme of the role-playing model training device that are not described in detail can be referred to the description of the technical scheme of the role-playing model training method described above.

[0190] Corresponding to the task processing method embodiment described above, the present specification also provides a task processing device embodiment, Figure 10 A structural schematic diagram of a task processing device provided by an embodiment of the present specification is shown. As shown in Figure 10 The device includes:

[0191] The third obtaining module 1002 is configured to obtain task data of a target task.

[0192] The input module 1004 is configured to input the task data into a task processing model to obtain a task processing result of the target task, where the task processing model is obtained based on a task processing model training method.

[0193] According to the scheme of the embodiments of the present specification, the task data of the target task is obtained and input into the task processing model trained by the task processing model training method, so that a high-quality task processing result can be obtained.

[0194] The above is a schematic scheme of the task processing device of the present embodiment. It should be noted that the technical scheme of the task processing device belongs to the same concept as the technical scheme of the task processing method described above, and the details of the technical scheme of the task processing device that are not described in detail can be referred to the description of the technical scheme of the task processing method.

[0195] Corresponding to the above-mentioned task processing model-based request processing method embodiment, the present specification also provides a task processing model-based request processing device embodiment, Figure 11 A structural schematic diagram of a task processing model-based request processing device according to an embodiment of the present specification is shown. As shown in the figure, Figure 11 The device is applied to a task platform and includes:

[0196] The receiving module 1102 is configured to receive a model request sent by a terminal device.

[0197] The determining module 1104 is configured to determine a target task processing model from a plurality of task processing models based on the model request, where the plurality of task processing models are obtained based on a task processing model training method.

[0198] Optionally, the determining module 1104 is further configured to, in a case where the model request includes a scene identifier of a target scene, search for a target task processing model suitable for the target scene from a first model library based on the scene identifier, where the first model library stores a plurality of task processing models suitable for different scenes; in a case where the model request includes scene input data of the target scene, determine a to-be-trained task processing model suitable for the target scene from the plurality of task processing models, and train the to-be-trained task processing model based on the scene input data to obtain the target task processing model; in a case where the model request includes a model specification parameter, search for a target task processing model corresponding to the model specification parameter from a second model library, where the second model library stores a plurality of task processing models with different model specification parameters.

[0199] Optionally, the apparatus further comprises a deployment module configured to deploy the target task processing model, and construct a task processing interface based on the target task processing model, so that the terminal device schedules the target task processing model to execute the target task through the task processing interface.

[0200] The scheme of the embodiments of the present specification is suitable for obtaining a target task processing model according to user demand, realizes personalized model service, provides an efficient, flexible and easy-to-use model service method for users, and improves user experience.

[0201] The above is a schematic scheme of the request processing apparatus based on the task processing model according to the present embodiment. It should be noted that the technical scheme of the request processing apparatus based on the task processing model belongs to the same concept as the technical scheme of the request processing method based on the task processing model described above, and the details of the technical scheme of the request processing apparatus based on the task processing model that are not described in detail can be referred to the description of the technical scheme of the request processing method based on the task processing model.

[0202] Figure 12 A structural block diagram of a computing device 1200 is shown, which is provided by an embodiment of the present specification. The computing device 1200 comprises a memory 1210 and a processor 1220; the memory 1210 is configured to store computer programs / instructions, and the processor 1220 is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor 1220, realize the steps of the task processing model training method or the role playing model training method or the task processing method or the request processing method based on the task processing model described above.

[0203] In one or more embodiments of the present specification, the computing device can be understood as an integrated intelligent terminal, including but not limited to a server, a desktop computer, a personal computer, a model integrated machine, a mobile phone, a tablet computer or other portable intelligent terminals, etc., and the computing device can be pre-installed with the model described in the embodiments of the present application.

[0204] Specifically, the computing device can preinstall multiple types of models, including but not limited to models in the fields of natural language processing, visual processing, speech processing, code processing, multi-modal task processing, etc., to provide diversified model selection. In different product forms, the computing device can support one or more model usage modes, including but not limited to model training, model calling, model fine-tuning, model deployment, model inference and application, etc. In some product forms, the computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative and other types of models), model version control (supporting the control of different model versions), model evaluation (based on a model evaluation tool, evaluating the performance and effect of a model), etc. In other product forms, the computing device can also create an application based on a model, provide an application programming interface (API, Application Programming Interface) calling capability, and can call a model into a created application through an API interface, while providing an application management tool to realize the management and monitoring of the application.

[0205] Further, the computing device can also include data management (supporting the creation and management of model optimization datasets), a training center (providing rich training resources to help users learn and master artificial intelligence technology), basic management and control capabilities (providing enterprise-level basic management and control capabilities to ensure the security and efficient operation of the system), through the above functions, a comprehensive and integrated artificial intelligence development, training, deployment and application device is provided.

[0206] Figure 13 A structural block diagram of an electronic device 1300 according to an embodiment of the present specification is shown.

[0207] The memory 1310 and the processor 1320 are connected through the bus 1330;

[0208] The memory 1310 is used to store computer programs / instructions, and the processor 1320 is used to execute the computer programs / instructions, which realize the steps of the above task processing model training method or role playing model training method or task processing method or request processing method based on the task processing model when executed by the processor 1320.

[0209] Specifically, the components of the electronic device 1300 include but are not limited to the memory 1310 and the processor 1320. The processor 1320 and the memory 1310 can be connected through the bus 1330.

[0210] The electronic device 1300 can also include an access device 1340 that enables the electronic device 1300 to access data held by a database 1350 via one or more networks 1360. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of networks such as the Internet. The access device 1340 can include one or more of any type of network interface (for example, a Network Interface Card (NIC)) such as an IEEE 802.11 Wireless Local Area Networks (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, or the like.

[0211] In one embodiment of the present specification, the above-described components of the electronic device 1300 and other components not shown in the Figure 13 may be connected to each other, for example, by a bus. It should be understood that Figure 13 The electronic device structure diagram shown is for the purpose of example only and is not a limitation on the scope of the present specification. Other components can be added or replaced by those skilled in the art as needed.

[0212] The electronic device 1300 can be any type of stationary or mobile electronic device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, or the like), a mobile phone (for example, a smartphone), a wearable electronic device (for example, a smartwatch, smartglasses, or the like), or other types of mobile devices, or a stationary electronic device such as a desktop computer or PC. The electronic device 1300 can also be a mobile or stationary electronic device.

[0213] The above is a schematic scheme of the electronic device of the embodiment. It should be noted that the technical scheme of the electronic device and the technical schemes of the task processing model training method, the role playing model training method, the task processing method, and the request processing method based on the task processing model belong to the same concept. For details of the technical scheme of the electronic device that are not described in detail, refer to the description of the technical scheme of the task processing model training method or the role playing model training method or the task processing method or the request processing method based on the task processing model.

[0214] An embodiment of the present specification further provides a computer readable storage medium storing computer programs / instructions, which, when executed by a processor, implement the steps of the task processing model training method or the role playing model training method or the task processing method or the request processing method based on the task processing model.

[0215] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical schemes of the task processing model training method, the role playing model training method, the task processing method, and the request processing method based on the task processing model belong to the same concept. For details of the technical scheme of the storage medium that are not described in detail, refer to the description of the technical scheme of the task processing model training method or the role playing model training method or the task processing method or the request processing method based on the task processing model.

[0216] An embodiment of the present specification further provides a computer program product, including computer programs / instructions, which, when executed by a processor, implement the steps of the task processing model training method or the role playing model training method or the task processing method or the request processing method based on the task processing model.

[0217] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical schemes of the task processing model training method, the role playing model training method, the task processing method, and the request processing method based on the task processing model belong to the same concept. For details of the technical scheme of the computer program product that are not described in detail, refer to the description of the technical scheme of the task processing model training method or the role playing model training method or the task processing method or the request processing method based on the task processing model.

[0218] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims which follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable attributes.

[0219] The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, and the like. It should be noted that the computer readable medium contains contents that can be appropriately added or reduced according to the requirements of patent practice. For example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0220] It should be noted that for the foregoing method embodiments, the acts described can be performed in a different order than that described, and some acts can be performed in parallel. It should also be noted that the embodiments described in the specification are preferred embodiments, and the acts and modules involved are not necessarily essential to the embodiments of the specification.

[0221] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0222] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and limit the application to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and describes these embodiments in order to better explain the principles and practical application of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification. The present specification is limited by the claims and their full scope and equivalents.

Claims

1. A method for training a task processing model, comprising: Using a task processing model, multiple sample response contents are obtained based on sample dialogue data; The multiple sample responses are compared and analyzed to obtain response indicators corresponding to each sample response. The response indicators are used to measure the quality of the corresponding sample responses. The response indicator corresponding to the first sample response is obtained by comparing the first sample response with the sample comparison content, which is selected from the second sample response content. The first sample response is any one of the multiple sample responses, and the second sample response is the sample response content other than the first sample response. The task processing model is trained based on the response metrics to obtain a trained task processing model.

2. The method according to claim 1, wherein comparing and analyzing the multiple sample responses to obtain response indicators corresponding to each of the multiple sample responses includes: For the response content of the first sample, at least one of the sample comparison contents is selected from the response content of the second sample; By comparing the response content of the first sample with the comparison content of the sample, the response index corresponding to the response content of the first sample is obtained.

3. The method according to claim 2, wherein comparing the first sample response content with the sample comparison content to obtain the response index corresponding to the first sample response content includes: Based on the sample dialogue data, the response content of the first sample is compared with the sample comparison content to obtain the response index corresponding to the response content of the first sample.

4. The method according to claim 2, wherein comparing the first sample response content with the sample comparison content to obtain the response index corresponding to the first sample response content includes: The comparison prompt information, the first sample response content, and the sample comparison content are input into the content comparison model to obtain the response index corresponding to the first sample response content. The comparison prompt information is used to guide the content comparison model to compare the first sample response content and the sample comparison content to obtain the response index.

5. The method according to claim 2, wherein comparing the first sample response content with the sample comparison content to obtain the response index corresponding to the first sample response content includes: Based on the content comparison strategy, the response content of the first sample is compared with the sample comparison content to obtain the response index corresponding to the response content of the first sample. The content comparison strategy is constructed based on at least one quality assessment dimension.

6. The method according to any one of claims 1 to 5, wherein before training the task processing model based on the response index to obtain the trained task processing model, the method further comprises: Based on the content length of the sample responses, the response metrics are constrained to obtain the target response metrics; The step of training the task processing model based on the response metrics to obtain the trained task processing model includes: The task processing model is trained based on the target response metric to obtain a trained task processing model.

7. The method according to claim 6, wherein constraining the response index based on the content length of the sample response content to obtain the target response index includes: Based on the content length of the sample responses, determine the content metrics corresponding to the multiple sample responses respectively; The content metrics and the response metrics are fused to obtain intermediate response metrics; The intermediate response indicators are standardized to obtain the target response indicators.

8. The method according to claim 7, wherein determining the content index corresponding to each of the plurality of sample response contents based on the content length of the sample response content includes: Obtain a first length parameter and a second length parameter, wherein the first length parameter is greater than the second length parameter, the first length parameter refers to the upper limit constraint of the length of the sample response content, the second length parameter refers to the length of the soft penalty interval, and the second length parameter is used to define the penalty transition region; Based on the first length parameter, the second length parameter, and the content length of the sample response content, the content indicators corresponding to the multiple sample response contents are determined respectively.

9. The method according to claim 1, wherein training the task processing model based on the response index to obtain a trained task processing model comprises: Based on the response metrics, a policy loss is calculated, wherein the policy loss is used to measure the difference between the model performance of the task processing model and the expected model performance; Based on the responses from the multiple samples, a model difference index is calculated, wherein the model difference index is used to measure the difference between the task processing model and the historical task processing model, and the historical task processing model is the model version of the task processing model at a previous time point. The task processing model is trained based on the policy loss and the model difference index to obtain a trained task processing model.

10. A method for training a role-playing model, comprising: Using a role-playing model, multiple sample response contents were obtained based on sample dialogue data; The multiple sample responses are compared and analyzed to obtain response indicators corresponding to each sample response. The response indicators are used to measure the quality of the corresponding sample responses. The response indicator corresponding to the first sample response is obtained by comparing the first sample response with the sample comparison content, which is selected from the second sample response content. The first sample response is any one of the multiple sample responses, and the second sample response is the sample response content other than the first sample response. The role-playing model is trained based on the response metrics to obtain a fully trained role-playing model.

11. A task processing method, comprising: Obtain the task data for the target task; The task data is input into the task processing model to obtain the task processing result of the target task, wherein the task processing model is trained based on the method described in any one of claims 1 to 9.

12. A request processing method based on a task processing model, applied to a task platform, comprising: Receive model requests sent by terminal devices; Based on the model request, a target task processing model is determined from a plurality of task processing models, wherein the plurality of task processing models are trained based on the method described in any one of claims 1 to 9.

13. The method according to claim 12, wherein determining the target task processing model from multiple task processing models based on the model request includes: If the model request includes a scene identifier of the target scene, a target task processing model suitable for the target scene is searched from the first model library based on the scene identifier. The first model library stores multiple task processing models suitable for different scenes. When the model request includes scene input data of the target scene, a task processing model to be trained that is suitable for the target scene is determined from the plurality of task processing models, and the task processing model to be trained is trained based on the scene input data to obtain the target task processing model. If the model request includes model specification parameters, the target task processing model corresponding to the model specification parameters is searched from the second model library, wherein the second model library stores multiple task processing models with different model specification parameters.

14. The method according to claim 12, further comprising, after determining the target task processing model from multiple task processing models based on the model request: Deploy the target task processing model, and based on the target task processing model, construct a task processing interface so that the terminal device can schedule the target task processing model to execute the target task through the task processing interface.

15. A task platform, comprising a request interface and a response unit; The request interface is used to receive model requests sent by the terminal device, wherein... The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters. The response unit is configured to determine a target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on the method described in any one of claims 1 to 9.

16. The task platform according to claim 15, further comprising a task processing interface, wherein the task processing interface is constructed based on the target task processing model; The task processing interface is used for the terminal device to schedule and execute target tasks.

17. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 14.

18. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 14.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.

Images