Task processing model training method, role playing model training method, and task processing method

Abstract

Embodiments of the present specification provide a task processing model training method, a role-playing model training method and a task processing method, wherein the task processing model training method comprises: obtaining a plurality of sample reply contents, the plurality of sample reply contents being reply contents generated by a task processing model based on sample dialogue data; inputting the plurality of sample reply contents into a target evaluation model to obtain target reply indicators corresponding to the plurality of sample reply contents respectively, the target reply indicators being used to measure the quality of the corresponding sample reply contents, the target evaluation model being obtained by reinforcement learning of a cold start evaluation model based on a meta evaluation model, the meta evaluation model being trained based on sample dialogue data, a plurality of sample reply contents and sample reply analysis information; and adjusting parameters of the task processing model according to the target reply indicators to obtain a trained task processing model. Based on the target reply indicators, the training efficiency and alignment performance of the task processing model are improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to methods for training task processing models, methods for training role-playing models, and methods for task processing. Background Technology

[0002] With the continuous advancement of artificial intelligence technology, large models are increasingly being used in tasks such as dialogue, creation, and role-playing. How to make the content generated by large models more aligned with human preferences and values ​​has gradually become a key research focus for improving user experience. Evaluation models, as a technique capable of quantifying human preferences, are widely used to align the generative behavior of large models.

[0003] Currently, traditional evaluation models are trained based on the partial order relationships between multiple candidate responses annotated manually. However, due to the lack of unified evaluation criteria, different annotators may make different judgments on the quality of responses based on their personality, cultural background, or differences in understanding. This results in a generally low consistency rate of manual annotation, which seriously restricts the training effect of the evaluation model and the subsequent model alignment performance. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a task processing model training method. One or more embodiments of this specification simultaneously relate to a role-playing model training method, a task processing method, a target evaluation model training method, a model evaluation method, a role-playing model evaluation method, a request processing method based on a task processing model, a task platform, a task processing model training device, a role-playing model training device, a task processing device, a target evaluation model training device, a model evaluation device, a role-playing model evaluation device, a request processing device based on a task processing model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for training a task processing model is provided, comprising: acquiring multiple sample response contents, wherein the multiple sample response contents are response contents generated by the task processing model based on sample dialogue data; inputting the multiple sample response contents into a target evaluation model to obtain target response indicators corresponding to the multiple sample response contents, wherein the target response indicators are used to measure the quality of the corresponding sample response contents, the target evaluation model is obtained by reinforcement learning of a cold start evaluation model based on a meta-evaluation model, and the meta-evaluation model is trained based on sample dialogue data, multiple sample response contents, and sample response analysis information; and adjusting the parameters of the task processing model according to the target response indicators to obtain a trained task processing model.

[0006] One embodiment of this specification provides a task processing model training method that uses a meta-evaluation model to guide reinforcement learning training to obtain a target evaluation model. This eliminates the reliance on low-quality, inconsistent manual partial ordering labels during the target evaluation model's training process. Instead, it utilizes the meta-evaluation model to integrate sample dialogue data, multiple sample response contents, and sample response analysis information for a more stable and fine-grained quality assessment. This significantly improves the accuracy and robustness of the target evaluation model in judging the quality of sample response content. Furthermore, the target response index output by the target evaluation model can provide richer feedback signals for the task processing model. Compared to traditional ranking labels, it can more precisely guide the optimization direction of the task processing model, thereby improving the training efficiency and alignment performance of the task processing model. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating a task processing model training method provided in one embodiment of this specification;

[0008] Figure 2 This is a schematic diagram of the training process of a task processing model training method provided in one embodiment of this specification;

[0009] Figure 3 This is a flowchart illustrating a role-playing model training method provided in one embodiment of this specification;

[0010] Figure 4 This is a flowchart illustrating a task processing method provided in one embodiment of this specification;

[0011] Figure 5 This is an architecture diagram of a task processing system provided in one embodiment of this specification;

[0012] Figure 6 This is a flowchart illustrating a target evaluation model training method provided in one embodiment of this specification;

[0013] Figure 7 This is a flowchart of a model evaluation method provided in one embodiment of this specification;

[0014] Figure 8 This is a flowchart of a role-playing model evaluation method provided in one embodiment of this specification;

[0015] Figure 9 This is a flowchart illustrating a request processing method based on a task processing model, as provided in one embodiment of this specification.

[0016] Figure 10 This is a schematic diagram of the structure of a task platform provided in one embodiment of this specification;

[0017] Figure 11This is a schematic diagram of the structure of a task processing model training device provided in one embodiment of this specification;

[0018] Figure 12 This is a schematic diagram of the structure of a role-playing model training device provided in one embodiment of this specification;

[0019] Figure 13 This is a schematic diagram of the structure of a task processing device provided in one embodiment of this specification;

[0020] Figure 14 This is a schematic diagram of the structure of a target evaluation model training device provided in one embodiment of this specification;

[0021] Figure 15 This is a schematic diagram of the structure of a model evaluation device provided in one embodiment of this specification;

[0022] Figure 16 This is a schematic diagram of the structure of a role-playing model evaluation device provided in one embodiment of this specification;

[0023] Figure 17 This is a schematic diagram of the structure of a request processing device based on a task processing model provided in one embodiment of this specification;

[0024] Figure 18 This is a structural block diagram of a computing device provided in one embodiment of this specification;

[0025] Figure 19 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0026] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0027] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0028] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0029] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0030] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0031] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0032] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0033] Supervised fine-tuning (SFT) is a method of further training a model based on a pre-trained model. In this method, the model is trained on a dataset containing pairs of inputs and desired outputs so that it learns how to generate responses closer to human-level performance. Supervised fine-tuning is often used to adapt a model to task-specific or domain-specific data, thereby improving its performance on those tasks.

[0034] Reinforcement Learning (RL) is a machine learning paradigm whose core idea is to enable an agent to learn how to take better actions in specific situations through continuous interaction with the environment, in order to maximize a certain cumulative reward signal. Utilizing reward models, in the training of LLMs, reinforcement learning can often fine-tune the LLM based on an evaluation model (also known as a reward model), making the text generated by the LLM more consistent with human preferences.

[0035] Group Relatively Policy Optimization (GRPO) is a reinforcement learning algorithm that uses normalized rewards from multiple responses to calculate the advantage value for each response and then updates the model parameters.

[0036] Proximal Policy Optimization (PPO) is a reinforcement learning algorithm that optimizes policies through advantage function estimation and KL divergence constraints.

[0037] Human Feedback Reinforcement Learning (RLHF) is a machine learning approach that combines traditional reinforcement learning with human guidance. In traditional reinforcement learning, the agent learns how to take actions to maximize cumulative rewards by interacting with the environment and receiving reward signals from it. In the RLHF framework, in addition to rewards provided by the environment, direct feedback from human experts or ordinary users is introduced as additional learning, optimizing the task processing model through reinforcement learning.

[0038] Alignment refers to the process and goal of aligning the goals, behaviors, and values ​​of a task processing model with human preferences, intentions, interests, and ethical norms.

[0039] Evaluation Model (RM): Also known as the reward model, it is trained on human feedback data and can judge the output quality of the task processing model and give an evaluation response metric (also known as the reward value).

[0040] Generative Reward Model (GRM): This refers to an evaluation model built on generative models, which can first generate a natural language analysis and then provide response metrics.

[0041] Meta-feedback refers to human feedback to an evaluation model. Since the evaluation model's role is to provide feedback to the task processing model, "feedback to the evaluation model" can be called "meta-feedback."

[0042] Deep self-attention (Transformer) models are network structures based on multi-head self-attention mechanisms, primarily used for processing sequential data. A Transformer model consists of repeatedly stacked encoder and decoder units. This design allows the Transformer to efficiently learn long-term dependencies, making it suitable for various natural language processing tasks, including machine translation, text summarization, and question answering systems.

[0043] Bidirectional Encoder Representations from Transformers (BERT) is a pre-trained NLP model. By learning from large amounts of unlabeled text data, this model can capture deep semantic information from text and achieves significant performance improvements on numerous NLP tasks.

[0044] The Text-to-Text Transfer Transformer (T5) model is an NLP model. The main characteristic of the T5 model is that it unifies all NLP tasks into a text-to-text format, meaning both the input and output are text sequences. This design makes the model more adaptable to various tasks, such as translation, question answering, and summarization.

[0045] RLHF is a common alignment method for LLM (Low-Level Metrics). The traditional RLHF process includes two stages: evaluation modeling and reinforcement learning. First, in the evaluation modeling stage, human annotators label the partial order relations (rankings between responses) of multiple candidate responses generated by the LLM, constructing a partially ordered training set and training an evaluation model. Then, in the reinforcement learning stage, the evaluation model is used to optimize the LLM through reinforcement learning algorithms, aligning it to human preferences.

[0046] Taking open-ended tasks such as role-playing as an example, the diverse evaluation criteria and differing annotation preferences among annotators lead to low consistency rates for partial order relations. Empirically, manual consistency rates can only reach 60% to 70%. To obtain high-quality partial order relation labels, each sample requires multiple annotators to simultaneously annotate and vote, resulting in low annotation efficiency. Conversely, reducing the number of annotators per sample to increase efficiency further lowers the quality of partial order relation labels. Therefore, in open-ended tasks, it is difficult to annotate large-scale, high-quality partial order data, hindering the alignment work of large models.

[0047] To address the aforementioned issues, this embodiment incorporates the analysis process of evaluating responses into the evaluation modeling, particularly when the quality of partial order labels is low. This improves the effectiveness and interpretability of the evaluation model. Specifically, a meta-evaluation model is trained based on sample dialogue data, multiple sample response contents, and sample response analysis information. Then, the meta-evaluation model provides reward signals, which are used to train the cold-start evaluation model using reinforcement learning to obtain the target evaluation model. During reinforcement learning, not only positive reward signals can be utilized, but also erroneous outputs of the model can be penalized. After obtaining the target evaluation model, it can be used to optimize the task processing model using reinforcement learning, resulting in a fully trained task processing model. This aligns the responses of the task processing model with human preferences, improving the response quality of the task processing model.

[0048] This specification provides a task processing model training method, and also relates to a role-playing model training method, a task processing method, a target evaluation model training method, a model evaluation method, a role-playing model evaluation method, a request processing method based on a task processing model, a task platform, a task processing model training device, a role-playing model training device, a task processing device, a target evaluation model training device, a model evaluation device, a role-playing model evaluation device, a request processing device based on a task processing model, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0049] See Figure 1 , Figure 1 This specification illustrates a flowchart of a task processing model training method according to an embodiment, which specifically includes the following steps:

[0050] Step 102: Obtain multiple sample response contents, which are response contents generated by the task processing model based on sample dialogue data.

[0051] It's important to note that a task processing model refers to a machine learning model designed to perform a specific task (such as role-playing, dialogue response generation, question answering, etc.). A task processing model can be a general, pre-trained large model or a domain-specific model fine-tuned for a particular task. It has the ability to receive input (such as user dialogue history or questions) and generate corresponding output (such as response content). During training, the model parameters can be adjusted using methods such as reinforcement learning to improve task performance.

[0052] Sample dialogue data refers to a set of real or constructed dialogue data used to train a task processing model. Sample dialogue data can be question-and-answer data with a dialogue objective or chat data without a dialogue objective. Sample dialogue data includes user input (such as questions or commands) and can also include dialogue role settings and context. Sample dialogue data can be data of different modalities, such as text, speech, video, and images. This data is provided as input to the task processing model to generate corresponding sample responses. There can be multiple sets of sample dialogue data. During each training iteration of the task processing model, a batch of data can be extracted to update the model parameters. After multiple iterations, reinforcement learning training is completed.

[0053] Multiple sample responses refer to the various responses generated by the task processing model for a single sample dialogue data through multiple sampling processes (such as using different generation strategies or random sampling). These multiple sample responses may differ in expression, information completeness, or quality, providing diverse objects for the subsequent evaluation process of the target assessment model.

[0054] In practical applications, there are various ways to obtain multiple sample response contents, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in the embodiments. In one possible implementation, sample dialogue data sent by the user through a client can be received, and the sample dialogue data can be input into a task processing model to obtain multiple sample response contents. In another possible implementation, multiple sample response contents can be read from other data acquisition devices or databases.

[0055] Step 104: Input multiple sample responses into the target evaluation model to obtain the target response index corresponding to each sample response. The target response index is used to measure the quality of the corresponding sample response. The target evaluation model is obtained by reinforcement learning of the cold start evaluation model based on the meta-evaluation model. The meta-evaluation model is trained based on sample dialogue data, multiple sample responses, and sample response analysis information.

[0056] It should be noted that the target response metric refers to the quality score output by the target evaluation model for each sample response, used to quantify the overall performance of the sample response across preset quality evaluation dimensions (such as relevance, fluency, factual accuracy, and security). The target response metric can be a single-dimensional evaluation result (such as fluency) or a weighted result combining multiple dimensions. The target response metric can serve as a reward signal in reinforcement learning, guiding parameter updates for the task processing model. It is worth noting that the sample dialogue data and multiple sample response contents used to train the cold-start evaluation model, meta-evaluation model, and target evaluation model can be the same or different, depending on the specific circumstances. This specification does not impose any limitations on this aspect in the embodiments.

[0057] Step 106: Adjust the parameters of the task processing model according to the target response index to obtain the trained task processing model.

[0058] It should be noted that when adjusting the parameters of the task processing model based on the target response metric, parameter adjustment refers to the process of updating the internal parameters of the task processing model through reinforcement learning algorithms (such as GRPO, PPO and their derivative algorithms) based on the reward signal fed back by the target response metric. The purpose is to enable the task processing model to generate higher quality responses, that is, to generate content that can obtain higher target response metrics.

[0059] A fully trained task processing model refers to a task processing model trained using reinforcement learning based on the target response metric. A fully trained task processing model possesses stronger generative capabilities, able to generate responses that better meet expected quality standards from the input dialogue data, and can be used for practical deployment and application.

[0060] The scheme described in this embodiment uses a meta-evaluation model to guide reinforcement learning training to obtain a target evaluation model, which improves the accuracy and robustness of the target evaluation model in judging the quality of sample responses. The target response index output by the target evaluation model can provide richer feedback signals to the task processing model, and can more precisely guide the optimization direction of the task processing model, thereby improving the training efficiency and alignment performance of the task processing model.

[0061] In one optional embodiment of this specification, the training process of the cold-start evaluation model used in the task processing model training method is described. That is, before inputting multiple sample response contents into the target evaluation model to obtain the target response indicators corresponding to the multiple sample response contents, the following steps may be included:

[0062] Obtain sample dialogue data and multiple sample response contents.

[0063] The response process of multiple sample responses is analyzed to obtain the sample analysis path and the sample response indicators corresponding to each sample response.

[0064] Based on sample dialogue data, multiple sample response contents, sample analysis paths, and sample response metrics, the parameters of the initial evaluation model are adjusted to obtain a trained cold start evaluation model.

[0065] It should be noted that response process analysis refers to the process of analyzing the evaluation dimensions and logic of sample responses. Response process analysis can be achieved through manual analysis, visualization of attention mechanisms, rule matching, or auxiliary analysis models, thereby revealing the sample analysis path behind the sample responses.

[0066] In one optional embodiment of this specification, when analyzing the response process of multiple sample responses, the response process can be analyzed individually for each sample response to obtain the sample analysis path and sample response index corresponding to each sample response. Alternatively, a comparative analysis of the response processes of multiple sample responses can be performed to obtain a single sample analysis path and sample response index corresponding to each sample response.

[0067] In another optional embodiment of this specification, when analyzing the response process of multiple sample responses, the analysis can be performed on only the sample responses to obtain the sample analysis path and the sample response indicators corresponding to each sample response. Alternatively, sample dialogue data can be introduced, and the correlation between the sample dialogue data and the sample response content can be considered during the response analysis to obtain the sample analysis path and the sample response indicators corresponding to each sample response.

[0068] A sample analysis path refers to the analytical process information obtained by analyzing the response process of multiple sample responses. The sample analysis path describes the methods for judging the quality of sample responses (e.g., the fluency of the response can determine its quality) and the characteristics of sample responses of different qualities (e.g., high-quality responses have higher fluency, while low-quality responses have lower fluency). Through the sample analysis path, the quality of sample responses can be evaluated, such as their reasonableness, interpretability, and logical completeness.

[0069] Sample response metrics refer to the quality scores generated for each sample response content through response process analysis. These scores quantify the overall performance of the sample response content across preset quality assessment dimensions (such as relevance, fluency, factual accuracy, and security). Sample response metrics can be manually annotated scores or quantitative scores generated by automated analysis tools.

[0070] The initial evaluation model refers to an evaluation model to be trained. It can be an untrained natural language processing model or a pre-trained large model; for example, it could be a neural network model based on the Transformer architecture (such as BERT or T5). The initial evaluation model has the ability to analyze the response process from sample dialogue data and multiple sample responses. That is, it has the ability to receive sample dialogue data and multiple sample responses as input, and output predictive analysis paths and predicted response metrics corresponding to each of the multiple sample responses.

[0071] When adjusting the parameters of the initial evaluation model based on sample dialogue data, multiple sample responses, sample analysis paths, and sample response metrics, parameter adjustment refers to the process of updating the model parameters of the initial evaluation model using a training set containing sample dialogue data, sample response content, sample analysis paths, and sample response metrics, through supervised fine-tuning or regression learning. The goal of parameter adjustment is to make the output of the initial evaluation model as close as possible to the actual sample analysis paths and sample response metrics.

[0072] A cold-start evaluation model refers to an evaluation model obtained by fully training an initial evaluation model. Since the initial evaluation model has poor evaluation capabilities and is unsuitable for direct reinforcement learning, it can be cold-started using SFT (Solution-First Fourier Transform) to obtain a cold-start evaluation model. This cold-start evaluation model has learned how to generate accurate sample analysis paths and sample response metrics based on sample dialogue data and sample response content, providing reliable reward signals for subsequent reinforcement learning or alignment tasks. It's worth noting that because the cold-start evaluation model can generate not only sample response metrics but also sample analysis paths, it can be understood as a generative evaluation model. Replacing traditional evaluation models with generative evaluation models can improve the effectiveness and interpretability of evaluation models on open-ended tasks.

[0073] For example, firstly, sample dialogue data is obtained. By task processing model Perform multiple samplings to generate Different sample response content , forming a set ,in, It is a positive integer greater than 1. Next, for... Analyze the response process to obtain the sample analysis path. and give each Sample response metrics , forming a set Then, according to the set and set Building the training set for the initial evaluation model Finally, in Based on this, the parameters of the initial evaluation model are adjusted (e.g., SFT) to obtain the cold start evaluation model. This cold start assessment model has the ability to receive... and Output and The ability.

[0074] The scheme implemented in this specification, through a response process analysis mechanism, not only focuses on the sample response content generated by the task processing model but also delves into the underlying sample analysis path. Combined with sample response metrics, a more comprehensive and interpretable evaluation model training set is constructed. Compared to traditional evaluation model training methods that rely solely on manual preference ranking, one or more embodiments of this specification utilize sample analysis paths and fine-grained sample response metrics to significantly improve the accuracy and robustness of the cold-start evaluation model in judging response quality, alleviating the problem of inconsistent manual annotation. Simultaneously, by using the sample analysis path as one of the model's input features, the cold-start evaluation model can understand "what kind of response is better and what features make a response better," providing high-quality, low-noise feedback signals for the alignment of subsequent task processing models. This effectively reduces reliance on costly manual annotation, improving overall alignment efficiency and model performance.

[0075] In one optional embodiment of this specification, taking the cold start evaluation model trained via SFT as an example, the above-mentioned parameter adjustment of the initial evaluation model based on sample dialogue data, multiple sample response contents, sample analysis paths, and sample response metrics to obtain the trained cold start evaluation model may include the following steps:

[0076] The sample dialogue data and multiple sample responses are input into the initial evaluation model to analyze the response process and obtain the predictive analysis path and the predictive response indicators corresponding to the multiple sample responses.

[0077] Based on the sample analysis path and the predictive analysis path, calculate the path prediction error, and based on the sample response index and the predicted response index, calculate the index prediction error.

[0078] Based on the path prediction error and the indicator prediction error, the parameters of the initial evaluation model are adjusted to obtain the trained cold start evaluation model.

[0079] It should be noted that the predictive analysis path refers to the analysis path predicted by the initial evaluation model after receiving sample dialogue data and sample response content. The predictive response index refers to the quality score output by the initial evaluation model for each sample response content, reflecting its judgment on the quality of that sample response content. The predictive analysis path and predictive response index are manifestations of the initial evaluation model's understanding and may differ from the actual sample analysis path and sample response index. They are used for subsequent comparison with the actual sample analysis path and sample response index to calculate errors.

[0080] Path prediction error measures the deviation between the predicted analysis path and the actual sample analysis path predicted by the initial evaluation model. Path prediction error reflects the accuracy of the initial evaluation model in terms of reasoning logic or decision-making process. Indicator prediction error measures the deviation between the predicted response indicator and the actual sample response indicator predicted by the initial evaluation model. Indicator prediction error reflects the accuracy of the initial evaluation model in terms of quantitative evaluation.

[0081] When adjusting the parameters of the initial evaluation model based on the sample analysis path, predictive analysis path, sample response index, and predictive response index, parameter adjustment refers to the process of optimizing the model parameters of the initial evaluation model by backpropagating gradients through loss functions (such as cross-entropy, mean squared error, etc.) based on the path prediction error between the sample analysis path and the predictive analysis path, and the index prediction error between the sample response index and the predictive response index. The goal is to make the predictive analysis path and the predictive response index as close as possible to the true values ​​(sample analysis path and sample response index).

[0082] By applying the scheme of the embodiments in this specification, sample dialogue data and sample response content are input into the initial evaluation model to obtain predictive analysis paths and predictive response indicators. These are then compared with highly reliable sample analysis paths and sample response indicators, reducing the impact of subjective bias. This allows the finally trained cold-start evaluation model to more reliably guide the alignment optimization of the task processing model, reducing reliance on large-scale manual annotation, and improving overall training efficiency and system robustness.

[0083] In the training process of a cold-start evaluation model, taking manual annotation as an example, humans not only need to provide sample response metrics but also need to write sample analysis paths, leading to high task complexity and cost. Furthermore, performing SFT on manual annotation provides positive training signals but cannot penalize the model's erroneous outputs. Based on this, the embodiments in this specification train a meta-evaluation model. In the training process of the meta-evaluation model, taking manual annotation as an example, humans only need to provide evaluation metrics for each sample response analysis information through scoring, making annotation simpler and more efficient. Moreover, by utilizing the meta-evaluation model and reinforcement learning, not only positive training signals can be utilized, but also erroneous outputs of the model can be penalized.

[0084] In one optional embodiment of this specification, the training process of the meta-evaluation model used in the task processing model training method is described. That is, before inputting multiple sample response contents into the target evaluation model to obtain the target response indicators corresponding to the multiple sample response contents, the following steps may be included:

[0085] Obtain sample dialogue data and multiple sample response contents.

[0086] The sample dialogue data and multiple sample responses are input into the cold start evaluation model to analyze the response process and obtain multiple sets of sample response analysis information. Each set of sample response analysis information includes multiple auxiliary response indicators and corresponding auxiliary analysis paths. The auxiliary response indicators correspond one-to-one with the sample response content, and different sample response analysis information corresponds to different auxiliary analysis paths.

[0087] Based on sample dialogue data, multiple sample response contents, and analysis information from multiple sets of sample responses, the parameters of the initial evaluation model are adjusted to obtain a trained meta-evaluation model.

[0088] It should be noted that the sample response analysis information refers to the complete set of analysis results output by the cold start evaluation model after analyzing the response process of sample dialogue data and multiple sample responses. Each set of sample response analysis information includes multiple auxiliary response indicators and the corresponding auxiliary analysis paths.

[0089] Auxiliary response indicators are quality scores generated for each sample response content by the cold start assessment model. There is a one-to-one correspondence between auxiliary response indicators and sample response content. Auxiliary analysis paths refer to the analytical information inferred by the cold start assessment model corresponding to a specific set of auxiliary response indicators. The auxiliary analysis path describes the method used to judge the quality of sample response content when generating auxiliary response indicators, as well as the characteristics of sample response content of different quality levels.

[0090] A meta-evaluation model is an evaluation model obtained by adjusting the parameters of an initial evaluation model. The meta-evaluation model has learned how to generate accurate evaluation metrics based on sample dialogue data, multiple sample response contents, and analysis information from multiple sets of sample responses. This can provide reliable reward signals for subsequent reinforcement learning or alignment tasks.

[0091] When adjusting the parameters of the initial evaluation model based on sample dialogue data, multiple sample response contents, and analysis information from multiple sets of sample responses, parameter adjustment refers to the process of updating the model parameters of the initial evaluation model using training samples constructed from sample dialogue data, sample response contents, and analysis information from multiple sets of sample responses, through supervised fine-tuning or contrastive learning. The goal of parameter adjustment is to make the output of the meta-evaluation model as close as possible to the evaluation metrics based on the actual sample response analysis information.

[0092] For example, referring to the above example, sample dialogue data is obtained. By task processing model Perform multiple samplings to generate Different sample response content , forming a set Given sample dialogue data and set The cold start evaluation model Perform multiple samplings to generate A set of different auxiliary analysis paths and corresponding auxiliary response indicators, this Group information constitutes a set ,in, It is the first One auxiliary analysis path, It is the first Group auxiliary response indicators.

[0093] By applying the scheme of the embodiments in this specification, the cold start evaluation model is used to sample the same input multiple times to generate diverse sample response analysis information. On this basis, the meta-evaluation model is trained to learn how to form evaluation indicators of sample response analysis information based on sample dialogue data, multiple sample response contents, and multiple sets of sample response analysis information. This reduces the amount and difficulty of annotation in the training process of the meta-evaluation model and obtains a high-precision meta-evaluation model.

[0094] In one optional embodiment of this specification, taking the training of a meta-evaluation model via SFT as an example, the above-mentioned parameter adjustment of the initial evaluation model based on sample dialogue data, multiple sample response contents, and multiple sets of sample response analysis information to obtain the trained meta-evaluation model may include the following steps:

[0095] The quality of the response analysis information from multiple groups of samples was assessed to obtain evaluation indicators for the sample response analysis information.

[0096] The sample dialogue data, multiple sample response contents, and analysis information of multiple sets of sample responses are input into the initial evaluation model to obtain the prediction index.

[0097] Based on the prediction and evaluation indicators, the parameters of the initial evaluation model are adjusted to obtain the meta-evaluation model.

[0098] It should be noted that the evaluation metrics are quantitative scores of the overall quality of the response analysis information for each group of samples, and can be used to measure the quality of the sample response analysis information. Evaluation metrics can be obtained through methods such as manual annotation, scoring by a high-confidence benchmark model, and rule-based judgment, and serve as supervisory signals for training the meta-evaluation model.

[0099] Taking manual annotation as an example, for multiple sample responses generated by the task processing model, the cold-start evaluation model first generates multiple sets of sample response analysis information. Human annotators then select from these sets, determining which set's auxiliary response metrics and analysis paths best align with their preferences, thus annotating the evaluation metrics. Compared to the traditional RLHF scheme, this stage of manual annotation targets the cold-start evaluation model's responses, not the task processing model's. Therefore, the evaluation metrics of the sample response analysis information can be called meta-feedback. After collecting the meta-feedback, a meta-reward model can be trained. This model provides reward signals, which are then used to train the cold-start evaluation model using reinforcement learning. Reinforcement learning utilizes both positive training signals and penalizes incorrect model outputs. After completing the reinforcement learning of the cold-start evaluation model, a target evaluation model is obtained, which can then be used to train the task processing model for reinforcement learning optimization. By having humans provide meta-feedback to the cold start evaluation model, and then having the cold start evaluation model provide feedback to the task processing model, the efficiency and quality of human feedback are improved, thereby enhancing the alignment effect of the task processing model.

[0100] Predictive metrics refer to the predicted scores output by the initial evaluation model after receiving sample dialogue data, sample response content, and analysis information from multiple sets of sample responses, assessing the quality of each set of sample response analysis information. The predicted scores reflect the initial evaluation model's preference for auxiliary analysis paths and auxiliary response metrics within the predicted scores, and are compared with the actual evaluation metrics to optimize the initial evaluation model.

[0101] When adjusting the parameters of the initial evaluation model based on the predicted and evaluation metrics, parameter adjustment refers to the process of constructing a loss function (such as mean squared error, cross-entropy, etc.) based on the difference between the predicted and actual evaluation metrics, and updating the model parameters of the initial evaluation model through the backpropagation algorithm. The goal is to make the predicted metrics as close as possible to the evaluation metrics.

[0102] For example, referring to the above example, firstly, for the cold start evaluation model The generated set The quality is judged manually and for Each output in Scoring is used to obtain a set of evaluation indicators. ,in The evaluation metrics are then represented. Next, a batch of data is labeled to form a meta-feedback dataset. Finally, using The parameters of the initial evaluation model are adjusted to obtain the meta-evaluation model. This meta-evaluation model has the ability to receive... , as well as Output Each The ability to evaluate indicators.

[0103] The scheme implemented in this specification introduces a "quality re-evaluation" mechanism to score the quality of multiple sets of sample response analysis information (i.e., evaluation index), and trains an initial evaluation model to predict this evaluation index. This allows the initial evaluation model to gradually evolve into a meta-evaluation model that can provide feedback signals with a higher signal-to-noise ratio for subsequent task processing model alignment. This reduces the reliance on manual annotation while improving alignment efficiency and generation quality.

[0104] In one optional embodiment of this specification, the training process of the target evaluation model used in the task processing model training method is described. That is, before inputting multiple sample response contents into the target evaluation model to obtain the target response indicators corresponding to the multiple sample response contents, the following steps may be included:

[0105] Obtain sample dialogue data and multiple sample response contents.

[0106] Input the sample dialogue data and multiple sample responses into the cold start evaluation model to obtain the cold start response indicators corresponding to each of the multiple sample responses.

[0107] Input the cold start response metrics into the meta-evaluation model to obtain the meta-evaluation metrics.

[0108] Based on the meta-evaluation metrics, the parameters of the cold start evaluation model are adjusted to obtain the target evaluation model after training.

[0109] It should be noted that the cold start response index refers to the quality score output by the cold start assessment model for each sample response, which can be used to measure the quality of the sample response. The cold start response index reflects the current capability of the cold start assessment model and serves as an input signal for the meta-assessment model.

[0110] The meta-evaluation metric refers to the reward signal output by the meta-evaluation model, used to optimize the cold-start evaluation model. The meta-evaluation metric measures the overall quality of the cold-start response metrics. When adjusting the parameters of the cold-start evaluation model based on the meta-evaluation metric, parameter adjustment refers to the process of using the meta-evaluation metric as a reward signal to perform reinforcement learning (reinforcement learning algorithms such as GRPO, PPO, and their derivatives) on the cold-start evaluation model. The goal is to gradually bring the cold-start response metrics output by the cold-start evaluation model closer to the high-quality evaluation criteria recognized by the meta-evaluation model.

[0111] For example, referring to the above example, in the cold start evaluation model Based on this, the meta-evaluation model is used. The provided meta-evaluation signals, for By optimizing with reinforcement learning, a better target evaluation model that more closely resembles human feedback can be obtained. In obtaining Then, reinforcement learning algorithms are used to improve the task processing model. By performing reinforcement learning optimization, a better-performing task processing model can be obtained. .

[0112] By applying the scheme of the embodiments in this specification, the cold start response index output by the cold start evaluation model is re-evaluated using a meta-evaluation model to generate a higher-level meta-evaluation index. This reward signal is then used to optimize the cold start evaluation model itself, achieving iterative improvement in evaluation capabilities. The final trained target evaluation model has stronger generalization ability and robustness, and its output target response index is more reliable. This can effectively improve the efficiency and quality of subsequent task processing model alignment, reducing reliance on large-scale manual annotation while pushing the results of task processing models closer to high-quality, multi-dimensional human preference standards.

[0113] See Figure 2 , Figure 2 This specification illustrates a schematic diagram of the training process of a task processing model training method according to an embodiment, specifically including:

[0114] Obtain sample dialogue data; input the sample dialogue data into the task processing model to obtain multiple sample response contents.

[0115] The response process of multiple sample responses was analyzed, and the sample analysis path and corresponding sample response indicators for each sample response were manually labeled. Based on the sample dialogue data, multiple sample response contents, sample analysis path, and sample response indicators, the parameters of the initial evaluation model were adjusted to obtain a trained cold-start evaluation model.

[0116] The sample dialogue data and multiple sample responses are input into the trained cold-start evaluation model to analyze the response process and obtain multiple sets of sample response analysis information. Based on the sample dialogue data, multiple sample response content, and multiple sets of sample response analysis information, the parameters of the initial evaluation model are adjusted to obtain the trained meta-evaluation model.

[0117] Based on the trained meta-evaluation model, reinforcement learning is performed on the trained cold-start evaluation model to obtain the trained target evaluation model.

[0118] Input the responses from multiple samples into the trained target evaluation model to obtain the target response metrics corresponding to each sample response; adjust the parameters of the task processing model based on the target response metrics to obtain the trained task processing model.

[0119] The following is in conjunction with the appendix Figure 3 Taking the application of the task processing model training method provided in this specification in a role-playing scenario as an example, the task processing model training method will be further explained. Among other things, Figure 3 This specification shows a flowchart of a role-playing model training method according to an embodiment, which specifically includes the following steps:

[0120] Step 302: Obtain multiple sample response contents, which are response contents generated by the role-playing model based on sample dialogue data.

[0121] Step 304: Input multiple sample responses into the target evaluation model to obtain the target response index corresponding to each sample response. The target response index is used to measure the quality of the corresponding sample response. The target evaluation model is obtained by reinforcement learning of the cold start evaluation model based on the meta-evaluation model. The meta-evaluation model is trained based on sample dialogue data, multiple sample responses, and sample response analysis information.

[0122] Step 306: Adjust the parameters of the role-playing model according to the target response index to obtain the trained role-playing model.

[0123] It's important to note that role-playing models allow users to create virtual characters with personalities, memories, and behavioral logic tailored to their needs. These virtual characters can be used in a variety of scenarios, including games, emotional support, virtual assistants, education and training, and psychological counseling. The sample dialogue data input into the role-playing model includes, but is not limited to, user information, character information, and dialogue context. User information refers to background or state information about the user, used to help the role-playing model better understand the context of the dialogue and personalized needs. Character information defines the role the role-playing model should play and its related attributes. It determines the model's behavior, tone, knowledge boundaries, and personality settings. Dialogue context refers to the historical content of the current dialogue, including multi-turn dialogue records between the user and the character.

[0124] In practical applications, the implementation methods of steps 302 to 306 are the same as those of steps 102 to 106, and will not be described again in the embodiments of this specification.

[0125] The scheme implemented in this specification uses a meta-evaluation model to guide reinforcement learning training to obtain a target evaluation model, thereby improving the accuracy and robustness of the target evaluation model in judging the quality of sample responses. By providing richer feedback signals to the role-playing model through target response metrics, the optimization direction of the role-playing model can be guided more precisely, thus improving the training efficiency and alignment performance of the role-playing model.

[0126] See Figure 4 , Figure 4 This specification shows a flowchart of a task processing method according to an embodiment, which specifically includes the following steps:

[0127] Step 402: Obtain the task data for the target task.

[0128] Step 404: Input the task data into the task processing model to obtain the task processing result of the target task. The task processing model is trained based on the task processing model training method.

[0129] It should be noted that the target task refers to the specific task to be completed, such as question answering, reasoning, classification, generation, and dialogue. The target task can be a task in different scenarios, such as a multi-turn dialogue task in a role-playing scenario or a consultation task in a customer service scenario. Task data refers to the input data for the target task, used to feed into the task processing model to generate the task processing result. Task data includes at least one of the following: text, images, video, audio, and tables.

[0130] A task processing model is a deep learning model capable of generating task processing results based on input task data. Task processing models include, but are not limited to, language models, classification models, and inference models. Because task processing models are trained using specific task processing model training methods, they possess high accuracy and high alignment capabilities, generating task processing results that are accurate and conform to user preferences. The task processing result refers to the output generated by the task processing model based on the input task data; it represents the model's solution or prediction of the target task.

[0131] By applying the solutions in the embodiments of this specification, task processing results that are highly accurate and meet user preferences can be obtained by inputting the task data of the target task into a task processing model that has been trained by the task processing model training method.

[0132] Considering the large number of model parameters in the task processing model and the limited computing resources of the client, the task processing method proposed in the embodiments of this specification can be applied to, for example, Figure 5 The task processing system shown is not limited to this. See also Figure 5 , Figure 5This specification illustrates an architecture diagram of a task processing system according to an embodiment of the present specification. The task processing system may include a client 502 and a server 504.

[0133] Client 502 is used to send the target task's task data to server 504;

[0134] The server-side 504 is used to input task data into the task processing model to obtain the task processing result of the target task. The task processing model is trained based on the task processing model training method. The task processing result is sent to the client 502.

[0135] Client 502 is also used to receive task processing results sent by server 504.

[0136] like Figure 5 As shown, the task processing model is deployed in server 504. Server 504 can connect to one or more clients 502 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Client 502 may include, but is not limited to, smartphones, tablets, laptops, PDAs, personal computers (PCs), smart home devices, and in-vehicle devices. Client 502 can also interact with the user through a graphical user interface to invoke the task processing model, thereby implementing the task processing method provided in the embodiments of this specification.

[0137] It is worth noting that the task processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, if the client's runtime resources can meet the deployment and runtime conditions of the task processing model, the client can also have similar functions to the server, thereby executing the task processing methods provided in the embodiments of this specification. In other embodiments, the task processing methods provided in the embodiments of this specification can also be executed jointly by the client and the server.

[0138] See Figure 6 , Figure 6 This specification illustrates a flowchart of a target evaluation model training method according to an embodiment, which specifically includes the following steps:

[0139] Step 602: Obtain the cold start evaluation model and the meta-evaluation model. The meta-evaluation model is trained based on sample dialogue data, multiple sample response contents, and sample response analysis information. The multiple sample response contents are the response contents generated by the task processing model based on the sample dialogue data.

[0140] Step 604: Based on the meta-evaluation model, perform reinforcement learning on the cold start evaluation model to obtain the target evaluation model, wherein the target evaluation model is used in the processing of the task processing model training method.

[0141] It should be noted that the cold start evaluation model is trained based on sample dialogue data, multiple sample response contents, sample analysis paths, and sample response metrics. The training methods for the cold start evaluation model and the meta-evaluation model can be found in [reference needed]. Figure 1 The training methods for the cold start evaluation model and the meta-evaluation model in the task processing model training method shown are as follows; the implementation method of "reinforcement learning of the cold start evaluation model based on the meta-evaluation model" can be referred to the training method of the "target evaluation model" mentioned above, and will not be described again in the embodiments of this specification.

[0142] In practical applications, there are multiple ways to obtain the cold start evaluation model and the meta-evaluation model, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in the embodiments. In one possible implementation, the cold start evaluation model and the meta-evaluation model can be read from a model library. In another possible implementation, the initial evaluation model can be trained separately to obtain the cold start evaluation model and the meta-evaluation model.

[0143] By applying the scheme of the embodiments in this specification, the cold start evaluation model is reinforced by the meta-evaluation model to obtain a target evaluation model that can generate more reliable target evaluation indicators, thereby improving the evaluation accuracy and robustness of the target evaluation model and reducing the reliance on manual annotation.

[0144] See Figure 7 , Figure 7 This specification shows a flowchart of a model evaluation method provided in one embodiment, which specifically includes the following steps:

[0145] Step 702: Obtain the evaluation data of the model to be evaluated. The evaluation data includes evaluation task data and evaluation task results. The evaluation task results are the results generated by the model to be evaluated based on the evaluation task data.

[0146] Step 704: Input the evaluation task data and evaluation task results into the target evaluation model to obtain the evaluation results of the model to be evaluated. The target evaluation model is trained based on the target evaluation model training method.

[0147] It should be noted that the model to be evaluated refers to a model that requires performance assessment (such as accuracy, alignment capability, etc.). The model to be evaluated can be... Figure 1 The task processing model trained in the illustrated task processing model training method can also be other models capable of generating responses based on input task data.

[0148] Evaluation data refers to a set of data used to assess the performance of a model under evaluation, including evaluation task data and evaluation task results. Evaluation task data refers to the input content of the model under evaluation, such as user questions, dialogue context, role settings, and instructions, which trigger the model to generate a response. Evaluation task results refer to the actual output generated by the model based on the evaluation task data, i.e., the model's response result. Evaluation task results are the direct object for evaluating the model. Evaluation data can be obtained through methods such as receiving data sent by users via a client, reading data from other data acquisition devices or databases, etc.

[0149] Evaluation results refer to the final evaluation conclusion of the target evaluation model on the output of the model to be evaluated. It can be expressed as at least one quantitative indicator, such as alignment degree, quality score, preference level, etc., which can reflect the performance of the model to be evaluated on a specific task.

[0150] The solution implemented in this specification, because the target evaluation model has deep semantic understanding and multi-dimensional evaluation capabilities, can overcome the problems of high cost and low consistency of manual evaluation. Therefore, the target evaluation model can be used to achieve automatic and accurate evaluation of the model to be evaluated, which significantly improves the objectivity, efficiency and scalability of model evaluation.

[0151] See Figure 8 , Figure 8 This specification shows a flowchart of a role-playing model evaluation method according to an embodiment, which specifically includes the following steps:

[0152] Step 802: In response to the model evaluation request sent by the client for the role-playing model, obtain the evaluation data of the role-playing model. The evaluation data includes evaluation task data and evaluation task results. The evaluation task results are the results generated by the role-playing model based on the evaluation task data.

[0153] Step 804: Input the evaluation task data and evaluation task results into the target evaluation model to obtain the evaluation results of the role-playing model, and send the evaluation results to the client. The target evaluation model is trained based on the target evaluation model training method.

[0154] It should be noted that the implementation methods of steps 802 to 804 can refer to the implementation methods of steps 702 to 704 above, and will not be repeated in the embodiments of this specification.

[0155] The solution implemented in this specification, because the target evaluation model has deep semantic understanding and multi-dimensional evaluation capabilities, can overcome the problems of high cost and low consistency of manual evaluation. Therefore, the target evaluation model can be used to achieve automatic and accurate evaluation of role-playing models, significantly improving the objectivity, efficiency and scalability of model evaluation.

[0156] See Figure 9 , Figure 9 The flowchart illustrates a request processing method based on a task processing model according to an embodiment of this specification. This method is applied to a task platform and specifically includes the following steps:

[0157] Step 902: Receive the model request sent by the terminal device.

[0158] Step 904: Based on the model request, determine the target task processing model from multiple task processing models, wherein the multiple task processing models are trained based on the task processing model training method.

[0159] It should be noted that multiple task processing models can have different model specifications and parameters, and be adapted to different scenarios. Since the task processing models are trained using task processing model training methods, they possess high stability and accuracy. The target task processing model is a task processing model suitable for the target scenario. The target scenario can be different scenarios, such as role-playing scenarios, multi-turn emotional dialogue scenarios, etc. The model request includes at least one of the following: the scenario identifier of the target scenario, the scenario input data of the target scenario, and the model specifications and parameters.

[0160] In practical applications, there are multiple ways to determine the target task processing model from multiple task processing models based on a model request. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In the first possible implementation, the target task processing model can be searched from the task processing models included in the first model library based on the scene identifier included in the model request. In the second possible implementation, the target task processing model can be trained based on the scene input data included in the model request. In the third possible implementation, the target task processing model can be searched from the task processing models included in the second model library based on the model specification parameters included in the model request.

[0161] For example, based on the scene identifier of the target scene, at least one pre-trained task processing model can be searched from the first model library. Then, based on the model specification parameters, an intermediate task processing model can be selected from the at least one task processing model. Finally, based on the scene input data of the target scene, the selected intermediate task processing model can be trained to obtain a target task processing model suitable for user needs.

[0162] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target task processing models, realize personalized model services, provide users with an efficient, flexible and easy-to-use model service method, and improve user experience.

[0163] In one optional embodiment of this specification, determining the target task processing model from multiple task processing models based on a model request may include the following steps:

[0164] If the model request includes a scene identifier for the target scene, the target task processing model suitable for the target scene is searched from the first model library based on the scene identifier. The first model library stores multiple task processing models suitable for different scenes.

[0165] When the model request includes scene input data of the target scene, the task processing model to be trained is determined from multiple task processing models that are suitable for the target scene, and the task processing model to be trained is trained based on the scene input data to obtain the target task processing model.

[0166] If the model request includes model specification parameters, the target task processing model corresponding to the model specification parameters is searched from the second model library. The second model library stores multiple task processing models with different model specification parameters.

[0167] It should be noted that scene identifiers are unique or specific labels used to distinguish different scenes. The First Model Library is a database for storing and managing various pre-trained deep learning models. Multiple task processing models adapted to different scenes cover various application scenarios and needs. The First Model Library allows users to select the appropriate task processing model according to their needs, or directly call the appropriate task processing model to execute the target task through the application programming interface. The multiple task processing models adapted to different scenes are various models stored in the First Model Library suitable for different scenarios, each optimized for a specific application environment. For example, based on the scene identifier "role-playing," one can search the First Model Library for a target task processing model suitable for the role-playing scenario.

[0168] Different task processing models are adapted to different scenarios. For example, task processing model one is suitable for scenarios one and two, while task processing model two is suitable for scenarios two and three. A task processing model to be trained refers to a model among multiple task processing models that is suitable for the target scenario, but whose performance can be further optimized. If the target scenario is scenario one, then the task processing model to be trained is task processing model one adapted to scenario one. A task processing model to be trained may not only be applicable to the target scenario but also to other scenarios, making it a general task processing model applicable to different scenarios. The task processing model to be trained can be used to execute the target task, but the results may not be very good. In this case, the task processing model to be trained can be optimized based on the scenario input data of the target scenario. For example, optimizing the task processing model to be trained based on the scenario input data of a role-playing scenario can yield a target task processing model suitable for the role-playing scenario. The scenario input data of the target scenario can be understood as the sample set (sample dialogue data and corresponding multiple sample response contents) used in the target scenario to train the task processing model.

[0169] Model specifications refer to the various parameters that define the structure and behavior of a model. These parameters can be broadly categorized into two types: model parameters (learnable parameters) and hyperparameters. Model parameters are those automatically adjusted during model training using the backpropagation algorithm, including but not limited to weight matrices and bias terms. For example, in a simple fully connected layer, the weight matrix is ​​a two-dimensional tensor connecting neurons in the input and output layers; the bias term is a one-dimensional vector providing additional offset values ​​for each output neuron. Hyperparameters are parameters set before model training begins, used to control the model's learning process and architecture. Hyperparameters include, but are not limited to, the learning rate and the number of neurons per layer, selected based on specific requirements.

[0170] The solutions implemented in this specification, based on scenario requirements, accurately identify target task processing models suitable for the corresponding scenarios through scenario identifiers, making the target task processing more accurate and more aligned with the target scenario; based on scenario requirements, further train a general task processing model to be trained using scenario input data to obtain a target task processing model adapted to the target scenario, making the target task processing model more aligned with the target scenario, thereby improving user experience and the processing quality of the target task; based on model specification parameters, the corresponding target task processing model can be accurately found, ensuring the efficient and stable operation of the target task processing model and improving user experience.

[0171] In one optional embodiment of this specification, after determining the target task processing model from multiple task processing models based on the model request, the following steps may be further included:

[0172] Deploy the target task processing model, and build a task processing interface based on the target task processing model so that the terminal device can schedule the target task processing model to execute the target task through the task processing interface.

[0173] It should be noted that the task processing interface is an interactive programming interface for terminal devices to schedule target task processing models to process target tasks, and it is usually provided in the form of an application programming interface (API). Through the task processing interface, users can input task data for the target task to be processed.

[0174] In practical applications, there are various ways to deploy the target task processing model, and the specific method should be chosen based on the actual situation. This specification does not impose any limitations on this approach. One possible implementation of this specification is to deploy the target task processing model on cloud-side devices using infrastructure provided by a cloud service provider. Another possible implementation of this specification is to deploy the target task processing model on edge devices using a lightweight framework. For example, the target task processing model can be deployed on a distributed system, and a task processing interface can be built based on the target task processing model and provided to terminal devices, enabling the terminal devices to schedule the target task processing model to execute the target task.

[0175] By applying the solutions in the embodiments of this specification, deploying the target task processing model, and constructing the task processing interface, the terminal device can efficiently call the target task processing model, thereby improving the processing quality and response speed of the target task.

[0176] See Figure 10 , Figure 10 This specification shows a schematic diagram of the structure of a task platform 1000 provided in one embodiment of the specification. The task platform 1000 includes a request interface 1002 and a response unit 1004.

[0177] Request interface 1002 is used to receive model requests sent by terminal devices, wherein the model request includes at least one of the following: scene identifier of the target scene, scene input data of the target scene, and model specification parameters.

[0178] The response unit 1004 is used to determine the target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on the task processing model training method.

[0179] In one optional embodiment of this specification, the task platform further includes a task processing interface, which is constructed based on the target task processing model.

[0180] The task processing interface is used to allow terminal devices to schedule and execute target tasks.

[0181] The above is an illustrative scheme of a task platform according to this embodiment. It should be noted that the technical solution of this task platform and the technical solution of the request processing method based on the task processing model described above belong to the same concept. For details not described in detail in the technical solution of the task platform, please refer to the description of the technical solution of the request processing method based on the task processing model described above.

[0182] Corresponding to the above-described embodiments of the task processing model training method, this specification also provides embodiments of the task processing model training apparatus. Figure 11 A schematic diagram of a task processing model training device according to one embodiment of this specification is shown. Figure 11 As shown, the device includes:

[0183] The first acquisition module 1102 is configured to acquire multiple sample response contents, which are response contents generated by the task processing model based on sample dialogue data.

[0184] The first input module 1104 is configured to input multiple sample response contents into the target evaluation model to obtain target response indicators corresponding to the multiple sample response contents. The target response indicators are used to measure the quality of the corresponding sample response contents. The target evaluation model is obtained by reinforcement learning of the cold start evaluation model based on the meta-evaluation model. The meta-evaluation model is trained based on sample dialogue data, multiple sample response contents, and sample response analysis information.

[0185] The first adjustment module 1106 is configured to adjust the parameters of the task processing model according to the target response index to obtain the trained task processing model.

[0186] Optionally, the device further includes: a second adjustment module configured to acquire sample dialogue data and multiple sample response contents; input the sample dialogue data and multiple sample response contents into a cold start evaluation model for response process analysis to obtain multiple sets of sample response analysis information, wherein any set of sample response analysis information includes multiple auxiliary response indicators and auxiliary analysis paths corresponding to the multiple auxiliary response indicators, the auxiliary response indicators correspond one-to-one with the sample response contents, and different auxiliary analysis paths correspond to different sample response analysis information; and adjust the parameters of the initial evaluation model based on the sample dialogue data, multiple sample response contents, and multiple sets of sample response analysis information to obtain a trained meta-evaluation model.

[0187] Optionally, the second adjustment module is further configured to perform quality assessment on the analysis information of multiple sets of sample responses to obtain evaluation indicators for the sample response analysis information; input the sample dialogue data, multiple sample response contents, and multiple sets of sample response analysis information into the initial evaluation model to obtain prediction indicators; and adjust the parameters of the initial evaluation model based on the prediction indicators and evaluation indicators to obtain a meta-evaluation model.

[0188] Optionally, the device further includes: a third adjustment module, configured to acquire sample dialogue data and multiple sample response contents; perform response process analysis on the multiple sample response contents to obtain sample analysis paths and sample response indicators corresponding to the multiple sample response contents respectively; and adjust the parameters of the initial evaluation model based on the sample dialogue data, multiple sample response contents, sample analysis paths and sample response indicators to obtain a trained cold start evaluation model.

[0189] Optionally, the third adjustment module is further configured to input sample dialogue data and multiple sample response contents into the initial evaluation model for response process analysis, obtain the predicted analysis path and the predicted response indicators corresponding to the multiple sample response contents respectively; calculate the path prediction error based on the sample analysis path and the predicted analysis path, and calculate the indicator prediction error based on the sample response indicators and the predicted response indicators; adjust the parameters of the initial evaluation model based on the path prediction error and the indicator prediction error to obtain the trained cold start evaluation model.

[0190] Optionally, the device further includes: a fourth adjustment module configured to acquire sample dialogue data and multiple sample response contents; input the sample dialogue data and multiple sample response contents into a cold start evaluation model to obtain cold start response indicators corresponding to the multiple sample response contents respectively; input the cold start response indicators into a meta-evaluation model to obtain meta-evaluation indicators; and adjust the parameters of the cold start evaluation model according to the meta-evaluation indicators to obtain the trained target evaluation model.

[0191] The scheme implemented in this specification, using a meta-evaluation model to guide reinforcement learning training to obtain a target evaluation model, improves the accuracy and robustness of the target evaluation model in judging the quality of sample responses. The target response index output by the target evaluation model can provide richer feedback signals to the task processing model, and can more precisely guide the optimization direction of the task processing model, thereby improving the training efficiency and alignment performance of the task processing model.

[0192] The above is a schematic scheme of a task processing model training device according to this embodiment. It should be noted that the technical solution of this task processing model training device and the technical solution of the task processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the task processing model training device, please refer to the description of the technical solution of the task processing model training method described above.

[0193] Corresponding to the above-described embodiments of the role-playing model training method, this specification also provides embodiments of the role-playing model training device. Figure 12 A schematic diagram of a role-playing model training device according to one embodiment of this specification is shown. Figure 12 As shown, the device includes:

[0194] The second acquisition module 1202 is configured to acquire multiple sample response contents, which are response contents generated by the role-playing model based on sample dialogue data.

[0195] The second input module 1204 is configured to input multiple sample response contents into the target evaluation model to obtain target response indicators corresponding to the multiple sample response contents. The target response indicators are used to measure the quality of the corresponding sample response contents. The target evaluation model is obtained by reinforcement learning of the cold start evaluation model based on the meta-evaluation model. The meta-evaluation model is trained based on sample dialogue data, multiple sample response contents, and sample response analysis information.

[0196] The fifth adjustment module 1206 is configured to adjust the parameters of the role-playing model according to the target response index to obtain the trained role-playing model.

[0197] The scheme implemented in this specification uses a meta-evaluation model to guide reinforcement learning training to obtain a target evaluation model, thereby improving the accuracy and robustness of the target evaluation model in judging the quality of sample responses. By providing richer feedback signals to the role-playing model through target response metrics, the optimization direction of the role-playing model can be guided more precisely, thus improving the training efficiency and alignment performance of the role-playing model.

[0198] The above is an illustrative scheme of a role-playing model training device according to this embodiment. It should be noted that the technical solution of this role-playing model training device and the technical solution of the role-playing model training method described above belong to the same concept. For details not described in detail in the technical solution of the role-playing model training device, please refer to the description of the technical solution of the role-playing model training method described above.

[0199] Corresponding to the above-described task processing method embodiments, this specification also provides embodiments of a task processing apparatus. Figure 13A schematic diagram of a task processing apparatus according to one embodiment of this specification is shown. Figure 13 As shown, the device includes:

[0200] The third acquisition module 1302 is configured to acquire task data of the target task;

[0201] The third input module 1304 is configured to input task data into the task processing model to obtain the task processing result of the target task, wherein the task processing model is trained based on the task processing model training method.

[0202] By applying the solutions in the embodiments of this specification, task processing results that are highly accurate and meet user preferences can be obtained by inputting the task data of the target task into a task processing model that has been trained by the task processing model training method.

[0203] The above is an illustrative scheme of a task processing device according to this embodiment. It should be noted that the technical solution of this task processing device and the technical solution of the task processing method described above belong to the same concept. For details not described in detail in the technical solution of the task processing device, please refer to the description of the technical solution of the task processing method described above.

[0204] Corresponding to the above-described embodiments of the target evaluation model training method, this specification also provides embodiments of the target evaluation model training apparatus. Figure 14 A schematic diagram of a target evaluation model training device according to one embodiment of this specification is shown. Figure 14 As shown, the device includes:

[0205] The fourth acquisition module 1402 is configured to acquire the cold start evaluation model and the meta-evaluation model. The meta-evaluation model is trained based on sample dialogue data, multiple sample response contents, and sample response analysis information. The multiple sample response contents are response contents generated by the task processing model based on the sample dialogue data.

[0206] Learning module 1404 is configured to perform reinforcement learning on the cold start evaluation model based on the meta-evaluation model to obtain the target evaluation model, wherein the target evaluation model is used in the processing of the task processing model training method.

[0207] By applying the scheme of the embodiments in this specification, the cold start evaluation model is reinforced by the meta-evaluation model to obtain a target evaluation model that can generate more reliable target evaluation indicators, thereby improving the evaluation accuracy and robustness of the target evaluation model and reducing the reliance on manual annotation.

[0208] The above is a schematic scheme of a target evaluation model training device according to this embodiment. It should be noted that the technical solution of this target evaluation model training device and the technical solution of the target evaluation model training method described above belong to the same concept. For details not described in detail in the technical solution of the target evaluation model training device, please refer to the description of the technical solution of the target evaluation model training method described above.

[0209] Corresponding to the above-described model evaluation method embodiments, this specification also provides embodiments of a model evaluation device. Figure 15 A schematic diagram of a model evaluation device provided in one embodiment of this specification is shown. Figure 15 As shown, the device includes:

[0210] The fifth acquisition module 1502 is configured to acquire the evaluation data of the model to be evaluated. The evaluation data includes evaluation task data and evaluation task results. The evaluation task results are the results generated by the model to be evaluated based on the evaluation task data.

[0211] The fourth input module 1504 is configured to input the evaluation task data and evaluation task results into the target evaluation model to obtain the evaluation results of the model to be evaluated, wherein the target evaluation model is trained based on the target evaluation model training method.

[0212] The solution implemented in this specification, because the target evaluation model has deep semantic understanding and multi-dimensional evaluation capabilities, can overcome the problems of high cost and low consistency of manual evaluation. Therefore, the target evaluation model can be used to achieve automatic and accurate evaluation of the model to be evaluated, which significantly improves the objectivity, efficiency and scalability of model evaluation.

[0213] The above is a schematic scheme of a model evaluation device according to this embodiment. It should be noted that the technical solution of this model evaluation device and the technical solution of the model evaluation method described above belong to the same concept. For details not described in detail in the technical solution of the model evaluation device, please refer to the description of the technical solution of the model evaluation method described above.

[0214] Corresponding to the above-described embodiments of the role-playing model evaluation method, this specification also provides embodiments of the role-playing model evaluation device. Figure 16 A schematic diagram of a role-playing model evaluation device according to one embodiment of this specification is shown. Figure 16 As shown, the device includes:

[0215] The sixth acquisition module 1602 is configured to respond to a model evaluation request sent by the client for the role-playing model and acquire the evaluation data of the role-playing model. The evaluation data includes evaluation task data and evaluation task results, and the evaluation task results are the results generated by the role-playing model based on the evaluation task data.

[0216] The fifth input module 1604 is configured to input the evaluation task data and evaluation task results into the target evaluation model, obtain the evaluation results of the role-playing model, and send the evaluation results to the client. The target evaluation model is trained based on the target evaluation model training method.

[0217] The solution implemented in this specification, because the target evaluation model has deep semantic understanding and multi-dimensional evaluation capabilities, can overcome the problems of high cost and low consistency of manual evaluation. Therefore, the target evaluation model can be used to achieve automatic and accurate evaluation of role-playing models, significantly improving the objectivity, efficiency and scalability of model evaluation.

[0218] The above is a schematic scheme of a role-playing model evaluation device according to this embodiment. It should be noted that the technical solution of this role-playing model evaluation device and the technical solution of the role-playing model evaluation method described above belong to the same concept. For details not described in detail in the technical solution of the role-playing model evaluation device, please refer to the description of the technical solution of the role-playing model evaluation method described above.

[0219] Corresponding to the above-described embodiments of the request processing method based on the task processing model, this specification also provides embodiments of the request processing apparatus based on the task processing model. Figure 17 This specification illustrates a schematic diagram of a request processing apparatus based on a task processing model, according to one embodiment of the present specification. Figure 17 As shown, the device is applied to a mission platform and includes:

[0220] The receiving module 1702 is configured to receive model requests sent by the terminal device;

[0221] The determination module 1704 is configured to determine the target task processing model from multiple task processing models based on a model request, wherein the multiple task processing models are trained based on a task processing model training method.

[0222] Optionally, the determining module 1704 is further configured to: when the model request includes a scene identifier of the target scene, search for a target task processing model adapted to the target scene from a first model library based on the scene identifier, wherein the first model library stores multiple task processing models adapted to different scenes; when the model request includes scene input data of the target scene, determine a task processing model to be trained adapted to the target scene from multiple task processing models, and train the task processing model to be trained based on the scene input data to obtain the target task processing model; when the model request includes model specification parameters, search for a target task processing model corresponding to the model specification parameters from a second model library, wherein the second model library stores multiple task processing models with different model specification parameters.

[0223] Optionally, the device further includes: a deployment module configured to deploy a target task processing model and, based on the target task processing model, construct a task processing interface so that the terminal device can schedule the target task processing model to execute the target task through the task processing interface.

[0224] The solutions implemented in the embodiments of this specification are adapted to user needs to obtain target task processing models, realize personalized model services, provide users with an efficient, flexible and easy-to-use model service method, and improve user experience.

[0225] The above is an illustrative scheme of a request processing device based on a task processing model according to this embodiment. It should be noted that the technical solution of this request processing device based on a task processing model and the technical solution of the request processing method based on a task processing model described above belong to the same concept. For details not described in detail in the technical solution of the request processing device based on a task processing model, please refer to the description of the technical solution of the request processing method based on a task processing model described above.

[0226] Figure 18 A structural block diagram of a computing device 1800 according to an embodiment of this specification is shown. The computing device 1800 includes a memory 1810 and a processor 1820; the memory 1810 stores computer programs / instructions, and the processor 1820 executes the computer programs / instructions. When executed by the processor 1820, the computer programs / instructions implement the steps of the aforementioned task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, or request processing method based on a task processing model.

[0227] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, personal computer, all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0228] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing application programming interface (API) invocation capabilities. Models can be invoked into created applications through the API interface, and application management tools are provided for application management and monitoring.

[0229] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master artificial intelligence technologies), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for artificial intelligence development, training, deployment, and application.

[0230] Figure 19 A structural block diagram of an electronic device 1900 provided according to one embodiment of this specification is shown.

[0231] The memory 1910 and the processor 1920 are connected via a bus 1930;

[0232] The memory 1910 is used to store computer programs / instructions, and the processor 1920 is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor 1920, they implement the steps of the above-mentioned task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, or request processing method based on the task processing model.

[0233] Specifically, the components of the electronic device 1900 include, but are not limited to, a memory 1910 and a processor 1920. The processor 1920 and the memory 1910 can be connected via a bus 1930.

[0234] Electronic device 1900 may also include access device 1940, which enables electronic device 1900 to communicate with database 1950 storing data via one or more networks 1960. Examples of such networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 1940 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as IEEE 802.11 Wireless Local Area Networks (WLAN) wireless interface, Wi-MAX (World Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0235] In one embodiment of this specification, the above-described components of the electronic device 1900 and Figure 19 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 19 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0236] Electronic device 1900 can be any type of stationary or mobile electronic device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or PCs. Electronic device 1900 can also be a mobile or stationary electronic device.

[0237] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the technical solutions of the aforementioned task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, and request processing method based on task processing model. Details not described in detail in the technical solution of the electronic device can be found in the descriptions of the aforementioned technical solutions of the task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, or request processing method based on task processing model.

[0238] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, or request processing method based on a task processing model.

[0239] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solutions of the aforementioned task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, and request processing method based on the task processing model. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the aforementioned task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, or request processing method based on the task processing model.

[0240] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, or request processing method based on a task processing model.

[0241] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the aforementioned task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, and request processing method based on task processing model. Details not described in detail in the technical solution of the computer program product can be found in the descriptions of the aforementioned technical solutions of the task processing model training method, role-playing model training method, task processing method, target evaluation model training method, model evaluation method, role-playing model evaluation method, or request processing method based on task processing model.

[0242] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0243] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0244] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0245] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0246] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a task processing model, comprising: Multiple sample response contents are obtained. The multiple sample response contents are response contents generated by the task processing model based on sample dialogue data. The sample dialogue data includes at least one of text, voice, video, and image. The multiple sample responses are input into a target evaluation model to obtain target response indicators corresponding to each sample response. These target response indicators measure the quality of the corresponding sample responses. The target evaluation model is obtained by reinforcement learning of a cold-start evaluation model based on a meta-evaluation model. The meta-evaluation model is trained based on the sample dialogue data, the multiple sample responses, and sample response analysis information. The sample response analysis information is obtained by the cold-start evaluation model through response process analysis of the sample dialogue data and the multiple sample responses. The cold-start evaluation model is trained based on the sample dialogue data, the multiple sample responses, the sample analysis path, and the sample response indicators. The sample analysis path and the sample response indicators are obtained through response process analysis of the multiple sample responses. Based on the target response metric, the parameters of the task processing model are adjusted to obtain a trained task processing model.

2. The method according to claim 1, before inputting the plurality of sample response contents into the target evaluation model to obtain the target response indicators corresponding to the plurality of sample response contents respectively, further includes: Obtain the sample dialogue data and the content of the multiple sample responses; The sample dialogue data and the multiple sample response contents are input into the cold start evaluation model for response process analysis to obtain multiple sets of sample response analysis information. Each set of sample response analysis information includes multiple auxiliary response indicators and auxiliary analysis paths corresponding to the multiple auxiliary response indicators. The auxiliary response indicators correspond one-to-one with the sample response content, and the auxiliary analysis paths corresponding to different sample response analysis information are different. Based on the sample dialogue data, the content of the multiple sample responses, and the analysis information of the multiple sets of sample responses, the parameters of the initial evaluation model are adjusted to obtain the trained meta-evaluation model.

3. The method according to claim 2, wherein adjusting the parameters of the initial evaluation model based on the sample dialogue data, the content of the multiple sample responses, and the analysis information of the multiple sets of sample responses to obtain the trained meta-evaluation model includes: The quality of the multiple sets of sample response analysis information is evaluated to obtain the evaluation index of the sample response analysis information; The sample dialogue data, the content of the multiple sample responses, and the analysis information of the multiple sets of sample responses are input into the initial evaluation model to obtain the prediction index. Based on the prediction indicators and the evaluation indicators, the parameters of the initial evaluation model are adjusted to obtain the meta-evaluation model.

4. The method according to claim 1, before inputting the plurality of sample response contents into the target evaluation model to obtain the target response indicators corresponding to the plurality of sample response contents respectively, further includes: Obtain the sample dialogue data and the content of the multiple sample responses; The response process of the multiple sample responses is analyzed to obtain the sample analysis path and the sample response indicators corresponding to the multiple sample responses respectively. Based on the sample dialogue data, the content of the multiple sample responses, the sample analysis path, and the sample response indicators, the parameters of the initial evaluation model are adjusted to obtain the trained cold start evaluation model.

5. The method according to claim 4, wherein adjusting the parameters of the initial evaluation model based on the sample dialogue data, the content of the plurality of sample responses, the sample analysis path, and the sample response indicators to obtain the trained cold start evaluation model comprises: The sample dialogue data and the multiple sample responses are input into the initial evaluation model to analyze the response process and obtain the predictive analysis path and the predictive response indicators corresponding to the multiple sample responses respectively. Based on the sample analysis path and the predictive analysis path, calculate the path prediction error, and based on the sample response index and the predictive response index, calculate the index prediction error. Based on the path prediction error and the index prediction error, the parameters of the initial evaluation model are adjusted to obtain the trained cold start evaluation model.

6. The method according to any one of claims 1 to 5, wherein before inputting the plurality of sample response contents into the target evaluation model to obtain the target response indicators corresponding to the plurality of sample response contents respectively, the method further includes: Obtain the sample dialogue data and the content of the multiple sample responses; The sample dialogue data and the multiple sample response contents are input into the cold start evaluation model to obtain the cold start response index corresponding to the multiple sample response contents respectively. The cold start response index is input into the meta-evaluation model to obtain the meta-evaluation index; Based on the meta-evaluation index, the parameters of the cold start evaluation model are adjusted to obtain the trained target evaluation model.

7. A method for training a role-playing model, comprising: Obtain multiple sample response contents, which are response contents generated by the role-playing model based on sample dialogue data, and the sample dialogue data includes at least one of text, voice, video, and image; The multiple sample responses are input into a target evaluation model to obtain target response indicators corresponding to each sample response. These target response indicators measure the quality of the corresponding sample responses. The target evaluation model is obtained by reinforcement learning of a cold-start evaluation model based on a meta-evaluation model. The meta-evaluation model is trained based on the sample dialogue data, the multiple sample responses, and sample response analysis information. The sample response analysis information is obtained by the cold-start evaluation model through response process analysis of the sample dialogue data and the multiple sample responses. The cold-start evaluation model is trained based on the sample dialogue data, the multiple sample responses, the sample analysis path, and the sample response indicators. The sample analysis path and the sample response indicators are obtained through response process analysis of the multiple sample responses. Based on the target response metric, the parameters of the role-playing model are adjusted to obtain a trained role-playing model.

8. A task processing method, comprising: Obtain the task data for the target task; The task data is input into the task processing model to obtain the task processing result of the target task, wherein the task processing model is trained based on the method described in any one of claims 1 to 6.

9. A method for training a target evaluation model, comprising: A cold start evaluation model and a meta-evaluation model are obtained. The meta-evaluation model is trained based on sample dialogue data, multiple sample response contents, and sample response analysis information. The multiple sample response contents are response contents generated by the task processing model based on the sample dialogue data. The sample dialogue data includes at least one of text, voice, video, and image. The sample response analysis information is obtained by the cold start evaluation model through response process analysis of the sample dialogue data and the multiple sample response contents. The cold start evaluation model is trained based on the sample dialogue data, the multiple sample response contents, the sample analysis path, and the sample response indicators. The sample analysis path and the sample response indicators are obtained through response process analysis of the multiple sample response contents. Based on the meta-evaluation model, reinforcement learning is performed on the cold start evaluation model to obtain the target evaluation model, wherein the target evaluation model is used in the processing of the method as described in any one of claims 1 to 6.

10. A model evaluation method, comprising: Obtain evaluation data for the model to be evaluated, wherein the evaluation data includes evaluation task data and evaluation task results, and the evaluation task results are the results generated by the model to be evaluated based on the evaluation task data; The evaluation task data and the evaluation task results are input into the target evaluation model to obtain the evaluation result of the model to be evaluated, wherein the target evaluation model is trained based on the method described in claim 9.

11. A method for evaluating role-playing models, comprising: In response to a model evaluation request sent by a client for a role-playing model, the evaluation data of the role-playing model is obtained, wherein the evaluation data includes evaluation task data and evaluation task results, and the evaluation task results are the results generated by the role-playing model based on the evaluation task data; The evaluation task data and the evaluation task results are input into the target evaluation model to obtain the evaluation result of the role-playing model, and the evaluation result is sent to the client, wherein the target evaluation model is trained based on the method described in claim 9.

12. A request processing method based on a task processing model, applied to a task platform, comprising: Receive model requests sent by terminal devices; Based on the model request, a target task processing model is determined from a plurality of task processing models, wherein the plurality of task processing models are trained based on the method described in any one of claims 1 to 6.

13. The method according to claim 12, wherein determining the target task processing model from multiple task processing models based on the model request includes: If the model request includes a scene identifier of the target scene, a target task processing model suitable for the target scene is searched from the first model library based on the scene identifier. The first model library stores multiple task processing models suitable for different scenes. When the model request includes scene input data of the target scene, a task processing model to be trained that is suitable for the target scene is determined from the plurality of task processing models, and the task processing model to be trained is trained based on the scene input data to obtain the target task processing model. If the model request includes model specification parameters, the target task processing model corresponding to the model specification parameters is searched from the second model library, wherein the second model library stores multiple task processing models with different model specification parameters.

14. The method according to claim 13, further comprising, after determining the target task processing model from multiple task processing models based on the model request: Deploy the target task processing model, and based on the target task processing model, construct a task processing interface so that the terminal device can schedule the target task processing model to execute the target task through the task processing interface.

15. A task platform, comprising a request interface and a response unit; The request interface is used to receive model requests sent by the terminal device, wherein... The model request includes at least one of the following: the scene identifier of the target scene, the scene input data of the target scene, and the model specification parameters. The response unit is configured to determine a target task processing model from multiple task processing models based on the model request, wherein the multiple task processing models are trained based on the method described in any one of claims 1 to 6.

16. The task platform according to claim 15, further comprising a task processing interface, wherein the task processing interface is constructed based on the target task processing model; The task processing interface is used for the terminal device to schedule and execute target tasks.

17. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 14.

18. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 14.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 14.

Images