Training method and device of verbal skill selection model and related equipment

Abstract

The invention provides a training method and device for a verbal skill selection model and related equipment. The method comprises the steps that a training data set is acquired; performing model fine tuning on a pre-trained first verbal skill selection model based on the training data set to obtain a second verbal skill selection model; and based on a reward model and the training data set, performing reinforcement learning on the second verbal skill selection model to obtain a third verbal skill selection model. According to the method, the training data set comprising the multiple dialogue texts is acquired, so that model fine tuning is performed on the pre-trained verbal skill selection model based on the training data set, and reinforcement learning is performed on the verbal skill selection model based on the reward model and the training data set; the verbal skill answer generation quality and efficiency of the verbal skill selection model are improved through two aspects of model fine tuning and reinforcement learning.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a training method, apparatus and related equipment for a speech selection model. Background Technology

[0002] With the rapid development of large-scale modeling technology, it is being used in an increasing number of fields. For example, in the field of intelligent customer service, large-scale modeling technology can accurately determine user needs and generate corresponding scripted responses. However, in existing technologies, a common approach is to directly combine script selection models with Retrieval-Augmented Generation (RAG). This approach integrates retrieval mechanisms and generative models, aiming to enhance the output of the generative model by retrieving relevant knowledge from a knowledge base. However, the quality of this approach is highly dependent on the quality of the retrieved documents and the performance of the generative model. Furthermore, in real-time customer service scenarios, the retrieval and generation processes of the RAG system can lead to long response times. Therefore, existing technologies suffer from poor quality and efficiency in generating scripted responses using script selection models. Summary of the Invention

[0003] This application provides a training method, apparatus, and related equipment for a dialogue selection model to solve the problem of poor quality and efficiency in the dialogue responses generated by existing dialogue selection models.

[0004] To solve the above problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a training method for a speech selection model, the method comprising:

[0006] Obtain a training dataset, which includes multiple dialogue texts, each of which includes an input question and a corresponding verbal response to the input question;

[0007] Based on the training dataset, the pre-trained first script selection model is fine-tuned to obtain the second script selection model.

[0008] Based on the reward model and the training dataset, reinforcement learning is performed on the second dialogue selection model to obtain a third dialogue selection model. The third dialogue selection model is used to match the target question input by the user and output a dialogue answer that matches the target question.

[0009] Optionally, before obtaining the training dataset, the method further includes:

[0010] Acquire multiple historical dialogue texts, each of which includes a historical input question and a historical response to the historical input question;

[0011] The multiple historical dialogue texts are vectorized to obtain multiple feature vectors;

[0012] Multiple clusters are generated by performing multiple clustering on the multiple feature vectors. Each cluster includes at least two feature vectors. The similarity between at least two historical input questions corresponding to the at least two feature vectors included in each cluster meets a preset condition.

[0013] Based on the multiple clusters, the multiple dialogue texts are determined from the multiple historical dialogue texts;

[0014] The training dataset is generated based on the multiple dialogue texts.

[0015] Optionally, determining the plurality of dialogue texts from the plurality of historical dialogue texts based on the plurality of clusters includes:

[0016] Each cluster is assigned an intent reversal score and an emotion reversal score. The intent reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's intent, and the emotion reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's emotion.

[0017] For a target cluster, if the intention reversal score corresponding to the target cluster is greater than a first preset score value and the emotion reversal score corresponding to the target cluster is greater than a second preset score value, then at least two historical dialogue texts corresponding to the target cluster are determined as the dialogue texts, and the target cluster is any one of the plurality of clusters.

[0018] Optionally, the step of fine-tuning the pre-trained first dialogue selection model based on the training dataset to obtain the second dialogue selection model includes:

[0019] The multiple dialogue texts are classified to determine a positive sample set and a negative sample set. The positive sample set includes multiple positive sample dialogue texts, and the negative sample set includes multiple negative sample dialogue texts. The ratio of the multiple positive sample dialogue texts to the multiple negative sample dialogue texts is a preset ratio.

[0020] Obtain a prompt word template, the prompt word template includes an output format and output content, the output format is a text format, and the output content is a verbal response;

[0021] Based on the positive sample set, the negative sample set, and the prompt word template, the pre-trained first dialogue selection model is fine-tuned to obtain the second dialogue selection model.

[0022] Optionally, before performing reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain the third dialogue selection model, the method further includes:

[0023] The system sets up a first reward rule, a second reward rule, and a third reward rule. The first reward rule includes a reward value of x+1 when the matching degree between the output of the third dialogue selection model and the user-input target question is greater than or equal to a first preset value; a reward value of xα when the matching degree between the output of the third dialogue selection model and the user-input target question is less than the first preset value but greater than the second preset value; and a reward value of -1 when the matching degree between the output of the third dialogue selection model and the user-input target question is less than or equal to the second preset value. α∈(0,1).

[0024] The reward model is generated based on the first reward rule, the second reward rule, and the third reward rule.

[0025] Optionally, the step of performing reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain the third dialogue selection model includes:

[0026] The reward model is calculated based on the near-end policy optimization algorithm to obtain the advantage function;

[0027] Based on the advantage function and the training dataset, reinforcement learning is performed on the second dialogue selection model to obtain the third dialogue selection model.

[0028] Secondly, embodiments of this application provide a method for selecting dialogue scripts, the method comprising:

[0029] Obtain the target problem as input;

[0030] The target question is input into the third dialogue selection model described in any of the above embodiments to match the target question, and the dialogue answer that matches the target question is output.

[0031] Thirdly, embodiments of this application also provide a training apparatus for a speech selection model, comprising:

[0032] The first acquisition module is used to acquire a training dataset, which includes multiple dialogue texts, each of which includes an input question and a corresponding verbal response to the input question.

[0033] The first training module is used to fine-tune the pre-trained first dialogue selection model based on the training dataset to obtain the second dialogue selection model.

[0034] The second training module is used to perform reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain a third dialogue selection model. The third dialogue selection model is used to match the target question input by the user and output a dialogue answer that matches the target question.

[0035] Fourthly, embodiments of this application also provide a speech selection device, comprising:

[0036] The second acquisition module is used to acquire the target problem as input.

[0037] The matching module is used to input the target question into the third dialogue selection model described in any one of the first aspects to match the target question and output a dialogue response that matches the target question.

[0038] Fifthly, embodiments of this application also provide an electronic device, including: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is configured to read the program in the memory to implement the steps in the method described in the first aspect above, or to implement the steps in the method described in the second aspect above.

[0039] In a sixth aspect, embodiments of this application also provide a readable storage medium for storing a program, which, when executed by a processor, implements the steps of the method described in the first aspect above, or implements the steps of the method described in the second aspect above.

[0040] In a seventh aspect, embodiments of this application also provide a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps in the method described in the first aspect, or to implement the steps in the method described in the second aspect.

[0041] This application provides a training method, apparatus, and related equipment for a dialogue selection model. The method includes: acquiring a training dataset comprising multiple dialogue texts, each dialogue text including an input question and a corresponding dialogue response; fine-tuning a pre-trained first dialogue selection model based on the training dataset to obtain a second dialogue selection model; and performing reinforcement learning on the second dialogue selection model based on a reward model and the training dataset to obtain a third dialogue selection model. The third dialogue selection model is used to match a target question input by a user and output a dialogue response that matches the target question. This application improves the quality and efficiency of the dialogue responses generated by the dialogue selection model by acquiring a training dataset comprising multiple dialogue texts, fine-tuning the pre-trained dialogue selection model based on the training dataset, and performing reinforcement learning on the dialogue selection model based on a reward model and the training dataset. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the training method of the speech selection model provided in this application embodiment;

[0044] Figure 2 This is a flowchart illustrating the generation of the training dataset provided in the embodiments of this application;

[0045] Figure 3 This is a schematic diagram of the training process of the speech selection model provided in the embodiments of this application;

[0046] Figure 4 This is a flowchart illustrating the method for selecting delivery scripts provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the structure of the training device for the speech selection model provided in the embodiments of this application;

[0048] Figure 6 This is a schematic diagram of the device for selecting speech according to an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.

[0052] See Figure 1 , Figure 1 This is a flowchart illustrating the training method of the discourse selection model provided in this application embodiment. Figure 1 The training method for the speech selection model shown can be executed on the server side.

[0053] like Figure 1 As shown, the training method for the dialogue selection model may include the following steps:

[0054] Step 101: Obtain the training dataset, which includes multiple dialogue texts, each of which includes an input question and a corresponding verbal response.

[0055] In this embodiment, the script selection model is an artificial intelligence-based intelligent system that automatically selects the most suitable script to reply to customers by analyzing their historical dialogue records, preset dialogue strategies, and a high-quality script library, thereby improving the efficiency of intelligent customer service and customer satisfaction.

[0056] In this embodiment, the dialogue selection model can be a large language model, which is a deep learning model with massive parameters that can process and generate natural language text, providing powerful language understanding and generation capabilities for intelligent customer service.

[0057] The training dataset includes multiple dialogue texts, which contain input questions entered by the user on the server and dialogue responses generated based on a dialogue selection model. For example, the input question could be "Please check your phone bill balance," and the dialogue response could be "Hello, we have found that your phone bill balance is XX yuan."

[0058] It should be noted that the dialogue texts selected in this embodiment are generally high-quality dialogue texts, meaning their wording and responses can meet the user's input question requirements. Therefore, training the dialogue selection model using a high-quality training dataset can improve training accuracy.

[0059] Step 102: Based on the training dataset, fine-tune the pre-trained first dialogue selection model to obtain the second dialogue selection model.

[0060] In this embodiment, model fine-tuning (SFT) refers to targeted training of a pre-trained large language model to better adapt it to the specific tasks and domains of intelligent customer service.

[0061] It should be noted that in this embodiment, the pre-trained large language model is first fine-tuned to obtain the second dialogue selection model. The general-purpose large language model already possesses strong contextual understanding and a certain level of logical reasoning ability; therefore, even without specific fine-tuning, these models perform well in handling simple customer questions, such as the Qwen2.5-72B model. In the optimization scheme, LORA (Low-Rank Adaptation) technology can also be used to fine-tune the Qwen2.5-7B model to enhance its ability to follow dialogue strategies. This approach not only ensures the accuracy of the dialogue selection model but also improves its reasoning efficiency, giving it a significant advantage over the Qwen2.5-72B model.

[0062] During model training, historical customer service response data was used as labels to fine-tune the large language model. Simultaneously, to train the model's ability to match historical dialogue records with dialogue strategies, both realistic response strategies and obfuscated response strategies were provided. Furthermore, to enable the model to possess strong rejection capabilities—that is, to refuse to answer unreasonable customer questions or questions that cannot be answered by the script database—so that the system can transfer the call to a human operator, corresponding rejection data was manually constructed based on the script selection model's capability definition and added to the model fine-tuning dataset.

[0063] Step 103: Based on the reward model and the training dataset, perform reinforcement learning on the second dialogue selection model to obtain a third dialogue selection model. The third dialogue selection model is used to match the target question input by the user and output a dialogue answer that matches the target question.

[0064] In this embodiment, reinforcement learning (RL) is a category of machine learning that involves an agent learning how to make decisions in a specific task through interaction with its environment. Unlike supervised and unsupervised learning, the primary goal of reinforcement learning is to maximize cumulative reward through trial and error. In reinforcement learning, the agent interacts with the environment at each time step, observing the current state, selecting an action, and receiving feedback from the environment (i.e., reward and the next state). The agent's goal is to learn a policy that selects the optimal action in each state to maximize future cumulative reward.

[0065] During model training, a pre-defined reward model is used to reinforce the second dialogue selection model, resulting in a trained third dialogue selection model. The reward model evaluates the effectiveness of an agent's action in a specific state. It provides feedback to the agent, indicating whether the action is effective or to what extent, thus helping the agent learn how to optimize its strategy to maximize cumulative rewards.

[0066] Therefore, by using an adaptive reinforcement learning method for customer service scripts based on a semantic similarity reward mechanism, the trained large model has good script selection and rejection capabilities. Constructing a reward function by measuring semantic similarity can avoid the need for manual labeling, and the accuracy and efficiency of the script selection model output can be significantly improved.

[0067] This application improves the quality and efficiency of the dialogue responses generated by the dialogue selection model by obtaining a training dataset containing multiple dialogue texts, fine-tuning the pre-trained dialogue selection model based on the training dataset, and performing reinforcement learning on the dialogue selection model based on the reward model and the training dataset.

[0068] In some feasible implementations, optionally, before obtaining the training dataset, the method further includes:

[0069] Acquire multiple historical dialogue texts, each of which includes a historical input question and a historical response to the historical input question;

[0070] The multiple historical dialogue texts are vectorized to obtain multiple feature vectors;

[0071] Multiple clusters are generated by performing multiple clustering on the multiple feature vectors. Each cluster includes at least two feature vectors. The similarity between at least two historical input questions corresponding to the at least two feature vectors included in each cluster meets a preset condition.

[0072] Based on the multiple clusters, the multiple dialogue texts are determined from the multiple historical dialogue texts;

[0073] The training dataset is generated based on the multiple dialogue texts.

[0074] In this embodiment, as Figure 2 As shown, Figure 2 This flowchart illustrates the generation of the training dataset in this embodiment. In acquiring the training dataset, it's crucial to identify high-quality datasets. Specifically, this involves first acquiring multiple historical dialogue texts, such as through in-depth analysis of massive amounts of historical customer service dialogue records. These records cover various customer questions and customer service responses, providing valuable resources for extracting effective response strategies. By employing advanced natural language processing techniques, efficient and practical customer service response strategies can be automatically identified and extracted. These strategies often quickly resolve customer issues and improve customer satisfaction.

[0075] Specifically, after obtaining multiple historical dialogue texts, Qwen3-Embedding is used to transform the multiple historical dialogue texts into vectors, generating multiple feature vectors. These multiple feature vectors take into account both semantic depth and long text modeling capabilities, providing a highly discriminative feature space for subsequent clustering.

[0076] After generating multiple feature vectors, multiple rounds of HDBSCAN adaptive clustering are performed on these feature vectors to divide customer questions into clusters with highly consistent intents. Each cluster includes at least two feature vectors, and the similarity between at least two historical input questions corresponding to the at least two feature vectors in each cluster meets a preset condition. That is, the historical input questions corresponding to the feature vectors included in each cluster are similar. It should be noted that in this embodiment, there is no need to preset the number of clusters; they can be automatically merged or split as the business evolves, ensuring real-time accuracy of intent boundaries.

[0077] Multiple clusters are filtered to identify multiple dialogue texts from multiple historical dialogue texts, and these multiple dialogue texts are used to form a training dataset.

[0078] Optionally, determining the plurality of dialogue texts from the plurality of historical dialogue texts based on the plurality of clusters includes:

[0079] Each cluster is assigned an intent reversal score and an emotion reversal score. The intent reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's intent, and the emotion reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's emotion.

[0080] For a target cluster, if the intention reversal score corresponding to the target cluster is greater than a first preset score value and the emotion reversal score corresponding to the target cluster is greater than a second preset score value, then at least two historical dialogue texts corresponding to the target cluster are determined as the dialogue texts, and the target cluster is any one of the plurality of clusters.

[0081] In this embodiment, an intent reversal score and an emotion reversal score are first established. The intent reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to a cluster on the user's intent, while the emotion reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to a cluster on the user's emotion. Specifically, the intent reversal dimension includes a shift in the intent cluster to which the customer's question belongs before and after the customer service script, indicating that the script's guidance was successful and it is marked as high-quality. The emotion reversal dimension includes introducing a fine-grained emotion recognition model to monitor a significant shift in the customer's emotion from negative to positive before and after the customer service script, serving as a supplementary verification to the intent dimension, providing double verification and reducing false labeling.

[0082] Specifically, by calculating the intent reversal score and emotion reversal score corresponding to each cluster, the target cluster is determined by filtering among multiple clusters based on the intent reversal score and emotion reversal score. The target cluster is a cluster among multiple clusters whose intent reversal score is greater than a first preset score value and whose emotion reversal score is greater than a second preset score value. Thus, the multiple historical dialogue texts corresponding to the target cluster are identified as multiple dialogue texts, thereby completing the selection of a high-quality training dataset.

[0083] The carefully selected dialogue strategies and high-quality scripts not only provide a solid data foundation for the subsequent training of the script selection model, but also offer rich material for prompt writing. Through this high-quality data, the script selection model can better learn how to generate accurate and user-relevant responses. This not only improves the overall performance of the intelligent customer service system, but also ensures that the system can provide more accurate and targeted answers to various customer inquiries.

[0084] In this embodiment, a method for automatically judging the quality of customer service scripts based on intent cluster transition and emotion reversal is used to extract high-quality scripts from massive historical dialogue data.

[0085] Optionally, the step of fine-tuning the pre-trained first dialogue selection model based on the training dataset to obtain the second dialogue selection model includes:

[0086] The multiple dialogue texts are classified to determine a positive sample set and a negative sample set. The positive sample set includes multiple positive sample dialogue texts, and the negative sample set includes multiple negative sample dialogue texts. The ratio of the multiple positive sample dialogue texts to the multiple negative sample dialogue texts is a preset ratio.

[0087] Obtain a prompt word template, the prompt word template includes an output format and output content, the output format is a text format, and the output content is a verbal response;

[0088] Based on the positive sample set, the negative sample set, and the prompt word template, the pre-trained first dialogue selection model is fine-tuned to obtain the second dialogue selection model.

[0089] In this embodiment, as Figure 3 As shown, Figure 3 This diagram illustrates the training process of the dialogue selection model in this embodiment. During the fine-tuning phase, the large language model is fine-tuned using the training dataset. Simultaneously, to train the model's ability to match historical dialogue records and dialogue strategies, realistic response strategies and obfuscated response strategies are provided. Furthermore, to enable the model to possess good rejection capabilities—that is, to reject unreasonable questions from customers or questions that the dialogue database cannot answer—so that the system can transfer the call to a human operator, corresponding rejection data is manually constructed based on the dialogue selection model's capability definition and added to the model fine-tuning dataset.

[0090] This embodiment uses high-quality customer service corpus to initialize the strategy, thereby fine-tuning the pre-trained first dialogue selection model.

[0091] Specifically, multiple dialogue texts are categorized to determine a positive sample set and a negative sample set. The positive sample set includes multiple positive dialogue texts, and the negative sample set includes multiple negative dialogue texts. The positive dialogue texts are triples automatically generated using rule scripts, combining "historical dialogue - trigger condition - correct phrase ID." Quality inspectors conduct a second review of boundary cases to ensure 100% label accuracy. The negative dialogue texts are scenarios with "completely no match" selected by rules, supplemented by large-scale model rewriting intent to generate adversarial examples, uniformly labeled with the rejection symbol "@". It should be noted that in this embodiment, the ratio of multiple positive dialogue texts to multiple negative dialogue texts can be 8:2 to avoid gradient imbalance. Each sample format is: Input = historical dialogue + policy library and phrase library (positive samples and distractors); Output = correct phrase ID or "@".

[0092] In this embodiment, the prompt template is called a Prompt. A Prompt is a guiding text input into a large language model to specify a task or question, helping the model generate a response that meets the requirements. The following is the Prompt template used by the dialogue selection model, which clearly defines the role and task of the dialogue selection model in the intelligent customer service system using Markdown format. This template emphasizes that the large language model must accurately select the most appropriate dialogue to respond to the customer based on the provided dialogue strategy, and ultimately only output the ID of the corresponding dialogue from the dialogue library, thereby effectively reducing inference latency.

[0093] #Role

[0094] Customer service specialist

[0095] #Task

[0096] As a customer service representative, please select the most appropriate response from the script library based on historical conversations and strictly follow the following dialogue strategy, then output the corresponding letter code.

[0097] #HistoricalDialogue

[0098] {chat_history}

[0099] #Dialogue strategy{script_strategy}

[0100] #Scripts library{scripts}

[0101] #Important Notes

[0102] - It is necessary to ensure that the output letter codes are selected from the script library.

[0103] - It is essential to pay attention to historical conversations and accurately understand the latest issues raised by clients, analyze their situations, and respond strictly in accordance with the established conversation strategy.

[0104] - You must strictly follow the dialogue strategy when responding; if there is no corresponding script in the script library for the customer's question or situation, it is best to refuse to answer directly, and it is forbidden to respond with scripts arbitrarily.

[0105] #Output Format <str>

[0106] #Output

[0107] After one round of model fine-tuning, the confidence scores of the labels of all training samples in the output of the fine-tuned model are calculated, and samples with too low confidence are filtered out. These samples with too low confidence may be due to inappropriate responses from the original customer service representatives and can therefore be considered noise samples. Finally, the model is trained for several more rounds on the dataset after filtering out noise samples to obtain the final optimized second dialogue selection model.

[0108] Optionally, before performing reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain the third dialogue selection model, the method further includes:

[0109] The system sets up a first reward rule, a second reward rule, and a third reward rule. The first reward rule includes a reward value of x+1 when the matching degree between the output of the third dialogue selection model and the user-input target question is greater than or equal to a first preset value; a reward value of xα when the matching degree between the output of the third dialogue selection model and the user-input target question is less than the first preset value but greater than the second preset value; and a reward value of -1 when the matching degree between the output of the third dialogue selection model and the user-input target question is less than or equal to the second preset value. α∈(0,1).

[0110] The reward model is generated based on the first reward rule, the second reward rule, and the third reward rule.

[0111] In this embodiment, the reward model is defined as follows:

[0112] First reward rule: Corresponding script: High-quality customer service script that perfectly matches real conversations will receive a fixed reward of (x) +1.

[0113] The second reward rule is to randomly select from historical high-quality phrases, and the reward is calculated as α × (semantic similarity), where the similarity is the cosine of the sentence vectors of the candidate phrase and the golden phrase; α∈(0,1) is the confidence discount to ensure that the reward is always less than +1.

[0114] Third reward rule: The reward for this type of distracting dialogue is directly set to -1. Negative sample for refusal to answer: Output refusal to answer: fixed reward +1. Random alternative dialogue: Because in this data scenario, the reasonable response for the large model should be refusal to answer, if there is no correct output of refusal to answer, the reward is directly set to -1.

[0115] Therefore, the reward model is generated through the first reward rule, the second reward rule, and the third reward rule.

[0116] Optionally, the step of performing reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain the third dialogue selection model includes:

[0117] The reward model is calculated based on the near-end policy optimization algorithm to obtain the advantage function;

[0118] Based on the advantage function and the training dataset, reinforcement learning is performed on the second dialogue selection model to obtain the third dialogue selection model.

[0119] In this embodiment, Proximal Policy Optimization (PPO) is a reinforcement learning algorithm belonging to the policy gradient method. The design goal of PPO is to improve the stability and efficiency of training. Especially for large-scale problems, PPO is widely welcomed due to its simplicity and effectiveness.

[0120] The reward model is calculated using a proximal policy optimization algorithm to obtain the advantage function. Based on the advantage function and the training dataset, reinforcement learning is performed on the second dialogue selection model. After KL regularization and gradient pruning, the policy is updated to achieve continuous adaptive optimization of the dialogue policy, and finally the third dialogue selection model is obtained.

[0121] In this embodiment, an adaptive reinforcement learning method for customer service scripts based on a semantic similarity reward mechanism is used to enable the trained large model to have good script selection and rejection capabilities. Constructing a reward function by measuring semantic similarity can avoid the need for manual labeling.

[0122] This application improves the quality and efficiency of the dialogue responses generated by the dialogue selection model by obtaining a training dataset containing multiple dialogue texts, fine-tuning the pre-trained dialogue selection model based on the training dataset, and performing reinforcement learning on the dialogue selection model based on the reward model and the training dataset.

[0123] like Figure 4 As shown, Figure 4 A method for selecting dialogue scripts is provided, the method comprising:

[0124] Step 401: Obtain the target problem as input;

[0125] Step 402: Input the target question into the third dialogue selection model in any of the above embodiments to match the target question and output the dialogue answer that matches the target question.

[0126] In this embodiment, after the third dialogue selection model is launched, the intelligent customer service system can be officially put into use and begin providing services to customers. When the system receives customer input, it integrates the historical dialogue records between the intelligent customer service representative and the customer, the pre-set response strategies, and a carefully selected high-quality dialogue script library to form the input prompts for the dialogue selection model. Based on this rich information, the dialogue selection model will conduct in-depth analysis and make corresponding decisions.

[0127] In practical use, if the model determines that a customer's request or problem exceeds the capabilities of the intelligent customer service system, the system will automatically exit the automatic reply mode and transfer the conversation to a human customer service representative to ensure that this special situation is handled properly. Under normal circumstances, the script selection model will choose the most suitable script from the script library as a reply template. Before officially sending a reply to the customer, the system will also optimize it through a polishing model. The polishing model is usually a large language model with a small number of parameters (e.g., Qwen3-1.7B). It mainly focuses on the customer's latest statements and adds some semantically coherent or emotionally reassuring expressions to the standard script to improve customer satisfaction. In addition, to ensure the diversity and freshness of the replies and avoid a decrease in customer satisfaction due to the use of overly similar or identical scripts, the system will remove the selected script from the script library after the current round of replies.

[0128] This application improves the quality and efficiency of the dialogue responses generated by the dialogue selection model by obtaining a training dataset containing multiple dialogue texts, fine-tuning the pre-trained dialogue selection model based on the training dataset, and performing reinforcement learning on the dialogue selection model based on the reward model and the training dataset.

[0129] See Figure 5 , Figure 5 This is a structural diagram of the training device for the discourse selection model provided in the embodiments of this application. Figure 5 As shown, the training device 500 for the dialogue selection model includes:

[0130] The first acquisition module 510 is used to acquire a training dataset, which includes multiple dialogue texts, each of which includes an input question and a corresponding verbal response to the input question.

[0131] The first training module 520 is used to fine-tune the pre-trained first dialogue selection model based on the training dataset to obtain the second dialogue selection model.

[0132] The second training module 530 is used to perform reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain a third dialogue selection model. The third dialogue selection model is used to match the target question input by the user and output a dialogue answer that matches the target question.

[0133] Optional, also includes:

[0134] The text acquisition module is used to acquire multiple historical dialogue texts, each of which includes a historical input question and a historical response to the historical input question.

[0135] The conversion module is used to perform vector conversion on the multiple historical dialogue texts to obtain multiple feature vectors;

[0136] The clustering module is used to perform multiple clustering on the multiple feature vectors to generate multiple clusters. Each cluster includes at least two feature vectors, and the similarity between at least two historical input questions corresponding to the at least two feature vectors included in each cluster meets a preset condition.

[0137] A determination module is used to determine the plurality of dialogue texts from the plurality of historical dialogue texts based on the plurality of clusters;

[0138] The generation module is used to generate the training dataset based on the multiple dialogue texts.

[0139] Optionally, the determination module includes:

[0140] The first determining submodule is used to determine the intention reversal score and the emotion reversal score corresponding to each cluster, wherein the intention reversal score is used to indicate the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's intention, and the emotion reversal score is used to indicate the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's emotion.

[0141] The second determining submodule is used to determine at least two historical dialogue texts corresponding to the target cluster as the dialogue text when the intention reversal score corresponding to the target cluster is greater than a first preset score value and the emotion reversal score corresponding to the target cluster is greater than a second preset score value. The target cluster is any one of the plurality of clusters.

[0142] Optionally, the first training module 520 includes:

[0143] The classification submodule is used to classify the multiple dialogue texts, determine the positive sample set and the negative sample set, wherein the positive sample set includes multiple positive sample dialogue texts, the negative sample set includes multiple negative sample dialogue texts, and the ratio of the multiple positive sample dialogue texts to the multiple negative sample dialogue texts is a preset ratio.

[0144] The acquisition submodule is used to acquire the prompt word template, which includes an output format and output content. The output format is a text format, and the output content is a dialogue response.

[0145] The fine-tuning submodule is used to fine-tune the pre-trained first dialogue selection model based on the positive sample set, the negative sample set, and the prompt word template to obtain the second dialogue selection model.

[0146] Optional, also includes:

[0147] The setting module is used to set a first reward rule, a second reward rule, and a third reward rule. The first reward rule includes a reward value of x+1 when the matching degree between the dialogue question and answer output by the third dialogue selection model and the target question input by the user is greater than or equal to a first preset value; the first reward rule also includes a reward value of xα when the matching degree between the dialogue question and answer output by the third dialogue selection model and the target question input by the user is less than the first preset value but greater than the second preset value; the third reward rule includes a reward value of -1 when the matching degree between the dialogue question and answer output by the third dialogue selection model and the target question input by the user is less than or equal to the second preset value, where α∈(0,1).

[0148] The generation module is used to generate the reward model based on the first reward rule, the second reward rule, and the third reward rule.

[0149] Optionally, the second training module 530 includes:

[0150] The calculation submodule is used to calculate the reward model based on the near-end policy optimization algorithm to obtain the advantage function;

[0151] The reinforcement learning module is used to perform reinforcement learning on the second dialogue selection model based on the advantage function and the training dataset to obtain the third dialogue selection model.

[0152] This application improves the quality and efficiency of the dialogue responses generated by the dialogue selection model by obtaining a training dataset containing multiple dialogue texts, fine-tuning the pre-trained dialogue selection model based on the training dataset, and performing reinforcement learning on the dialogue selection model based on the reward model and the training dataset.

[0153] See Figure 6 , Figure 6 This is a structural diagram of the speech selection device provided in an embodiment of this application. Figure 6 As shown, the speech selection device 600 includes:

[0154] The second acquisition module 610 is used to acquire the input target problem;

[0155] The matching module 620 is used to input the target question into the third dialogue selection model described in any of the above embodiments to match the target question and output a dialogue answer that matches the target question.

[0156] This application improves the quality and efficiency of the dialogue responses generated by the dialogue selection model by obtaining a training dataset containing multiple dialogue texts, fine-tuning the pre-trained dialogue selection model based on the training dataset, and performing reinforcement learning on the dialogue selection model based on the reward model and the training dataset.

[0157] This application also provides an electronic device. Please refer to [link to relevant documentation]. Figure 7 The electronic device may include a processor 701, a memory 702, and a program 7021 stored in the memory 702 and capable of running on the processor 701.

[0158] When program 7021 is executed by processor 701, it can achieve the following: Figure 1 or Figure 4 Any step in the corresponding method embodiment:

[0159] Obtain a training dataset, which includes multiple dialogue texts, each of which includes an input question and a corresponding verbal response to the input question;

[0160] Based on the training dataset, the pre-trained first script selection model is fine-tuned to obtain the second script selection model.

[0161] Based on the reward model and the training dataset, reinforcement learning is performed on the second dialogue selection model to obtain a third dialogue selection model. The third dialogue selection model is used to match the target question input by the user and output a dialogue answer that matches the target question.

[0162] Optionally, before obtaining the training dataset, the method further includes:

[0163] Acquire multiple historical dialogue texts, each of which includes a historical input question and a historical response to the historical input question;

[0164] The multiple historical dialogue texts are vectorized to obtain multiple feature vectors;

[0165] Multiple clusters are generated by performing multiple clustering on the multiple feature vectors. Each cluster includes at least two feature vectors. The similarity between at least two historical input questions corresponding to the at least two feature vectors included in each cluster meets a preset condition.

[0166] Based on the multiple clusters, the multiple dialogue texts are determined from the multiple historical dialogue texts;

[0167] The training dataset is generated based on the multiple dialogue texts.

[0168] Optionally, determining the plurality of dialogue texts from the plurality of historical dialogue texts based on the plurality of clusters includes:

[0169] Each cluster is assigned an intent reversal score and an emotion reversal score. The intent reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's intent, and the emotion reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's emotion.

[0170] For a target cluster, if the intention reversal score corresponding to the target cluster is greater than a first preset score value and the emotion reversal score corresponding to the target cluster is greater than a second preset score value, then at least two historical dialogue texts corresponding to the target cluster are determined as the dialogue texts, and the target cluster is any one of the plurality of clusters.

[0171] Optionally, the step of fine-tuning the pre-trained first dialogue selection model based on the training dataset to obtain the second dialogue selection model includes:

[0172] The multiple dialogue texts are classified to determine a positive sample set and a negative sample set. The positive sample set includes multiple positive sample dialogue texts, and the negative sample set includes multiple negative sample dialogue texts. The ratio of the multiple positive sample dialogue texts to the multiple negative sample dialogue texts is a preset ratio.

[0173] Obtain a prompt word template, the prompt word template includes an output format and output content, the output format is a text format, and the output content is a verbal response;

[0174] Based on the positive sample set, the negative sample set, and the prompt word template, the pre-trained first dialogue selection model is fine-tuned to obtain the second dialogue selection model.

[0175] Optionally, before performing reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain the third dialogue selection model, the method further includes:

[0176] The system sets up a first reward rule, a second reward rule, and a third reward rule. The first reward rule includes a reward value of x+1 when the matching degree between the output of the third dialogue selection model and the user-input target question is greater than or equal to a first preset value; a reward value of xα when the matching degree between the output of the third dialogue selection model and the user-input target question is less than the first preset value but greater than the second preset value; and a reward value of -1 when the matching degree between the output of the third dialogue selection model and the user-input target question is less than or equal to the second preset value. α∈(0,1).

[0177] The reward model is generated based on the first reward rule, the second reward rule, and the third reward rule.

[0178] Optionally, the step of performing reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain the third dialogue selection model includes:

[0179] The reward model is calculated based on the near-end policy optimization algorithm to obtain the advantage function;

[0180] Based on the advantage function and the training dataset, reinforcement learning is performed on the second dialogue selection model to obtain the third dialogue selection model.

[0181] or

[0182] Obtain the target problem as input;

[0183] The target question is input into the third dialogue selection model described in any of the above embodiments to match the target question, and the dialogue answer that matches the target question is output.

[0184] This application improves the quality and efficiency of the dialogue responses generated by the dialogue selection model by obtaining a training dataset containing multiple dialogue texts, fine-tuning the pre-trained dialogue selection model based on the training dataset, and performing reinforcement learning on the dialogue selection model based on the reward model and the training dataset.

[0185] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described training method for the speech selection model or the speech selection method embodiment, and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0186] This application also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described training method or method embodiment of the speech selection model, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0187] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0188] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0189] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.< / str>

Claims

1. A training method for a dialogue selection model, characterized in that, The method includes: Obtain a training dataset, which includes multiple dialogue texts, each of which includes an input question and a corresponding verbal response to the input question; Based on the training dataset, the pre-trained first dialogue selection model is fine-tuned to obtain the second dialogue selection model. Based on the reward model and the training dataset, reinforcement learning is performed on the second dialogue selection model to obtain a third dialogue selection model. The third dialogue selection model is used to match the target question input by the user and output a dialogue answer that matches the target question.

2. The method according to claim 1, characterized in that, Before obtaining the training dataset, the method further includes: Obtain multiple historical dialogue texts, each of which includes a historical input question and a historical response to the historical input question; The multiple historical dialogue texts are vectorized to obtain multiple feature vectors; Multiple clusters are generated by performing multiple clustering on the multiple feature vectors. Each cluster includes at least two feature vectors. The similarity between at least two historical input questions corresponding to the at least two feature vectors included in each cluster meets a preset condition. Based on the multiple clusters, the multiple dialogue texts are determined from the multiple historical dialogue texts; The training dataset is generated based on the multiple dialogue texts.

3. The method according to claim 2, characterized in that, The step of determining the multiple dialogue texts from the multiple historical dialogue texts based on the multiple clusters includes: Each cluster is assigned an intent reversal score and an emotion reversal score. The intent reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's intent, and the emotion reversal score indicates the degree of influence of at least two historical dialogue texts corresponding to the cluster on the user's emotion. For a target cluster, if the intention reversal score corresponding to the target cluster is greater than a first preset score value and the emotion reversal score corresponding to the target cluster is greater than a second preset score value, then at least two historical dialogue texts corresponding to the target cluster are determined as the dialogue texts, and the target cluster is any one of the plurality of clusters.

4. The method according to claim 1, characterized in that, The step of fine-tuning the pre-trained first dialogue selection model based on the training dataset to obtain the second dialogue selection model includes: The multiple dialogue texts are classified to determine a positive sample set and a negative sample set. The positive sample set includes multiple positive sample dialogue texts, and the negative sample set includes multiple negative sample dialogue texts. The ratio of the multiple positive sample dialogue texts to the multiple negative sample dialogue texts is a preset ratio. Obtain a prompt word template, the prompt word template includes an output format and output content, the output format is a text format, and the output content is a verbal response; Based on the positive sample set, the negative sample set, and the prompt word template, the pre-trained first dialogue selection model is fine-tuned to obtain the second dialogue selection model.

5. The method according to claim 1, characterized in that, Before performing reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain the third dialogue selection model, the method further includes: The system sets up a first reward rule, a second reward rule, and a third reward rule. The first reward rule includes a reward value of x+1 when the matching degree between the output of the third dialogue selection model and the user-input target question is greater than or equal to a first preset value; a reward value of xα when the matching degree between the output of the third dialogue selection model and the user-input target question is less than the first preset value but greater than the second preset value; and a reward value of -1 when the matching degree between the output of the third dialogue selection model and the user-input target question is less than or equal to the second preset value. α∈(0,1). The reward model is generated based on the first reward rule, the second reward rule, and the third reward rule.

6. The method according to claim 5, characterized in that, The third dialogue selection model is obtained by performing reinforcement learning on the second dialogue selection model based on the reward model and the training dataset, including: The reward model is calculated based on the near-end policy optimization algorithm to obtain the advantage function; Based on the advantage function and the training dataset, reinforcement learning is performed on the second dialogue selection model to obtain the third dialogue selection model.

7. A method for selecting sales scripts, characterized in that, The method includes: Obtain the target problem as input; The target question is input into the third dialogue selection model according to any one of claims 1-6 to match the target question, and the dialogue answer that matches the target question is output.

8. A training device for a dialogue selection model, characterized in that, The device includes: The first acquisition module is used to acquire a training dataset, which includes multiple dialogue texts, each of which includes an input question and a corresponding verbal response to the input question. The first training module is used to fine-tune the pre-trained first dialogue selection model based on the training dataset to obtain the second dialogue selection model. The second training module is used to perform reinforcement learning on the second dialogue selection model based on the reward model and the training dataset to obtain a third dialogue selection model. The third dialogue selection model is used to match the target question input by the user and output a dialogue answer that matches the target question.

9. A device for selecting dialogue scripts, characterized in that, The device includes: The second acquisition module is used to acquire the target problem as input. The matching module is used to input the target question into the third dialogue selection model according to any one of claims 1-6 to match the target question and output a dialogue answer that matches the target question.

10. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program in the memory to implement the steps in training a speech selection model as described in any one of claims 1 to 6, or to implement the steps in the speech selection method as described in claim 7.

Images