Dialogue data generation method and device, equipment, medium and computer program product

Abstract

The invention provides a dialogue data generation method and device, equipment, a medium and a computer program product, and the method comprises the steps: obtaining at least one sample dialogue path according to original voice dialogue data; each clustering label contained in the sample dialogue path corresponds to a dialogue fragment category; obtaining a transition probability between clustering labels in the sample dialogue path to form a dialogue transition matrix; generating a simulated dialogue path according to the dialogue transfer matrix; screening the simulated dialogue paths to obtain a target dialogue path; and obtaining simulated dialogue data according to the target dialogue path. According to the scheme, the simulation dialogue data can be generated based on the real dialogue text (namely the original voice dialogue data), it is ensured that the generated data structure conforms to the actual scene, the accuracy of the generated dialogue data is improved, and the problem that in the prior art, a dialogue data obtaining scheme is low in accuracy is well solved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device, medium and computer program product for generating dialogue data. Background Technology

[0002] With the widespread application of intelligent customer service and AI (Artificial Intelligence) assistants (such as AI sales assistants), high-quality dialogue data (such as telemarketing dialogue data) is crucial for training AI models; however, existing methods for obtaining dialogue data have the following drawbacks:

[0003] a) Existing methods are mostly based on template filling or random generation of LLM (Large Language Model), which lacks modeling of real dialogue logic and may result in inconsistent data.

[0004] b) Directly using dialogue data generated by LLM often results in unclear commercial value, and the generated scripts may not conform to the actual interaction logic.

[0005] As shown above, existing methods for obtaining dialogue data suffer from problems such as low accuracy. Summary of the Invention

[0006] The purpose of this application is to provide a method, apparatus, device, medium, and computer program product for generating dialogue data, so as to solve the problem of low accuracy in existing solutions for obtaining dialogue data.

[0007] To address the aforementioned technical problems, embodiments of this application provide a method for generating dialogue data, including:

[0008] Based on the original voice dialogue data, at least one sample dialogue path is obtained; each clustering label contained in the sample dialogue path corresponds to a dialogue segment category.

[0009] Obtain the transition probabilities between clustered tags in the sample dialogue path to form a dialogue transition matrix;

[0010] Based on the dialogue transition matrix, a simulated dialogue path is generated;

[0011] The simulated dialogue path is filtered to obtain the target dialogue path;

[0012] Based on the target dialogue path, simulated dialogue data is obtained.

[0013] Optionally, obtaining at least one sample dialogue path based on the original voice dialogue data includes:

[0014] Convert the raw voice dialogue data into raw dialogue text; the raw dialogue text includes at least one dialogue text.

[0015] Obtain the sentence vector corresponding to each dialogue text in the original dialogue text;

[0016] Based on the sentence vectors corresponding to each dialogue text, sample dialogue paths corresponding to each dialogue text are formed.

[0017] Optionally, the clustering labels in the sample dialogue path include: a first clustering label and a second clustering label;

[0018] The step of obtaining the transition probabilities between clustered labels in the sample dialogue path to form a dialogue transition matrix includes:

[0019] Obtain the initial probability of the first cluster label transitioning to the second cluster label;

[0020] If the initial probability is less than the threshold, the first value is used as the transition probability of the first cluster label to the second cluster label;

[0021] If the initial probability is greater than or equal to the threshold, the initial probability is used as the transition probability for the first cluster label to the second cluster label;

[0022] Based on the aforementioned transition probabilities, a dialogue transition matrix is ​​formed.

[0023] Optional, also includes:

[0024] For at least one of the simulated dialogue paths, an annotation operation is performed to obtain training data;

[0025] Based on the training data, a classification model is trained.

[0026] The step of filtering the simulated dialogue path to obtain the target dialogue path includes:

[0027] The simulated dialogue path is scored using the classification model to obtain the scoring results;

[0028] Based on the scoring results, the simulated dialogue paths are filtered to obtain the target dialogue path.

[0029] Optionally, one clustering label corresponds to at least one dialogue sample;

[0030] The step of obtaining simulated dialogue data based on the target dialogue path includes:

[0031] For each cluster label in the target dialogue path, a dialogue sample random extraction operation is performed to obtain the target dialogue sample;

[0032] Based on the target dialogue sample, simulated dialogue data is obtained.

[0033] Optionally, the dialogue data generation method further includes:

[0034] The simulated dialogue data is adjusted using LLM to obtain the final simulated dialogue data.

[0035] And / or, the dialogue data generation method further includes:

[0036] The dialogue transition matrix is ​​adjusted to obtain the final dialogue transition matrix;

[0037] The step of generating a simulated dialogue path based on the dialogue transition matrix includes:

[0038] Based on the final dialogue transition matrix, a simulated dialogue path is generated.

[0039] This application also provides a method for generating dialogue data, including:

[0040] Receive user input;

[0041] Using a first model, dialogue data corresponding to the user input is obtained; wherein, the first model is trained using simulated dialogue data generated by the above-described dialogue data generation method.

[0042] This application also provides a dialogue data generation apparatus, including:

[0043] The first acquisition module is used to acquire at least one sample dialogue path based on the original voice dialogue data; each clustering label contained in the sample dialogue path corresponds to a dialogue segment category.

[0044] The first processing module is used to obtain the transition probabilities between clustering labels in the sample dialogue path and form a dialogue transition matrix.

[0045] The first generation module is used to generate a simulated dialogue path based on the dialogue transition matrix;

[0046] The first filtering module is used to filter the simulated dialogue path to obtain the target dialogue path;

[0047] The second processing module is used to obtain simulated dialogue data based on the target dialogue path.

[0048] Optionally, obtaining at least one sample dialogue path based on the original voice dialogue data includes:

[0049] Convert the raw voice dialogue data into raw dialogue text; the raw dialogue text includes at least one dialogue text.

[0050] Obtain the sentence vector corresponding to each dialogue text in the original dialogue text;

[0051] Based on the sentence vectors corresponding to each dialogue text, sample dialogue paths corresponding to each dialogue text are formed.

[0052] Optionally, the clustering labels in the sample dialogue path include: a first clustering label and a second clustering label;

[0053] The step of obtaining the transition probabilities between clustered labels in the sample dialogue path to form a dialogue transition matrix includes:

[0054] Obtain the initial probability of the first cluster label transitioning to the second cluster label;

[0055] If the initial probability is less than the threshold, the first value is used as the transition probability of the first cluster label to the second cluster label;

[0056] If the initial probability is greater than or equal to the threshold, the initial probability is used as the transition probability for the first cluster label to the second cluster label;

[0057] Based on the aforementioned transition probabilities, a dialogue transition matrix is ​​formed.

[0058] Optional, also includes:

[0059] The third processing module is used to perform annotation operations on at least one of the simulated dialogue paths to obtain training data;

[0060] The first training module is used to train a classification model based on the training data;

[0061] The step of filtering the simulated dialogue path to obtain the target dialogue path includes:

[0062] The simulated dialogue path is scored using the classification model to obtain the scoring results;

[0063] Based on the scoring results, the simulated dialogue paths are filtered to obtain the target dialogue path.

[0064] Optionally, one clustering label corresponds to at least one dialogue sample;

[0065] The step of obtaining simulated dialogue data based on the target dialogue path includes:

[0066] For each cluster label in the target dialogue path, a dialogue sample random extraction operation is performed to obtain the target dialogue sample;

[0067] Based on the target dialogue sample, simulated dialogue data is obtained.

[0068] Optionally, the dialogue data generation device further includes:

[0069] The first adjustment module is used to adjust the simulated dialogue data using LLM to obtain the final simulated dialogue data.

[0070] And / or, the dialogue data generation device further includes:

[0071] The second adjustment module is used to adjust the dialogue transition matrix to obtain the final dialogue transition matrix.

[0072] The step of generating a simulated dialogue path based on the dialogue transition matrix includes:

[0073] Based on the final dialogue transition matrix, a simulated dialogue path is generated.

[0074] This application also provides a dialogue data generation apparatus, including:

[0075] The first receiving module is used to receive user input;

[0076] The second acquisition module is used to acquire dialogue data corresponding to the user input using the first model; wherein the first model is trained using simulated dialogue data generated by the first set of dialogue data generation methods described above.

[0077] This application also provides a dialogue data generation device, including a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, it implements the dialogue data generation method as described above.

[0078] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the steps in the above-described dialogue data generation method.

[0079] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the above-described dialogue data generation method.

[0080] The beneficial effects of the above technical solution in this application are as follows:

[0081] In the above scheme, the dialogue data generation method obtains at least one sample dialogue path based on the original voice dialogue data; each cluster label contained in the sample dialogue path corresponds to a dialogue segment category; the transition probabilities between the cluster labels in the sample dialogue path are obtained to form a dialogue transition matrix; a simulated dialogue path is generated based on the dialogue transition matrix; the simulated dialogue path is filtered to obtain a target dialogue path; and simulated dialogue data is obtained based on the target dialogue path. This method can support the generation of simulated dialogue data based on real dialogue text (i.e., original voice dialogue data), ensuring that the generated data structure conforms to the actual scenario, avoiding data inconsistency caused by a lack of modeling of real dialogue logic, and avoiding data that does not conform to actual interaction logic, thereby improving the accuracy of the generated dialogue data and effectively solving the problem of low accuracy in existing schemes for obtaining dialogue data. Attached Figure Description

[0082] Figure 1 This is a schematic flowchart of the dialogue data generation method according to an embodiment of this application. Figure 1 ;

[0083] Figure 2 This is a schematic flowchart of the dialogue data generation method according to an embodiment of this application. Figure 2 ;

[0084] Figure 3 This is a schematic diagram of the dialogue data generation device according to an embodiment of this application. Figure 1 ;

[0085] Figure 4 This is a schematic diagram of the dialogue data generation device according to an embodiment of this application. Figure 2 . Detailed Implementation

[0086] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0087] This application addresses the problem of low accuracy in existing methods for obtaining dialogue data by providing a method for generating dialogue data, such as... Figure 1 As shown, it includes:

[0088] Step 11: Based on the original voice dialogue data, obtain at least one sample dialogue path; each cluster label contained in the sample dialogue path corresponds to a dialogue segment category.

[0089] This approach does not limit the specific method for obtaining sample dialogue paths. For example, the original voice dialogue data can be converted into text and then subjected to sentence vector transformation and clustering to obtain the sample dialogue paths. Each sample dialogue path can correspond to one dialogue, but this is not a limitation.

[0090] For example, the original voice dialogue data consists of speech containing text content (question A1, answer B1; question A2, answer B2); the obtained sample dialogue paths are (cluster label 1, cluster label 2), where cluster label 1 corresponds to the dialogue segment category of the dialogue segment "question A1, answer B1", and cluster label 2 corresponds to the dialogue segment category of the dialogue segment "question A2, answer B2". One dialogue segment category can correspond to multiple dialogue segments; this example only uses one dialogue segment.

[0091] Step 12: Obtain the transition probabilities between clustered tags in the sample dialogue path to form a dialogue transition matrix.

[0092] Specifically, for example: obtain the transition probability 1 of moving from cluster label 1 to cluster label 1 (i.e. staying in cluster label 1), the transition probability 2 of moving from cluster label 1 to cluster label 2, the transition probability 3 of moving from cluster label 2 to cluster label 1, and the transition probability 4 of moving from cluster label 2 to cluster label 2 (i.e. staying in cluster label 2), and form a dialogue transition matrix 1 based on the transition probabilities 1, 2, 3, and 4.

[0093] Step 13: Generate a simulated dialogue path based on the dialogue transition matrix;

[0094] In this scheme, the specific method for generating simulated dialogue paths is not limited. For example, the Monte Carlo method can be used to randomly generate simulated dialogue paths based on the dialogue transition matrix. Specifically, the Monte Carlo method can be used to randomly generate simulated dialogue path 1 (cluster label 1 -> cluster label 2) and simulated dialogue path 2 (cluster label 2 -> cluster label 1) based on dialogue transition matrix 1.

[0095] Step 14: Filter the simulated dialogue paths to obtain the target dialogue path;

[0096] Step 14 can be implemented by performing a rejection sampling operation on the simulated dialogue path to obtain the target dialogue path. Specifically, for example, it can be determined whether simulated dialogue paths 1 and 2 meet certain conditions, such as whether the model score corresponding to simulated dialogue path 1 is greater than a certain threshold. If they meet the conditions, they can be used as a target dialogue path; if they do not meet the conditions, they can be discarded or deleted. Assuming that simulated dialogue path 2 meets the conditions, simulated dialogue path 2 can be used as a target dialogue path.

[0097] Step 15: Obtain simulated dialogue data based on the target dialogue path.

[0098] This solution does not limit the specific method of obtaining simulated dialogue data. For example, when performing text backfilling for each cluster label in the target dialogue path, any one of the dialogue samples corresponding to the cluster label can be used to replace the cluster label, but this is not a limitation. Specifically, for simulated dialogue path 2 (cluster label 2 -> cluster label 1), text backfilling can be performed. For example, the dialogue fragment "Question A2, Answer B2" corresponding to cluster label 2 can be filled into the first path node position of simulated dialogue path 2, and the dialogue fragment "Question A1, Answer B1" corresponding to cluster label 1 can be filled into the second path node position of simulated dialogue path 2, resulting in simulated dialogue data: Question A2, Answer B2, Question A1, Answer B1.

[0099] The dialogue data generation method provided in this application obtains at least one sample dialogue path based on the original voice dialogue data; each cluster label contained in the sample dialogue path corresponds to a dialogue segment category; the transition probabilities between the cluster labels in the sample dialogue path are obtained to form a dialogue transition matrix; a simulated dialogue path is generated based on the dialogue transition matrix; the simulated dialogue path is filtered to obtain a target dialogue path; and simulated dialogue data is obtained based on the target dialogue path. This method can support the generation of simulated dialogue data based on real dialogue text (i.e., original voice dialogue data), ensuring that the generated data structure conforms to the actual scenario, avoiding data inconsistency caused by a lack of modeling of real dialogue logic, and avoiding data that does not conform to actual interaction logic, thereby improving the accuracy of the generated dialogue data and effectively solving the problem of low accuracy in existing methods for obtaining dialogue data. Furthermore, the dialogue data generation method provided in this application supports the automatic generation of simulated dialogue data, which can avoid the problem of high data collection costs in existing telemarketing (dialogue) data acquisition methods. This is because: a) real telemarketing data is usually collected internally by enterprises, involves customer privacy, and is difficult to share and label on a large scale; b) due to the different telemarketing models of different products and customer groups, a single data source is difficult to meet the needs of multiple scenarios. The solution provided in this application can support the automatic generation of simulated dialogue data for different scenario needs, avoiding the direct collection of real telemarketing data.

[0100] The step of obtaining at least one sample dialogue path based on the original voice dialogue data includes: converting the original voice dialogue data into original dialogue text; the original dialogue text includes at least one dialogue text; obtaining the sentence vectors corresponding to each dialogue text in the original dialogue text; and forming sample dialogue paths corresponding to each dialogue text based on the sentence vectors corresponding to each dialogue text. This allows for accurate acquisition of the sample dialogue paths. In this solution, before obtaining the sentence vectors corresponding to each dialogue text in the original dialogue text, adjustments can be made to the original dialogue text, such as correcting common errors using a domain vocabulary or a large model, to minimize the introduction of noise (such as misspellings or omissions). Furthermore, "one dialogue text" can correspond to a dialogue, such as including at least one dialogue segment. Additionally, "obtaining the sentence vectors corresponding to each dialogue text in the original dialogue text" can include mapping each dialogue text in the original dialogue text to a vector space to obtain at least one sentence vector, but is not limited to this. Furthermore, in this solution, the specific method for forming sample dialogue paths is not limited. For example, Start and End can be set as virtual nodes. The step of forming sample dialogue paths corresponding to each dialogue text based on the sentence vectors corresponding to each dialogue text can include: forming sample dialogue paths corresponding to each dialogue text based on the virtual nodes and the sentence vectors corresponding to each dialogue text. In this way, a complete dialogue path can be constructed for a dialogue, but it is not limited to this.

[0101] In this embodiment, the clustering labels in the sample dialogue path include: a first clustering label and a second clustering label; wherein, obtaining the transition probabilities between the clustering labels in the sample dialogue path to form a dialogue transition matrix includes: obtaining the initial probability of the first clustering label transitioning to the second clustering label; if the initial probability is less than a threshold, using a first value as the transition probability of the first clustering label transitioning to the second clustering label; if the initial probability is greater than or equal to the threshold, using the initial probability as the transition probability of the first clustering label transitioning to the second clustering label; and forming a dialogue transition matrix based on the transition probabilities. This can accurately form the dialogue transition matrix, where the first value can be 0, which avoids noisy data, effectively improves the efficiency of subsequent Monte Carlo simulation, and reduces the probability of generating incorrect paths. Furthermore, this solution does not limit the specific method for forming the dialogue transition matrix. For example, the first sorting of the clustering labels can be used as the row and column element sorting of the dialogue transition matrix, and the obtained transition probabilities of each clustering label can be filled into the corresponding matrix element positions to obtain the dialogue transition matrix, but this is not a limitation.

[0102] Furthermore, the dialogue data generation method further includes: labeling at least one of the simulated dialogue paths to obtain training data; training a classification model based on the training data; wherein, the step of filtering the simulated dialogue paths to obtain target dialogue paths includes: using the classification model to score the simulated dialogue paths to obtain scoring results; and filtering the simulated dialogue paths based on the scoring results to obtain target dialogue paths. This supports the specific implementation of rejection sampling operations for simulated dialogue paths. In this solution, the specific method for training the classification model is not limited; for example, inputting the training data into an initial binary classification model for model training to obtain the classification model is acceptable, but this is not a limitation. Additionally, in this solution, the specific method for filtering the simulated dialogue paths is not limited; for example, if the scoring result indicates a score greater than a preset threshold, the simulated dialogue path corresponding to that score is retained; otherwise, the corresponding simulated dialogue path is discarded, but this is not a limitation.

[0103] In this scheme, each cluster label corresponds to at least one dialogue sample. Obtaining simulated dialogue data based on the target dialogue path includes: performing a random dialogue sample extraction operation (i.e., randomly extracting corresponding text for filling) on ​​each cluster label in the target dialogue path to obtain a target dialogue sample; and obtaining simulated dialogue data based on the target dialogue sample. This supports text backfilling operations, specifically obtaining simulated dialogue data, i.e., a complete simulated dialogue. One dialogue sample can correspond to one dialogue segment, but is not limited to this. The specific method for obtaining simulated dialogue data is not limited in this scheme; for example, the target dialogue samples can be sorted according to the sorting position of their corresponding cluster labels to obtain simulated dialogue data, but this is not a limitation.

[0104] Furthermore, the dialogue data generation method also includes: using LLM to adjust the simulated dialogue data to obtain the final simulated dialogue data; this can support more natural data, reduce data errors, and enhance diversity.

[0105] And / or, the dialogue data generation method further includes: adjusting the dialogue transition matrix to obtain a final dialogue transition matrix; wherein, generating a simulated dialogue path based on the dialogue transition matrix includes: generating a simulated dialogue path based on the final dialogue transition matrix. This allows for modification of the dialogue transition matrix, introducing predetermined paths, ensuring sufficient data samples for rare paths, optimizing the long-tail problem, and addressing the shortcomings of existing technologies in optimizing the long-tail problem: a) In reality, some key but infrequently occurring dialogue patterns (such as rare customer objections or special sales strategies) account for a very small proportion of the data, making it difficult for models to learn these long-tail scenarios; b) Current methods mainly rely on manual annotation and rule generation, with limited scalability.

[0106] This application also provides a method for generating dialogue data, such as... Figure 2 As shown, it includes:

[0107] Step 21: Receive user input;

[0108] Step 22: Using the first model, obtain the dialogue data corresponding to the user input; wherein, the first model is a model that utilizes... Figure 1 The simulated dialogue data generated by the corresponding dialogue data generation method is used for training.

[0109] The dialogue data generation method provided in this application embodiment receives user input and uses a first model to obtain dialogue data corresponding to the user input; wherein, the first model is a method using... Figure 1 The dialogue data generation method is trained using simulated dialogue data generated by the corresponding dialogue data generation method; it can support the acquisition of dialogue data using a first model with high accuracy, thereby ensuring the acquisition of high-quality dialogue data and solving the problem of low accuracy in existing dialogue data acquisition schemes. Furthermore, the dialogue data generation method provided in this application embodiment supports the use of… Figure 1 The corresponding dialogue data generation method generates simulated dialogue data to train the first model. This avoids the problem of using real data for training due to inconsistent data quality, which could lead to the AI ​​model learning inefficient or incorrect sales scripts. Reasons for inconsistent data quality include: due to the different skill levels of telemarketing agents, the quality of real call data is uneven, and some content contains a lot of invalid information (such as redundant expressions, noise, and slips of the tongue).

[0110] The following is an example of the dialogue data generation method provided in the embodiments of this application, with the dialogue data being telemarketing dialogue data as an example.

[0111] To address the aforementioned technical problems, this application provides a method for generating dialogue data, specifically a method for generating telemarketing dialogue data. This method can be used for scenarios such as telemarketing dialogue data enhancement and telemarketing model training, solving the problem of generating telemarketing dialogue data. This solution mainly involves: based on the ideas of clustering and Monte Carlo search trees, as well as result text backfilling and generation enhancement, ultimately constructing high-quality telemarketing dialogue data. Specifically, this solution includes the following:

[0112] Step 1: Data preprocessing;

[0113] Specifically, this step may include: collecting raw voice dialogue data, and converting the raw voice dialogue data (such as audio from a telemarketing call) into text using methods such as ASR (Automatic Speech Recognition), for example as follows:

[0114] D = {(s1,c1),(s2,c2),...,(s...} n ,c n )};

[0115] Where D represents a dialogue text in the original voice dialogue data; s i The telemarketer's statement, i.e., the audio transcript of the telemarketer; c i This represents the customer's response, i.e., the customer's audio text; i = 1...n, where n is the total number of dialogue segments, and in this example, n is greater than 2.

[0116] In this solution, considering that ASR may introduce noise (such as misspellings or omissions), common ASR errors can be corrected by using a domain vocabulary or a large model, that is, by adjusting the output text of ASR.

[0117] Step one corresponds to the above-mentioned conversion of raw speech dialogue data into raw dialogue text; the raw dialogue text includes at least one dialogue text. The raw dialogue text may correspond to the output text of ASR, or the adjusted output text of ASR, and is not limited here.

[0118] Step 2: Semantic clustering;

[0119] Specifically, this step may include: mapping each dialogue to a vector space using sentence embeddings (such as BERT (bidirectional encoder representation from transformer) or SBERT (bidirectional encoder representation of sentences from transformer) as follows:

[0120] v i =Embedding(s i ), u i =Embedding(c i); that is, mapping the dialogue to sentence vectors; where i = 1...n; this operation corresponds to obtaining the sentence vectors corresponding to each dialogue text in the original dialogue text as described above; vi represents s i The corresponding sentence vector, ui, represents c. i The corresponding sentence vector, Embedding represents the mapping.

[0121] These sentence vectors can then be clustered using methods such as K-Means, hierarchical clustering, or community detection algorithms to obtain cluster labels, as follows:

[0122] C = {C1,C2,...,C} m};

[0123] Where C represents the set of cluster labels, m represents the total number of cluster labels, and each C k (k = 1···m) corresponds to a dialogue fragment category (each cluster label corresponds to a dialogue fragment category), and a node name can be set later for easier understanding.

[0124] In this scheme, Start and End are set as virtual nodes, so a dialogue (corresponding to a dialogue text) can construct a complete dialogue path C' (which can correspond to the sample dialogue path above), as shown below:

[0125] C′={Start,C1,C2,...,C m ,End}; can correspond to the above-mentioned sentence vectors corresponding to each dialogue text, forming sample dialogue paths corresponding to each dialogue text respectively.

[0126] Step 3: Construct the dialogue transition matrix;

[0127] Specifically, this step may include: calculating the transition probabilities between cluster labels to form a state transition matrix P (i.e., a dialogue transition matrix); corresponding to the aforementioned acquisition of the transition probabilities between cluster labels in the sample dialogue path, forming a dialogue transition matrix; wherein, the transition probability P ij It can be represented as follows:

[0128] P ij =P(C j |C i ); i = 1…m, j = 1…m;

[0129] That is, P ij This indicates that the current dialogue label (i.e., the current cluster label) is C. i In this case, move to cluster label C j The probability; the matrix P satisfies:

[0130] That is, the sum of all possible transition probabilities for each dialogue tag is 1.

[0131] Here's a specific example, assuming there are 3 types of dialogue fragments, i.e., 3 cluster labels C1, C2, and C3:

[0132] C1: Introduce the product;

[0133] C2: Inquire about customer needs;

[0134] C3: Handling objections.

[0135] The transition probability matrix calculated based on the above cluster labels might be as follows:

[0136]

[0137] in:

[0138] The first line indicates that C1 has a 10% probability of transitioning to C1 (i.e., remaining in its current state), a 60% probability of transitioning to C2, and a 30% probability of transitioning to C3.

[0139] The second line indicates that C2 has a 20% probability of transferring to C1, a 30% probability of transferring to C2 (i.e., remaining in the current state), and a 50% probability of entering C3.

[0140] The third line indicates that there is a 30% probability that C3 will move to C1, a 20% probability that it will move to C2, and a 50% probability that it will move to C3, meaning there is a 50% probability that it will remain in its current state (which may be due to continued objections from the customer).

[0141] Furthermore, in this scheme, matrix P can be modified (based on experience and other information), for example, by introducing manually set paths (i.e., increasing the number of rows and columns of matrix P or modifying the probability distribution), so that rare paths can have sufficient data samples to optimize the long-tail problem; corresponding to the dialogue data generation method described above, it also includes: adjusting the dialogue transition matrix to obtain the final dialogue transition matrix; subsequently, the step of generating a simulated dialogue path based on the dialogue transition matrix can include: generating a simulated dialogue path based on the final dialogue transition matrix.

[0142] In this scheme, the transition probability between cluster labels, such as the probability P(C2|C1) of C1 transitioning to C2, can be determined in the following way:

[0143] P(C2|C1) = {the total number of consecutive occurrences of C1 and C2 in all dialogue paths contained in the original voice dialogue data} / {the total number of occurrences of C1 in all dialogue paths contained in the original voice dialogue data}, the obtained value can be used as the initial probability; this part can correspond to the cluster labels in the above sample dialogue paths, including: the first cluster label and the second cluster label; obtain the initial probability of the first cluster label moving to the second cluster label.

[0144] Furthermore, a threshold can be set. When the probability P(C2|C1) is less than the threshold, it is directly set to the first value (e.g., 0), thus updating the value of probability P(C2|C1). This corresponds to the aforementioned case where, when the initial probability is less than the threshold, the first value is used as the transition probability to the second cluster label. This is because, considering that cases where probability P(C2|C1) is less than the threshold are usually noisy data, setting the probability to 0 can effectively improve the efficiency of subsequent Monte Carlo simulations and reduce the probability of generating incorrect paths. For cases where probability P(C2|C1) is not less than the threshold, the obtained value can be directly used as probability P(C2|C1); that is, corresponding to the aforementioned case where, when the initial probability is greater than or equal to the threshold, the initial probability is used as the transition probability to the second cluster label. Based on the obtained transition probabilities, a dialogue transition matrix can be formed.

[0145] Step 4: Generate the dialogue path;

[0146] Specifically, this step may include: using the Monte Carlo simulation method to randomly generate dialogue paths based on the (state) transition matrix P, which corresponds to generating simulated dialogue paths based on the aforementioned dialogue transition matrix; specifically, the following operations can be performed as an example:

[0147] 1. Set the initial dialogue state to Start.

[0148] 2. Randomly sample the next state: C (t+1) ~P C_t That is, the next state follows a probability distribution, such as C2 following the values ​​0.2, 0.3, and 0.5 in the example matrix P mentioned above.

[0149] 3. Repeat until the End state is reached.

[0150] For example, assuming the initial state is Start, using the transition matrix P for path sampling yields the following result:

[0151] (1)Start→C1;

[0152] (2) C1→C2;

[0153] (3) C2→C3;

[0154] (4) C3 → End;

[0155] Correspondingly, the generated dialogue path is:

[0156] Start->(C1)->(C2)->(C3)->End.

[0157] In addition, in this scheme, a certain number of dialogue paths generated in step four (i.e., a small batch of dialogue paths) can be manually labeled to determine whether the generated dialogue paths are qualified, thus obtaining training data. A classification model is trained based on the obtained training data, and subsequently, rejection sampling can be performed on dialogue paths generated by Monte Carlo simulation based on this model to enhance the quality of the generated dialogue. The labeling operation can be performed on at least one of the simulated dialogue paths to obtain training data. A classification model is then trained based on the training data. "Determining whether the generated dialogue paths are qualified" primarily involves judging from the logical rationality of the data, checking for inconsistencies or logical confusion. If these issues are present, the path is considered unqualified; otherwise, it is considered qualified. "Obtaining training data" can include: forming data pairs between the dialogue paths and the results of manual labeling to serve as training data; the results of manual labeling can be converted into corresponding values, such as converting qualified to 1 and unqualified to 0, etc., without limitation. Specifically, this part of the scheme may include the following:

[0158] a. Simulate and generate dialogue paths based on the Monte Carlo method;

[0159] b. The labeled data is obtained through manual labeling;

[0160] c. Train a classification model M based on labeled data;

[0161] d. The new dialogue paths generated by Monte Carlo method simulation (corresponding to the above-mentioned simulated dialogue paths) are scored using the classification model M to obtain a score (such as [0, 1]); among them, the classification model M can be a binary classification model, and the binary classification model structure usually passes through a sigmod function at the end, ensuring that the output will be within [0, 1]; in this solution, the higher the score, the more likely it can represent the qualified degree of the new dialogue path. Then, a random number p uniformly distributed on [0, 1] is generated (here is uniform distribution sampling, that is, directly generate a random number within [0, 1]). If p < score, the new dialogue path is accepted, otherwise it is rejected; doing so can make the qualified rate of the selected generated data higher in a statistical sense. This part of the operation can correspond to the above-mentioned screening of the simulated dialogue path to obtain the target dialogue path; the screening of the simulated dialogue path to obtain the target dialogue path includes: using the classification model to score the simulated dialogue path to obtain a scoring result; according to the scoring result, screening the simulated dialogue path to obtain the target dialogue path.

[0162] Step Five: Backfill the text;

[0163] In this solution, each label C i can correspondingly store multiple dialogue samples, corresponding to the above-mentioned one clustering label corresponding to at least one dialogue sample, for example, as follows:

[0164] where s ik represents the k-th dialogue sample of the clustering label C i ;

[0165] After that, in the generated dialogue paths, for each C i randomly extract corresponding text for filling; corresponding to the above-mentioned obtaining simulated dialogue data according to the target dialogue path; the obtaining simulated dialogue data according to the target dialogue path includes: performing a dialogue sample random extraction operation on each clustering label in the target dialogue path to obtain target dialogue samples; obtaining simulated dialogue data according to the target dialogue samples; for example, the extraction results are as follows:

[0166] Specifically, for cluster label C1, the j-th dialogue sample is extracted from the samples corresponding to cluster label C1; for cluster label C2, the k-th dialogue sample is extracted from the samples corresponding to cluster label C2; and for cluster label C3, the m-th dialogue sample is extracted from the samples corresponding to cluster label C3. In this way, a complete simulated dialogue is generated, i.e., simulated dialogue data is obtained. Alternatively, for a target dialogue path, any sample from the dialogue samples corresponding to each state (i.e., each node in the path) is used as the dialogue text for that state, thereby forming a simulated dialogue. In this scheme, this method can be used to generate at least one simulated dialogue for a target dialogue path, obtaining simulated dialogue data.

[0167] Step Six: Generalized Text Enhancement;

[0168] To increase data diversity, LLM can be used to rewrite the text selected in step five (i.e., the backfilled text), appropriately correcting any errors in the translated text and enhancing the diversity of the dialogue content; corresponding to the above-mentioned use of LLM, the simulated dialogue data can be adjusted to obtain the final simulated dialogue data; for example:

[0169] Original sentence:

[0170] "Hello, we have a newly launched health insurance product. Would you be interested in learning more?"

[0171] • LLM rewritten version:

[0172] "Sir, would you like to learn about our latest health insurance? It can provide you with better protection."

[0173] As shown above, LLM adjustments can make the generated data more natural and avoid repetitive patterns.

[0174] It should be noted that the values ​​of variables such as i, j, k, n, and m in different positions in the above content can be the same or different (for example, the values ​​of i in the first occurrence and i in the second occurrence can be the same or different), and no restriction is imposed here.

[0175] Based on the above, this plan involves the following:

[0176] 1) After converting the dialogue data into labels (i.e. cluster labels C), the dialogue path is modeled by the transition probability matrix, that is, a simulated dialogue path is generated; specifically, the Monte Carlo method can be used to generate a simulated dialogue path, and then real dialogue content can be backfilled into it.

[0177] 2) Supports manual intervention to adjust the dialogue transition probability (i.e., manually modify matrix P) to enhance data samples in low-frequency scenarios (such as special objection handling and rare strategies), and even add non-existent process paths, effectively improving data diversity.

[0178] Based on the above, this solution has the following beneficial effects:

[0179] 1) Data-driven and structurally controllable: By using real data, it can be ensured that the generated dialogue closely reflects the actual business logic.

[0180] 2) By adjusting the transition matrix, a specific path can be artificially guided to increase the amount of data in long-tail problems.

[0181] 3) Supports adding processes that are not present in the original data, such as adding a "verify customer identity" step to the original marketing task. That is, adding rows and / or columns to the matrix P above.

[0182] 4) Based on real dialogue text, combined with large models for text generalization, resulting in fewer data errors and greater diversity.

[0183] In summary, compared to traditional methods that directly generate text based on large models, this approach has the following advantages:

[0184] 1) Strong data-driven: Clustering is performed using real telemarketing data to ensure that the generated data structure conforms to the actual scenario.

[0185] 2) High controllability: The transition matrix can be manually adjusted (i.e., "modify matrix P") to guide specific dialogue strategies and improve data quality.

[0186] 3) Long-tail problem optimization: The scarce dialogue paths in the data can be artificially constructed (i.e., "modify matrix P") to enhance the model's generalization ability to low-frequency scenarios.

[0187] 4) High scalability: It can be combined with LLM for reference-based text generation (i.e., using LLM to adjust the text backfilled in step five above), further improving the diversity and fluency of the data, without causing a large number of illusions.

[0188] Furthermore, this solution can also train an AI model based on the simulated dialogue obtained above, and use this model to output dialogue data corresponding to user input; it can correspond to receiving user input as described above; using the first model, the dialogue data corresponding to the user input is obtained; wherein, the first model is based on the above... Figure 1 The simulated dialogue data generated by the corresponding dialogue data generation method is used for training.

[0189] This application also provides a dialogue data generation device, such as... Figure 3 As shown, it includes:

[0190] The first acquisition module 31 is used to acquire at least one sample dialogue path based on the original voice dialogue data; each clustering label contained in the sample dialogue path corresponds to a dialogue segment category.

[0191] The first processing module 32 is used to obtain the transition probabilities between clustering tags in the sample dialogue path and form a dialogue transition matrix.

[0192] The first generation module 33 is used to generate a simulated dialogue path based on the dialogue transition matrix;

[0193] The first filtering module 34 is used to filter the simulated dialogue path to obtain the target dialogue path;

[0194] The second processing module 35 is used to obtain simulated dialogue data based on the target dialogue path.

[0195] The dialogue data generation device provided in this application obtains at least one sample dialogue path based on the original voice dialogue data; each cluster label contained in the sample dialogue path corresponds to a dialogue segment category; the transition probabilities between the cluster labels in the sample dialogue path are obtained to form a dialogue transition matrix; a simulated dialogue path is generated based on the dialogue transition matrix; the simulated dialogue path is filtered to obtain a target dialogue path; and simulated dialogue data is obtained based on the target dialogue path. This device supports generating simulated dialogue data based on real dialogue text (i.e., original voice dialogue data), ensuring that the generated data structure conforms to the actual scenario, avoiding data inconsistency caused by a lack of modeling of real dialogue logic, and avoiding data that does not conform to actual interaction logic. This improves the accuracy of the generated dialogue data and effectively solves the problem of low accuracy in existing methods for obtaining dialogue data.

[0196] The step of obtaining at least one sample dialogue path based on the original voice dialogue data includes: converting the original voice dialogue data into original dialogue text; the original dialogue text includes at least one dialogue text; obtaining the sentence vector corresponding to each dialogue text in the original dialogue text; and forming the sample dialogue path corresponding to each dialogue text based on the sentence vector corresponding to each dialogue text.

[0197] In this embodiment of the application, the clustering labels in the sample dialogue path include: a first clustering label and a second clustering label; wherein, obtaining the transition probabilities between the clustering labels in the sample dialogue path to form a dialogue transition matrix includes: obtaining the initial probability of the first clustering label transitioning to the second clustering label; if the initial probability is less than a threshold, using a first value as the transition probability of the first clustering label transitioning to the second clustering label; if the initial probability is greater than or equal to the threshold, using the initial probability as the transition probability of the first clustering label transitioning to the second clustering label; and forming a dialogue transition matrix based on the transition probabilities.

[0198] Furthermore, the dialogue data generation device further includes: a third processing module, used to perform annotation operations on at least one of the simulated dialogue paths to obtain training data; a first training module, used to train a classification model based on the training data; wherein, the step of filtering the simulated dialogue paths to obtain target dialogue paths includes: using the classification model to score the simulated dialogue paths to obtain scoring results; and filtering the simulated dialogue paths based on the scoring results to obtain target dialogue paths.

[0199] Wherein, one clustering label corresponds to at least one dialogue sample; wherein, obtaining simulated dialogue data according to the target dialogue path includes: performing a dialogue sample random sampling operation on each clustering label in the target dialogue path to obtain a target dialogue sample; and obtaining simulated dialogue data according to the target dialogue sample.

[0200] Furthermore, the dialogue data generation device further includes: a first adjustment module, used to adjust the simulated dialogue data using LLM to obtain final simulated dialogue data; and / or, the dialogue data generation device further includes: a second adjustment module, used to adjust the dialogue transition matrix to obtain a final dialogue transition matrix; wherein, generating a simulated dialogue path based on the dialogue transition matrix includes: generating a simulated dialogue path based on the final dialogue transition matrix.

[0201] Among them, the above Figure 1 The implementation embodiments of the corresponding dialogue data generation methods are all applicable to the embodiments of the dialogue data generation device and can achieve the same technical effect.

[0202] This application also provides a dialogue data generation device, such as... Figure 4 As shown, it includes:

[0203] The first receiving module 41 is used to receive user input;

[0204] The second acquisition module 42 is used to acquire dialogue data corresponding to the user input using the first model; wherein, the first model is a model that utilizes... Figure 1 The simulated dialogue data generated by the corresponding dialogue data generation method is used for training.

[0205] The dialogue data generation device provided in this application embodiment receives user input and uses a first model to obtain dialogue data corresponding to the user input; wherein, the first model is a method using... Figure 1 The corresponding dialogue data generation method generates simulated dialogue data for training; it can support the acquisition of dialogue data using a first model with high accuracy, so as to ensure the acquisition of high-quality dialogue data and solve the problem of low accuracy in existing dialogue data acquisition schemes.

[0206] Among them, the above Figure 2 The implementation embodiments of the corresponding dialogue data generation methods are all applicable to the embodiments of the dialogue data generation device and can achieve the same technical effect.

[0207] This application also provides a dialogue data generation device, including a memory, a processor, and a program stored in the memory and executable on the processor; when the processor executes the program, it implements the dialogue data generation method described above.

[0208] The implementation embodiments of the above-described dialogue data generation method are all applicable to the embodiments of the dialogue data generation device, and can achieve the same technical effect.

[0209] This application also provides a readable storage medium storing a program that, when executed by a processor, implements the steps in the above-described dialogue data generation method.

[0210] The implementation embodiments of the above-described dialogue data generation method are all applicable to the embodiments of the readable storage medium and can achieve the same technical effect.

[0211] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described method embodiment for generating dialogue data, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0212] It should be noted that many of the functional components described in this specification are referred to as modules in order to more specifically emphasize the independence of their implementation.

[0213] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different bits, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0214] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable type of data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0215] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0216] The above describes the preferred embodiments of this application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for generating dialogue data, characterized in that, include: Based on the original voice dialogue data, obtain at least one sample dialogue path; Each clustering label contained in the sample dialogue path corresponds to a dialogue segment category; Obtain the transition probabilities between clustered tags in the sample dialogue path to form a dialogue transition matrix; Based on the dialogue transition matrix, a simulated dialogue path is generated; The simulated dialogue path is filtered to obtain the target dialogue path; Based on the target dialogue path, simulated dialogue data is obtained.

2. The dialogue data generation method according to claim 1, characterized in that, The step of obtaining at least one sample dialogue path based on the original voice dialogue data includes: Convert the raw voice dialogue data into raw dialogue text; the raw dialogue text includes at least one dialogue text. Obtain the sentence vector corresponding to each dialogue text in the original dialogue text; Based on the sentence vectors corresponding to each dialogue text, sample dialogue paths corresponding to each dialogue text are formed.

3. The dialogue data generation method according to claim 1, characterized in that, The clustering labels in the sample dialogue path include: a first clustering label and a second clustering label; The step of obtaining the transition probabilities between clustered labels in the sample dialogue path to form a dialogue transition matrix includes: Obtain the initial probability of the first cluster label transitioning to the second cluster label; If the initial probability is less than the threshold, the first value is used as the transition probability of the first cluster label to the second cluster label; If the initial probability is greater than or equal to the threshold, the initial probability is used as the transition probability for the first cluster label to the second cluster label; Based on the aforementioned transition probabilities, a dialogue transition matrix is ​​formed.

4. The dialogue data generation method according to claim 1, characterized in that, Also includes: For at least one of the simulated dialogue paths, an annotation operation is performed to obtain training data; Based on the training data, a classification model is trained. The step of filtering the simulated dialogue path to obtain the target dialogue path includes: The simulated dialogue path is scored using the classification model to obtain the scoring results; Based on the scoring results, the simulated dialogue paths are filtered to obtain the target dialogue path.

5. The dialogue data generation method according to claim 1, characterized in that, One clustering label corresponds to at least one dialogue sample; The step of obtaining simulated dialogue data based on the target dialogue path includes: For each cluster label in the target dialogue path, a dialogue sample random extraction operation is performed to obtain the target dialogue sample; Based on the target dialogue sample, simulated dialogue data is obtained.

6. The dialogue data generation method according to claim 1, characterized in that, The dialogue data generation method further includes: The simulated dialogue data is adjusted using a large language model to obtain the final simulated dialogue data. And / or, the dialogue data generation method further includes: The dialogue transition matrix is ​​adjusted to obtain the final dialogue transition matrix; The step of generating a simulated dialogue path based on the dialogue transition matrix includes: Based on the final dialogue transition matrix, a simulated dialogue path is generated.

7. A method for generating dialogue data, characterized in that, include: Receive user input; Using a first model, dialogue data corresponding to the user input is obtained; wherein, the first model is trained using simulated dialogue data generated by the dialogue data generation method according to any one of claims 1 to 6.

8. A dialogue data generation device, characterized in that, include: The first acquisition module is used to acquire at least one sample dialogue path based on the original voice dialogue data. Each clustering label contained in the sample dialogue path corresponds to a dialogue segment category; The first processing module is used to obtain the transition probabilities between clustering labels in the sample dialogue path and form a dialogue transition matrix. The first generation module is used to generate a simulated dialogue path based on the dialogue transition matrix; The first filtering module is used to filter the simulated dialogue path to obtain the target dialogue path; The second processing module is used to obtain simulated dialogue data based on the target dialogue path.

9. A dialogue data generation device, characterized in that, include: The first receiving module is used to receive user input; The second acquisition module is used to acquire dialogue data corresponding to the user input using the first model; wherein the first model is trained using simulated dialogue data generated by the dialogue data generation method according to any one of claims 1 to 6.

10. A dialogue data generation device, comprising a memory, a processor, and a program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the dialogue data generation method as described in any one of claims 1 to 7.

11. A readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the dialogue data generation method as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the dialogue data generation method as described in any one of claims 1 to 7.

Images