Training data generation method, device, and program product

Abstract

This application provides a training data generation method, device, and program product. The training data generation method includes: generating query text corresponding to each intent; generating response text corresponding to each query text; determining related intents among the intents; fusing the target query text corresponding to each related intent and the target response text corresponding to each target query text to generate fused query texts and fused response texts corresponding to each fused query text; generating training data based on the query texts, the response texts, the fused query texts, and the fused response texts corresponding to each fused query text; and using the training data to train a question-answering model. This application can improve the accuracy of response content in multi-intent scenarios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training data generation method, device, and program product. Background Technology

[0002] Against the backdrop of rapid development in artificial intelligence technology, deep learning-based models have become the core technological carrier for realizing complex human-computer interaction. Through massive parameter modeling and natural language processing technology, these models demonstrate human-like understanding and execution capabilities in fields such as smart office, intelligent bidding, and data analysis.

[0003] However, in actual enterprise privatization deployments, when user input involves complex requests with multiple intents (such as "query department A's budget and compare it with last year's data"), existing models struggle to accurately deconstruct the intent hierarchy, generate a reasonable execution path, and provide accurate response text. Summary of the Invention

[0004] This application provides a training data generation method, device, and program product to improve the accuracy of response content in multi-intent scenarios.

[0005] According to a first aspect of the embodiments of this application, a training data generation method is provided, comprising:

[0006] Generate the query text corresponding to each intent;

[0007] Generate the corresponding response text for each of the query texts;

[0008] Determine the associated intents among the various stated intents;

[0009] Based on the target query text corresponding to each of the aforementioned associated intents and the target response text corresponding to each of the aforementioned target query texts, the query texts and response texts corresponding to each of the aforementioned target query texts are fused together to generate fused query texts and fused response texts corresponding to each of the aforementioned fused query texts.

[0010] Training data is generated based on each of the query texts, the corresponding response texts for each of the query texts, the fused query texts, and the corresponding fused response texts for each of the fused query texts; the training data is used to train the question-answering model.

[0011] Optionally, determining the associated intent among the various intents includes:

[0012] Extract each intent from each sample query text and obtain the co-occurrence probability of each intent in the same sample query text;

[0013] Based on the co-occurrence probability of each of the stated intents in the same sample query text, related intents are determined from each of the stated intents.

[0014] Optionally, the step of fusing the target query text corresponding to each of the associated intents and the target response text corresponding to each of the target query texts to generate fused query texts and fused response texts corresponding to each of the fused query texts includes:

[0015] The target query texts corresponding to each of the aforementioned associated intents are fused together to generate various fused query texts.

[0016] Based on each of the fused query texts and the target response texts corresponding to each of the target query texts, a Monte Carlo tree search is used to obtain the response planning path corresponding to each of the fused query texts.

[0017] Based on the response planning path corresponding to each of the fused query texts, the fused response text corresponding to each of the fused query texts is generated.

[0018] Optionally, generating the query text corresponding to each intent includes:

[0019] Extract each intent from each sample query text, as well as the corresponding associated parameter values ​​for each intent;

[0020] Based on each intent, the associated parameter values ​​corresponding to each intent, and each query text template, a query text corresponding to each intent is generated; wherein, the query text template includes sub-templates at each level.

[0021] Optionally, generating the response text corresponding to each of the query texts includes:

[0022] Obtain the interface parameters corresponding to each intent; wherein, the interface parameters include each parameter field corresponding to the interface and the natural language corresponding to each parameter field;

[0023] Based on the interface parameters corresponding to each intent and the query text corresponding to each intent, a response text corresponding to each query text is generated.

[0024] Optionally, generating response text corresponding to each query text based on the interface parameters corresponding to each intent and the query text corresponding to each intent includes:

[0025] Based on the interface parameters corresponding to each intent and the query text corresponding to each intent, generate the initial response text corresponding to each query text.

[0026] Perform format validation on the initial response text corresponding to each of the query texts to obtain the initial response texts with format errors and the initial response texts without format errors;

[0027] Based on the target query text corresponding to the initial response text with the formatting error, and the interface parameters corresponding to the intent of the target query text, an updated response text corresponding to the target query text is generated;

[0028] Based on the initial response text that does not contain any formatting errors, and the updated response text corresponding to the target query text, a response text corresponding to each of the query texts is generated.

[0029] Optionally, generating training data based on each of the query texts, the corresponding response texts for each of the query texts, the fused query texts, and the corresponding fused response texts for each of the fused query texts includes:

[0030] A multi-dimensional quality inspection is performed on each of the query texts, the corresponding response texts for each of the query texts, the fused query texts, and the corresponding fused response texts for each of the fused query texts to obtain text pairs that pass the multi-dimensional quality inspection and text pairs that fail the quality inspection in at least one dimension.

[0031] The text pairs that fail the quality inspection of at least one dimension are regenerated to obtain updated text pairs;

[0032] Training data is generated based on the multi-dimensional quality-checked text pairs and the updated text pairs.

[0033] The text pair includes a query text and a corresponding response text, or a merged query text and a corresponding merged response text.

[0034] Optionally, the method further includes:

[0035] The user's query text is input into the question-and-answer model to obtain the response text corresponding to the user's query text;

[0036] The user query text and the corresponding reply text are divided into high-quality text pairs and low-quality text pairs; wherein, the high-quality text pair includes a high-quality user query text and the corresponding reply text, and the low-quality text pair includes a low-quality user query text and the corresponding reply text.

[0037] The inferior text pairs are optimized to obtain optimized text pairs;

[0038] Based on the high-quality text pairs and the optimized text pairs, incremental training data is generated, and the question-answering model is trained based on the incremental training data.

[0039] According to a second aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor;

[0040] The memory is connected to the processor and is used to store programs;

[0041] The processor is used to implement the training data generation method as described in the first aspect by running a program in the memory.

[0042] According to a third aspect of the embodiments of this application, a computer program product is provided, including computer program instructions that, when executed by a processor, cause the processor to perform the training data generation method as described in the first aspect.

[0043] In this application, query texts corresponding to each intent are generated, and response texts corresponding to each query text are generated. Related intents within each intent are determined. The target query texts corresponding to each related intent and the target response texts corresponding to each target query text are fused to generate fused query texts and fused response texts corresponding to each fused query text. The fused query texts are obtained by fusing the target query texts corresponding to each related intent, and each fused query text includes multiple related intents. The fused response texts are obtained by fusing the target response texts corresponding to each target query text, and each fused response text includes response texts corresponding to multiple related intents. Training data is generated based on each query text, each query text's corresponding response text, each fused query text, and each fused response text corresponding to each fused query text. The training data includes fused query texts and their corresponding fused response texts. This training data is used to train a question-answering model, enabling training for scenarios where query texts contain multiple related intents, thereby improving the accuracy of response content in multi-intent scenarios. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a training data generation method provided in an embodiment of this application;

[0046] Figure 2 This is a flowchart illustrating step 101 provided in an embodiment of this application;

[0047] Figure 3 This is a flowchart illustrating step 102 provided in an embodiment of this application;

[0048] Figure 4 This is a flowchart illustrating step 302 provided in an embodiment of this application;

[0049] Figure 5 This is a flowchart illustrating step 103 provided in an embodiment of this application;

[0050] Figure 6 This is a flowchart illustrating step 104 provided in an embodiment of this application;

[0051] Figure 7 This is a flowchart illustrating step 105 provided in an embodiment of this application;

[0052] Figure 8 This is a flowchart illustrating a training data generation method provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the structure of a training data generation device provided in the embodiments of this application;

[0054] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] Against the backdrop of rapid development in artificial intelligence technology, deep learning-based models have become the core technological carrier for realizing complex human-computer interaction. Through massive parameter modeling and natural language processing technology, these models demonstrate human-like understanding and execution capabilities in fields such as smart office, intelligent bidding, and data analysis.

[0056] However, in actual enterprise privatization deployments, when user input involves complex requests with multiple intents (such as "query department A's budget and compare it with last year's data"), existing models struggle to accurately deconstruct the intent hierarchy, generate a reasonable execution path, and provide accurate response text.

[0057] Current industry solutions mainly rely on manually labeled static training datasets, with manual labeling teams (e.g., 50-100 people) completing intent classification, slot labeling, and dialogue flow design. Training corpora are generated using limited templates (e.g., about 200-300 templates per scene), and then supervised fine-tuning (SFT) is performed on a large model. Manually constructed instruction-response pairing data are used, which has industry pain points such as long data iteration cycles (average 3-6 months), high scene coverage costs (50+ person-days per scene), and lagging knowledge updates.

[0058] Moreover, the accuracy rate of multi-person multi-intent composite annotation is less than 80% (benchmark tests show that in the "budget query + comparative analysis" type of request, 20% of the cases have missed or incorrect manual intent judgments).

[0059] For example, the process for multi-intent composite annotation is as follows: The user inputs questions 1, 2, 3, 4, and N into the business system. The business system includes a question-answering model and is configured with multiple interfaces, each corresponding to an intent. The user manually determines the corresponding interfaces for each question in the business system: Question 1 corresponds to interfaces B and C, Question 2 to interfaces A and D, Question 3 to interfaces B and C, Question 4 to interfaces C and N, and Question N to interface C. Question 2: "I'm going to Beijing for a three-day business trip tomorrow." Question 2 corresponds to interfaces A and D. Interface A corresponds to intent A: "Business trip application," and interface D corresponds to intent D: "Leave application." Intent A: "Business trip application" corresponds to the parameter fields of interface A. Interface A has three parameter fields: Field 1: Applicant, Field 2: Trip time, and Field 3: Trip location. Intent D: The slot corresponding to the leave application is the parameter field of interface D. Interface D has 4 parameter fields: Field 1: Applicant, Field 2: Leave Time, Field 3: Leave Type, Field 4: Leave Reason. The enumeration values ​​(i.e., parameter values) in the user input question are extracted based on the parameter fields of the interface through manual judgment. Intent A: Business Trip Application, Field 1: Applicant, Applicant: I, uniformly represented as Zhang San; Field 2: Business Trip Time, Business Trip Time: Tomorrow, Three Days, uniformly represented as March 9, 2025 - March 11, 2025; Field 3: Business Trip Location, Business Trip Location: Beijing, uniformly represented as 001-Beijing. Intent D: Leave application, Field 1: Applicant, Applicant: I, uniformly named Zhang San; Field 2: Leave time, Business trip time: Tomorrow, three days, uniformly named March 9, 2025 - March 11, 2025; Field 3: Leave type, Leave type: Business trip, uniformly named 0001-Business trip; Field 4: Reason for leave, Business trip to Beijing for three days.

[0060] Therefore, in order to improve the accuracy of response content in multi-intent scenarios, this application provides a training data generation method, device, and program product.

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] Exemplary Implementation Environment

[0063] The training data generation method according to the embodiments of this application can be executed by electronic devices such as terminal devices or servers. The terminal device can be a user device, mobile device, computing device, wearable device, etc., and the server can be an independent physical server, a server cluster composed of multiple physical servers, or a cloud server capable of cloud computing. This method can be implemented by a processor calling computer-readable program instructions stored in memory.

[0064] Exemplary methods

[0065] Please see Figure 1 In one exemplary embodiment, a method for generating training data is provided. For example... Figure 1 As shown, the process of training data generation mainly includes:

[0066] Step 101: Generate the query text corresponding to each intent.

[0067] In some embodiments, such as Figure 2 As shown, step 101 includes:

[0068] Step 201: Extract each intent from each sample query text, as well as the corresponding associated parameter values ​​for each intent.

[0069] In an exemplary embodiment, the business system includes a question-answering model. The business system is configured with multiple interfaces, which can use a domain rule engine to semantically decompose the sample query text input to the business system, extracting various intents (e.g., "order query," "fault reporting") and their corresponding associated parameter values ​​(e.g., time, user account). The sample query text input to the business system can be sample query text within a target scenario, such as an office, park, or production application.

[0070] In the exemplary embodiments, there is a correspondence between intents and interfaces. One intent may correspond to one interface, or one intent may correspond to multiple interfaces. This application does not limit this.

[0071] In the exemplary embodiment, the associated parameter value refers to the parameter value of the parameter field of the interface corresponding to the intent. It can be one associated parameter value for one intent or multiple associated parameter values ​​for one intent. This application does not limit this.

[0072] Step 202: Based on each intent, the associated parameter values ​​corresponding to each intent, and each query text template, generate the query text corresponding to each intent.

[0073] The query text template includes sub-templates at various levels.

[0074] In an exemplary embodiment, the sub-templates at each level include basic sentence templates (declarative or interrogative sentences), parameter placeholders (${parameters}), and intent combination rules. The sub-templates at each level may also include other sub-templates, and this application does not limit this. The sub-templates at each level can also support enterprise-customized extensions.

[0075] In an exemplary embodiment, the sub-templates at each level can adopt a tree-like storage structure.

[0076] In the exemplary embodiment, sub-templates at each level can be dynamically matched to obtain multiple query text templates.

[0077] In an exemplary embodiment, the intent combination rule can refer to which intents can be combined, for example, requesting leave and traveling can be combined.

[0078] In an exemplary embodiment, an intent and its corresponding associated parameter values ​​can be combined with multiple query text templates to obtain multiple query texts corresponding to the intent, thereby increasing the number and diversity of query texts.

[0079] In an exemplary embodiment, step 202 may include: inputting each intent, the associated parameter values ​​corresponding to each intent, and each query text template into the query text generation model to obtain the query text corresponding to each intent.

[0080] In the exemplary embodiment, the query text generation model can be trained using a fine-tuned generative large model combined with a reinforcement learning strategy. Furthermore, adversarial training of the query text generation model can reduce semantic redundancy and logical contradictions in the query texts generated by different intents.

[0081] Currently, the cost of building training data for a single business scenario exceeds 50 person-days (including annotation, validation, and rule configuration), and the efficiency of generating training data is low.

[0082] In this application, each intent and its corresponding associated parameter value are extracted from each sample query text. Based on each intent, its corresponding associated parameter value, and each query text template, query text corresponding to each intent is generated, realizing dynamic matching between intent and query text template. The query text template includes sub-templates at various levels, which can generate a larger number of query texts and improve the diversity of query texts. Compared with manually constructing training data, it significantly improves the efficiency of training data generation. Moreover, the dynamic matching between intent and query text template, and the inclusion of sub-templates at various levels in the query text template, overcomes the semantic rigidity problem of traditional static templates, and the diversity index is significantly improved.

[0083] In some embodiments, step 101 includes: generating query text prompts based on each intent, inputting the query text prompts into the large model, and obtaining the query text corresponding to each intent; wherein, the query text prompts are used to instruct the large model to generate the query text corresponding to each intent. Step 101 can also be implemented in other ways, which are not listed here. Step 102: generating the response text corresponding to each query text.

[0084] In some embodiments, such as Figure 3 As shown, step 102 includes:

[0085] Step 301: Obtain the interface parameters corresponding to each intent.

[0086] The interface parameters include the various parameter fields corresponding to the interface and the natural language corresponding to each parameter field.

[0087] In the exemplary embodiment, the business system includes a question-answering model. The business system is configured with multiple interfaces. This can be achieved by parsing the business system API (Application Programming Interface) documentation. The API documentation includes the parameter fields corresponding to each interface and their respective natural language representations. Only the parameter fields and their corresponding natural language representations need to be extracted from the API documentation, and a mapping relationship needs to be established between these relationships. Subsequently, the corresponding parameter fields and their corresponding natural language representations can be directly retrieved based on this mapping relationship. For example, "user_id" → "user account," where the parameter field is "user_id" and the natural language representation is "user account."

[0088] In the exemplary embodiments, there is a correspondence between intents and interfaces. One intent may correspond to one interface, or one intent may correspond to multiple interfaces. This application does not limit this.

[0089] Step 302: Based on the interface parameters corresponding to each intent and the query text corresponding to each intent, generate the response text corresponding to each query text.

[0090] In an exemplary embodiment, the interface parameters corresponding to each intent and the query text corresponding to each intent are input into the response text generation model to obtain the response text corresponding to each query text. This can be based on Retrieval-augmented Generation (RAG) technology, combining a knowledge base, the interface parameters corresponding to each intent, and the query text corresponding to each intent to generate the response text corresponding to each query text. The response text includes both structured data and natural language. For example, the structured data can be in JSON (JavaScript Object Notation) format or XML (Extensible Markup Language) format. The structured data in the response text can refer to the parameter fields and parameter values ​​corresponding to the interface, and the natural language in the response text can refer to the natural language corresponding to the parameter fields and parameter values ​​corresponding to the interface.

[0091] Currently, when faced with industry-specific interface specifications (such as order coding rules in ERP (Enterprise Resource Planning) systems), the lack of domain knowledge in general models leads to a significant decrease in parameter extraction accuracy. Using rule engines (such as regular expressions) to extract basic parameters requires pre-defining an industry terminology library (such as budget coding rules in finance, steel, and oil and gas), resulting in an error rate as high as 15% for identifying industry-specific interface parameters.

[0092] In this application, the interface parameters corresponding to each intent are obtained. These interface parameters include the parameter fields corresponding to the interface and the natural language corresponding to each parameter field. The interface parameters achieve a precise mapping between the parameter fields and the natural language. Based on the interface parameters corresponding to each intent and the query text corresponding to each intent, the response text corresponding to each query text is generated. This can accurately generate multimodal response text containing structured data and natural language, achieving a precise mapping between structured data and natural language, improving the accuracy of the response text, and thus improving the accuracy of the training data.

[0093] In other embodiments, step 102 includes: identifying slots corresponding to each intent from each sample query text; and generating response texts corresponding to each query text based on the query texts corresponding to each intent and the slots corresponding to each intent. Step 102 can also be implemented in other ways, which are not listed here.

[0094] In some embodiments, such as Figure 4 As shown, step 302 includes:

[0095] Step 401: Based on the interface parameters corresponding to each intent and the query text corresponding to each intent, generate the initial response text corresponding to each query text.

[0096] Step 402: Perform format validation on the initial response text corresponding to each query text to obtain the initial response text with format errors and the initial response text without format errors.

[0097] In the exemplary embodiment, format validation of the initial response text corresponding to each query text can be performed using a lightweight rule model.

[0098] In the exemplary embodiment, the categories of formatting errors may include missing fields, type errors, and other categories, which are not limited in this application. For example, if the question asks for a time period (year, month, day, hour, minute), the generated initial response text is missing the "hour," which is a missing field; if the question asks for a time period, the generated initial response text is a specific time, which is a type error.

[0099] Step 403: Based on the target query text corresponding to the initial response text with a formatting error, and the interface parameters corresponding to the intent of the target query text, generate the updated response text corresponding to the target query text.

[0100] In an exemplary embodiment, the updated response text corresponding to the target query text can be generated directly based on the target query text corresponding to the initial response text with a formatting error and the interface parameters corresponding to the intent of the target query text; alternatively, the updated response text corresponding to the target query text can be generated based on the target query text corresponding to the initial response text with a formatting error, the interface parameters corresponding to the intent of the target query text, and the formatting error category of the initial response text with a formatting error, so that the updated response text does not contain errors of this formatting error category as much as possible.

[0101] Step 404: Based on the initial response text that does not contain any formatting errors and the updated response text corresponding to the target query text, generate the response text corresponding to each query text.

[0102] It can regenerate initial response text with formatting errors to obtain updated response text, thereby improving the accuracy of the response text and thus improving the accuracy of the training data.

[0103] In other embodiments, step 302 includes: directly inputting the interface parameters corresponding to each intent and the query text corresponding to each intent into the response text generation model to obtain the response text corresponding to each query text. Step 302 can also be implemented in other ways, which are not listed here. Step 103: Determine the associated intents among the various intents.

[0104] In some embodiments, such as Figure 5 As shown, step 103 includes:

[0105] Step 501: Extract each intent from each sample query text and obtain the co-occurrence probability of each intent in the same sample query text.

[0106] In an exemplary embodiment, the business system includes a question-answering model that extracts various intents from a large number of sample query texts input into the business system, constructs an intent association network based on a probabilistic graphical model, and quantifies the co-occurrence probability of various intents in the same sample query text. Here, the probabilistic graphical model refers to using a graph to represent the joint probability distribution of variables related to the model. The co-occurrence probability of various intents in the same sample query text refers to the probability that various intents appear together in the same sample query text. For example, the co-occurrence probability of the intents "return application" and "logistics inquiry" is 0.76.

[0107] Step 502: Based on the co-occurrence probability of each intent in the same sample query text, determine the associated intents from each intent.

[0108] In an exemplary embodiment, intents with a co-occurrence probability greater than a probability threshold in the same sample query text can be identified as associated intents.

[0109] It can accurately extract related intents from various intents based on the co-occurrence probability of each intent in the same sample query text.

[0110] In other embodiments, step 103 includes: finding related intents among various intents based on a pre-built knowledge graph. Step 103 can also be implemented in other ways, which are not listed here.

[0111] Step 104: Based on the target query text corresponding to each associated intent and the target response text corresponding to each target query text, merge them to generate each merged query text and the merged response text corresponding to each merged query text.

[0112] In some embodiments, such as Figure 6 As shown, step 104 includes:

[0113] Step 601: Based on the target query text corresponding to each associated intent, merge them to generate each merged query text.

[0114] Step 602: Based on each fused query text and the corresponding target response text for each target query text, a Monte Carlo tree search is used to obtain the response planning path for each fused query text.

[0115] In an exemplary embodiment, based on each fused query text and the target response text corresponding to each target query text, a multi-turn dialogue path can be simulated through Monte Carlo Tree Search (MCTS) to generate a decision flowchart containing branching logic (if-else) and state transitions, which is to obtain the response planning path corresponding to each fused query text.

[0116] In an exemplary embodiment, the target query text corresponding to each associated intent and the target response text corresponding to each target query text can be input into the fusion model. The Monte Carlo tree search is performed through the fusion model to obtain each fused query text and the response planning path corresponding to each fused query text.

[0117] In an exemplary embodiment, a greedy strategy can be used during the Monte Carlo tree search to balance path depth and coverage, ensuring that a single set of data contains 3-5 related intents.

[0118] Step 603: Based on the response planning path corresponding to each fused query text, generate the fused response text corresponding to each fused query text.

[0119] By using Monte Carlo tree search, the corresponding response planning path for each merged query text is obtained, thereby obtaining the merged response text, which improves the accuracy of the merged response text and makes the logic of the merged response text more coherent.

[0120] In other embodiments, step 104 includes: concatenating the target query texts corresponding to each associated intent to obtain fused query texts; and concatenating the target response texts corresponding to any fused query text to obtain a fused response text corresponding to any fused query text. Step 104 can also be implemented in other ways, which are not listed here.

[0121] Step 105: Generate training data based on each query text, the corresponding response text for each query text, each fused query text, and the corresponding fused response text for each fused query text.

[0122] The training data is used to train the question-answering model.

[0123] In some embodiments, such as Figure 7 As shown, step 105 includes:

[0124] Step 701: Perform multi-dimensional quality checks on each query text, the corresponding response text for each query text, each merged query text, and the corresponding merged response text for each merged query text to obtain text pairs that pass the multi-dimensional quality checks and text pairs that fail at least one dimension of the quality checks.

[0125] The text pair includes a query text and the corresponding response text, or a merged query text and the corresponding merged response text.

[0126] In the exemplary embodiment, multi-dimensional quality inspection includes semantic quality inspection, logical quality inspection, and format quality inspection. Semantic quality inspection can detect referential ambiguity (such as "it" being unclear) and intent conflict based on the Bi-LSTM (Bidirectional Long Short-Term Memory) model. Intent conflict can occur when the intent in the fused query text (business trip) conflicts with the intent in the fused response text (booking a hotel for travel). Logical quality inspection can verify temporal inconsistencies (such as "payment" preceding "place order") using a Datalog (a data query language) rule engine. Format quality inspection can match JSON or XPath (XML Path Language) specifications using regular expressions.

[0127] In the exemplary embodiment, semantic quality inspection, logical quality inspection, and format quality inspection can be processed in parallel to achieve parallel quality inspection of semantics, logic, and format, thereby improving the defect detection rate of text pairs and thus improving the accuracy of training data.

[0128] Step 702: Regenerate text pairs that fail quality inspection in at least one dimension to obtain updated text pairs.

[0129] In an exemplary embodiment, the defect type of text pairs that fail quality inspection in at least one dimension can be identified, and then the text pairs that fail quality inspection in at least one dimension can be regenerated according to the defect type to obtain updated text pairs.

[0130] Step 703: Generate training data based on the multi-dimensional quality-checked text pairs and the updated text pairs.

[0131] Multi-dimensional quality checks are performed on each query text, its corresponding response text, the fused query text, and its corresponding fused response text. This results in text pairs that pass the multi-dimensional quality check and those that fail at least one dimension. By improving the defect detection rate of text pairs through multi-dimensional quality checks, the text pairs that fail at least one dimension are regenerated to obtain updated text pairs. Based on the text pairs that pass the multi-dimensional quality check and the updated text pairs, training data is generated. This method can improve the accuracy of training data by regenerating text pairs that fail at least one dimension while improving the defect detection rate of text pairs.

[0132] In some embodiments, step 105 includes: directly determining each query text, the corresponding response text for each query text, each fused query text, and the corresponding fused response text for each fused query text as training data. Step 105 can also be implemented in other ways, which are not listed here.

[0133] In some embodiments, such as Figure 8 As shown, training data generation methods also include:

[0134] Step 801: Input the user's query text into the question-answering model to obtain the response text corresponding to the user's query text.

[0135] In an exemplary embodiment, the user's query text is input into a question-answering model trained with training data to obtain the response text corresponding to the user's query text.

[0136] Step 802: Divide the user query text and the corresponding response text into high-quality text pairs and low-quality text pairs.

[0137] High-quality text pairs include a high-quality user query text and its corresponding response text, while low-quality text pairs include a low-quality user query text and its corresponding response text.

[0138] In an exemplary embodiment, a lightweight BERT (Bidirectional Encoder Representations from Transformers, pre-trained language model) can be used to identify the intent in the user's query text and to divide the user's query text and the corresponding response text into high-quality text pairs and low-quality text pairs.

[0139] In an exemplary embodiment, dividing the user query text and the corresponding response text into high-quality text pairs and low-quality text pairs can be achieved by using multiple preset standards to classify the quality of the user query text and the corresponding response text. For example, the preset standards could be that the prompt words are in English and the user query text is in Chinese, or various other preset standards. This application does not limit this.

[0140] Step 803: Optimize the poor text pairs to obtain optimized text pairs.

[0141] For example, poor text pairs, the corresponding defect types of poor text pairs, and the correct examples corresponding to the defect types of poor text pairs can be input into a large model to optimize the poor text pairs and obtain optimized text pairs.

[0142] Step 804: Generate incremental training data based on high-quality text pairs and optimized text pairs, and train the question answering model based on the incremental training data.

[0143] Current template-based data generation methods struggle to adapt to dynamic changes in business scenarios, leading to unstable agent performance in real-world business flows. This application, however, utilizes an online feedback mechanism to continuously optimize training data, supporting rapid deployment and iteration of the question-answering model in private scenarios. By dynamically optimizing the question-answering model through incremental training data, the cold start time for new scenarios is significantly reduced.

[0144] In an exemplary embodiment, an ES (Elasticsearch, a search and analytics engine) database can also be used to store various versions of the question-answering model during training, supporting backtracking, difference comparison, and A / B testing.

[0145] In summary, this application generates query texts corresponding to each intent, generates response texts corresponding to each query text, determines related intents within each intent, and fuses the target query texts corresponding to each related intent and the target response texts corresponding to each target query text to generate fused query texts and fused response texts corresponding to each fused query text. The fused query texts are obtained by fusing the target query texts corresponding to each related intent, and the fused query texts include multiple related intents. The fused response texts are obtained by fusing the target response texts corresponding to each target query text, and the fused response texts include response texts corresponding to multiple related intents. Based on each query text, each response text corresponding to each query text, each fused query text, and each fused response text corresponding to each fused response text, training data is generated. The training data includes fused query texts and their corresponding fused response texts. The training data is used to train the question-answering model, which can be trained for scenarios where the query text includes multiple related intents, thereby improving the accuracy of the response content in multi-intent scenarios.

[0146] Exemplary device

[0147] Accordingly, embodiments of this application also provide a training data generation apparatus, such as... Figure 9 As shown, the training data generation device includes:

[0148] The first generation unit 901 is used to generate the query text corresponding to each intent;

[0149] The second generation unit 902 is used to generate the response text corresponding to each of the query texts;

[0150] Processing unit 903 is used to determine the associated intent among the various intents;

[0151] The fusion unit 904 is used to fuse the target query text corresponding to each of the associated intents and the target response text corresponding to each of the target query texts to generate fused query texts and fused response texts corresponding to each of the fused query texts.

[0152] The third generation unit 905 is used to generate training data based on each of the query texts, the corresponding response texts for each of the query texts, the fused query texts, and the corresponding fused response texts for each of the fused query texts; the training data is used to train the question answering model.

[0153] Optionally, the processing unit 903 is specifically used for:

[0154] Extract each intent from each sample query text and obtain the co-occurrence probability of each intent in the same sample query text;

[0155] Based on the co-occurrence probability of each of the stated intents in the same sample query text, related intents are determined from each of the stated intents.

[0156] Optionally, the fusion unit 904 is specifically used for:

[0157] The target query texts corresponding to each of the aforementioned associated intents are fused together to generate various fused query texts.

[0158] Based on each of the fused query texts and the target response texts corresponding to each of the target query texts, a Monte Carlo tree search is used to obtain the response planning path corresponding to each of the fused query texts.

[0159] Based on the response planning path corresponding to each of the fused query texts, the fused response text corresponding to each of the fused query texts is generated.

[0160] Optionally, the first generating unit 901 is specifically used for:

[0161] Extract each intent from each sample query text, as well as the corresponding associated parameter values ​​for each intent;

[0162] Based on each intent, the associated parameter values ​​corresponding to each intent, and each query text template, a query text corresponding to each intent is generated; wherein, the query text template includes sub-templates at each level.

[0163] Optionally, the second generating unit 902 includes:

[0164] The acquisition subunit is used to acquire the interface parameters corresponding to each intent; wherein, the interface parameters include each parameter field corresponding to the interface and the natural language corresponding to each parameter field;

[0165] The generation subunit is used to generate the response text corresponding to each query text based on the interface parameters corresponding to each intent and the query text corresponding to each intent.

[0166] Optionally, sub-units are generated, specifically for:

[0167] Based on the interface parameters corresponding to each intent and the query text corresponding to each intent, generate the initial response text corresponding to each query text.

[0168] Perform format validation on the initial response text corresponding to each of the query texts to obtain the initial response texts with format errors and the initial response texts without format errors;

[0169] Based on the target query text corresponding to the initial response text with the formatting error, and the interface parameters corresponding to the intent of the target query text, an updated response text corresponding to the target query text is generated;

[0170] Based on the initial response text that does not contain any formatting errors, and the updated response text corresponding to the target query text, a response text corresponding to each of the query texts is generated.

[0171] Optionally, the third generating unit 905 is specifically used for:

[0172] A multi-dimensional quality inspection is performed on each of the query texts, the corresponding response texts for each of the query texts, the fused query texts, and the corresponding fused response texts for each of the fused query texts to obtain text pairs that pass the multi-dimensional quality inspection and text pairs that fail the quality inspection in at least one dimension.

[0173] The text pairs that fail the quality inspection of at least one dimension are regenerated to obtain updated text pairs;

[0174] Training data is generated based on the multi-dimensional quality-checked text pairs and the updated text pairs.

[0175] The text pair includes a query text and a corresponding response text, or a merged query text and a corresponding merged response text.

[0176] Optionally, the training data generation device further includes an incremental training unit; the incremental training unit is used for:

[0177] The user's query text is input into the question-and-answer model to obtain the response text corresponding to the user's query text;

[0178] The user query text and the corresponding reply text are divided into high-quality text pairs and low-quality text pairs; wherein, the high-quality text pair includes a high-quality user query text and the corresponding reply text, and the low-quality text pair includes a low-quality user query text and the corresponding reply text.

[0179] The inferior text pairs are optimized to obtain optimized text pairs;

[0180] Based on the high-quality text pairs and the optimized text pairs, incremental training data is generated, and the question-answering model is trained based on the incremental training data.

[0181] The training data generation apparatus provided in this embodiment belongs to the same concept as the training data generation method provided in the above embodiments of this application. It can execute the training data generation method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the training data generation method. Technical details not described in detail in this embodiment can be found in the specific processing content of the training data generation method provided in the above embodiments of this application, and will not be repeated here.

[0182] The functions implemented by the first generation unit 901, the second generation unit 902, the processing unit 903, the fusion unit 904 and the third generation unit 905 can be implemented by the same or different processors, and this application embodiment does not limit them.

[0183] It should be understood that the units in the above device can be implemented by a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units in the above device can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0184] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0185] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0186] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0187] Exemplary electronic devices

[0188] One embodiment of this application discloses an electronic device, see [link to relevant documentation] Figure 10 As shown, the device includes:

[0189] Memory 200 and processor 210;

[0190] The memory 200 is connected to the processor 210 and is used to store programs;

[0191] The processor 210 is configured to implement the training data generation method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0192] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0193] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them:

[0194] A bus can include a pathway for transmitting information between various components of a computer system.

[0195] The processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0196] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0197] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0198] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0199] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0200] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0201] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any training data generation method provided in the above embodiments of this application.

[0202] Exemplary computer program products and storage media

[0203] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the training data generation methods according to various embodiments of this application as described in any of the foregoing embodiments of this specification.

[0204] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0205] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor through steps in the training data generation method according to various embodiments of this application described above. Specifically, the following steps can be implemented:

[0206] Step 101: Generate the query text corresponding to each intent;

[0207] Step 102: Generate the corresponding response text for each query text;

[0208] Step 103: Identify the related intents among the various intents;

[0209] Step 104: Based on the target query text corresponding to each associated intent and the target response text corresponding to each target query text, merge them to generate each merged query text and the merged response text corresponding to each merged query text.

[0210] Step 105: Generate training data based on each query text, the corresponding response text for each query text, each fused query text, and the corresponding fused response text for each fused query text; the training data is used to train the question answering model.

[0211] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0212] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0213] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0214] The modules and sub-modules in the apparatus and terminal in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0215] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0216] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0217] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0218] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0219] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0220] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0221] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating training data, characterized in that, include: Generate the query text corresponding to each intent; Generate the corresponding response text for each of the query texts; Determine the associated intents among the various stated intents; The target query texts corresponding to each of the aforementioned related intents are fused together to generate various fused query texts. Based on each of the fused query texts and the target response texts corresponding to each of the target query texts, a Monte Carlo tree search is used to obtain the response planning path corresponding to each of the fused query texts. Based on the response planning path corresponding to each of the fused query texts, generate the fused response text corresponding to each of the fused query texts. Training data is generated based on each of the query texts, the corresponding response texts for each of the query texts, the fused query texts, and the corresponding fused response texts for each of the fused query texts. The training data is used to train the question-answering model.

2. The training data generation method according to claim 1, characterized in that, Determining the associated intent among the various intents includes: Extract each intent from each sample query text and obtain the co-occurrence probability of each intent in the same sample query text; Based on the co-occurrence probability of each of the stated intents in the same sample query text, related intents are determined from each of the stated intents.

3. The training data generation method according to claim 1, characterized in that, The generation of query text corresponding to each intent includes: Extract each intent from each sample query text, as well as the corresponding associated parameter values ​​for each intent; Based on each intent, the associated parameter values ​​corresponding to each intent, and each query text template, a query text corresponding to each intent is generated; wherein, the query text template includes sub-templates at each level.

4. The training data generation method according to claim 1, characterized in that, Generating the response text corresponding to each of the query texts includes: Obtain the interface parameters corresponding to each intent; wherein, the interface parameters include each parameter field corresponding to the interface and the natural language corresponding to each parameter field; Based on the interface parameters corresponding to each intent and the query text corresponding to each intent, a response text corresponding to each query text is generated.

5. The training data generation method according to claim 4, characterized in that, The step of generating response text corresponding to each query text based on the interface parameters corresponding to each intent and the query text corresponding to each intent includes: Based on the interface parameters corresponding to each intent and the query text corresponding to each intent, generate the initial response text corresponding to each query text. Perform format validation on the initial response text corresponding to each of the query texts to obtain the initial response texts with format errors and the initial response texts without format errors; Based on the target query text corresponding to the initial response text with the formatting error, and the interface parameters corresponding to the intent of the target query text, an updated response text corresponding to the target query text is generated; Based on the initial response text that does not contain any formatting errors, and the updated response text corresponding to the target query text, a response text corresponding to each of the query texts is generated.

6. The training data generation method according to claim 1, characterized in that, The step of generating training data based on each of the query texts, the corresponding response texts for each of the query texts, the fused query texts, and the corresponding fused response texts for each of the fused query texts includes: A multi-dimensional quality inspection is performed on each of the query texts, the corresponding response texts for each of the query texts, the fused query texts, and the corresponding fused response texts for each of the fused query texts to obtain text pairs that pass the multi-dimensional quality inspection and text pairs that fail the quality inspection in at least one dimension. Text pairs that fail quality inspection in at least one dimension are regenerated to obtain updated text pairs. Training data is generated based on the multi-dimensional quality-checked text pairs and the updated text pairs. The text pair includes a query text and a corresponding response text, or a merged query text and a corresponding merged response text.

7. The training data generation method according to claim 1, characterized in that, The method further includes: The user's query text is input into the question-and-answer model to obtain the response text corresponding to the user's query text; The user query text and the corresponding reply text are divided into high-quality text pairs and low-quality text pairs; wherein, the high-quality text pair includes a high-quality user query text and the corresponding reply text, and the low-quality text pair includes a low-quality user query text and the corresponding reply text. The inferior text pairs are optimized to obtain optimized text pairs; Based on the high-quality text pairs and the optimized text pairs, incremental training data is generated, and the question-answering model is trained based on the incremental training data.

8. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the training data generation method as described in any one of claims 1 to 7 by running a program in the memory.

9. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to perform the training data generation method as described in any one of claims 1 to 7.

Images