A method, system, apparatus, and medium for assessing the quality of dialogue training.

Abstract

This specification provides a method, system, apparatus, and medium for evaluating the quality of dialogue training. The method includes retrieving target training data and generating a target training library; responding to a target user conducting training based on the target training data in the target training library, the training including multiple rounds of dialogue; for each round of dialogue: acquiring the target user's round-based dialogue information, which includes multiple dialogue sub-data and multiple dialogue sub-features, and determining the quality data corresponding to the round-based dialogue information; generating multiple candidate training data based on standard difficulty data and the round-based dialogue information; and determining the round-based training data for the next round of dialogue based on the difficulty data and quality data corresponding to the multiple candidate training data. This specification can provide a more intuitive and effective training experience, enabling effective monitoring and statistical analysis of training results, real-time feedback and evaluation, and improving the efficiency and quality of training.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese application filed on April 22, 2025, with application number 202510504160.8, entitled "An Intelligent AI Scene Simulation Training Method, System, Device and Medium". Technical Field

[0003] This specification relates to the field of artificial intelligence technology, and in particular to a method, system, device, and medium for evaluating the quality of dialogue training. Background Technology

[0004] Traditional dialogue training methods mainly rely on artificially simulated dialogue scenarios or simple recordings for training. These methods lack authenticity and interactivity, making it difficult to reproduce the complexity and uncertainty of real dialogues, resulting in poor training effectiveness and user experience.

[0005] Therefore, this paper aims to propose a method, system, device, and medium for evaluating the quality of dialogue training. By simulating real-world dialogue scenarios, it provides a more intuitive and effective training experience. It should be able to effectively monitor and statistically analyze training results, provide real-time feedback and evaluation, improve training efficiency and quality, and enhance the user experience. Summary of the Invention

[0006] This specification provides one or more embodiments of a method for evaluating the quality of dialogue training. The method is executed by a processor and includes: retrieving target training data to generate a target training library; responding to a target user performing training based on the target training data in the target training library, wherein the target training data includes target application scenarios and simulated dialogue information, and the training includes multiple rounds of dialogue; for each round of dialogue: obtaining the target user's round-based dialogue information, wherein the round-based dialogue information corresponds to the round-based training data, and the round-based training data is the training data used in this round of dialogue, the round-based dialogue information including multiple dialogue sub-data and multiple dialogue sub-features; for a single dialogue sub-data among the multiple dialogue sub-data, based on the dialogue... The process involves: generating sub-quality data corresponding to the dialogue sub-data using sub-data, dialogue sub-features corresponding to the dialogue sub-data, the target application scenario, and the simulated dialogue information; determining quality data corresponding to the round-based dialogue information based on the multiple sub-quality data corresponding to the multiple dialogue sub-data; determining standard difficulty data based on the change features corresponding to the multiple quality data and the difficulty data corresponding to the round-based training data; generating multiple candidate training data based on the standard difficulty data and the round-based dialogue information; determining the round-based training data for the next round of dialogue based on the difficulty data corresponding to the multiple candidate training data and the quality data; and sending the round-based training data for the next round of dialogue and the quality data to the user terminal.

[0007] This specification provides one or more embodiments of a dialogue training quality assessment system, the system including a generation module and a training module; the generation module is configured to retrieve target training data and generate a target training library; the training module is configured to respond to a target user performing training based on the target training data in the target training library, the target training data including target application scenarios and simulated dialogue information, the training including multiple rounds of dialogue, for each round of dialogue: obtaining the target user's round dialogue information, the round dialogue information corresponding to the round training data, the round training data being the training data used in this round of dialogue, the round dialogue information including multiple dialogue sub-data and multiple dialogue sub-features; for a single dialogue sub-data in the multiple dialogue sub-data... The data includes: generating sub-quality data corresponding to the dialogue sub-data, the dialogue sub-features corresponding to the dialogue sub-data, the target application scenario, and the simulated dialogue information; determining quality data corresponding to the round-based dialogue information based on the multiple sub-quality data corresponding to the multiple dialogue sub-data; determining standard difficulty data based on the change features corresponding to the multiple quality data and the difficulty data corresponding to the round-based training data; generating multiple candidate training data based on the standard difficulty data and the round-based dialogue information; determining the round-based training data for the next round of dialogue based on the difficulty data corresponding to the multiple candidate training data and the quality data; and sending the round-based training data for the next round of dialogue and the quality data to the user terminal.

[0008] This specification provides one or more embodiments of a device for evaluating the quality of dialogue training. The device includes at least one processor and at least one memory. The at least one memory is used to store computer instructions. The at least one processor is used to execute at least a portion of the computer instructions to implement the method for evaluating the quality of dialogue training described in the above embodiments.

[0009] This specification provides one or more embodiments of a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions from the storage medium, the computer executes the dialogue training quality assessment method described in the above embodiments.

[0010] The beneficial effects of this manual include, but are not limited to: (1) providing a more intuitive and effective training experience; (2) enabling effective monitoring and statistical analysis of training results, real-time feedback and evaluation, and improving the efficiency and quality of training; and (3) using machine learning models to more meticulously evaluate the impact of multi-dimensional data on sub-quality data and improve the accuracy of determining sub-quality data. Attached Figure Description

[0011] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0012] Figure 1 These are exemplary schematic diagrams of an intelligent AI scene simulation training system according to some embodiments of this specification;

[0013] Figure 2 This is an exemplary flowchart of an intelligent AI scene simulation training method according to some embodiments of this specification;

[0014] Figure 3 These are exemplary schematic diagrams illustrating the determination of quality data according to some embodiments of this specification;

[0015] Figure 4 This is an exemplary schematic diagram of a quality assessment model shown according to some embodiments of this specification.

[0016] Figure labeling: 110: Generation module; 120: Training module; 311-1, ..., 311-n: Multiple dialogue sub-data; 312-1, ..., 312-n: Multiple dialogue sub-features; 313: Target application scenario A; 314-1, ..., 314-n: Multiple simulated dialogue information; 320-1, ..., 320-n: Multiple sub-quality data; 330: Quality data; 411: Training sub-data; 412: Target application scenario B; 413: Simulated dialogue information B; 420: Quality assessment model; 421: Information generation layer; 422: Quality assessment layer; 431: Standard dialogue sub-data; 432: Standard dialogue sub-features; 441: Dialogue sub-data B; 442: Dialogue sub-feature B; 450: Sub-quality data B. Detailed Implementation

[0017] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings do not represent all implementation methods.

[0018] It should be understood that the terms "system," "device," "unit," and / or "module" used herein are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. If other terms can achieve the same purpose, they may be replaced by other expressions.

[0019] Unless the context clearly indicates an exception, words such as "a," "an," "a kind," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0020] In the embodiments of this specification, the order of the steps described in the step-by-step instructions is interchangeable unless otherwise specified, and steps may be omitted. Other steps may also be included in the operation process.

[0021] Figure 1 This is an exemplary schematic diagram of an intelligent AI scene simulation training system according to some embodiments of this specification.

[0022] In some embodiments, the intelligent AI scene simulation training system 100 includes a generation module 110 and a training module 120.

[0023] In some embodiments, the generation module can be configured to retrieve target training data and generate a target training library.

[0024] In some embodiments, the training module can be configured to, in response to a target user training based on target training data in a target training library, obtain the target user's round-by-round dialogue information; generate quality data corresponding to the round-by-round dialogue information based on the round-by-round training data and the round-by-round dialogue information; generate multiple candidate training data based on the round-by-round dialogue information; determine the round-by-round training data for the next round of dialogue based on the difficulty data and quality data corresponding to the multiple candidate training data; and send the round-by-round training data and quality data for the next round of dialogue to the user terminal.

[0025] In some embodiments, for a single dialogue sub-data in a plurality of dialogue sub-data, the training module is further configured to: generate sub-quality data corresponding to the dialogue sub-data based on the dialogue sub-data, the dialogue sub-features corresponding to the dialogue sub-data, the target application scenario and simulated dialogue information; and determine quality data based on the plurality of sub-quality data corresponding to the plurality of dialogue sub-data.

[0026] In some embodiments, the training module is further configured to: generate sub-quality data corresponding to the dialogue sub-data through a quality assessment model based on the dialogue sub-data, the dialogue sub-features corresponding to the dialogue sub-data, the target application scenario, and the simulated dialogue information.

[0027] A user terminal refers to a terminal device used by the target user, such as a mobile phone or tablet computer. In some embodiments, the intelligent AI scene simulation training system is communicatively connected to the user terminal.

[0028] In some embodiments, the intelligent AI scene simulation training system may further include a processor and a storage device. The processor is configured to process data from at least one module of the intelligent AI scene simulation training system 100 or an external data source. The processor includes a central processing unit (CPU), an application-specific integrated circuit (ASIC), a controller, a microcontroller unit, a microprocessor, or any combination thereof.

[0029] The storage device is configured to store data, instructions, and / or information related to the intelligent AI scene simulation training system 100. For example, the storage device stores target training data and a target training library, etc. In some embodiments, the storage device may be integrated into a processor or located in the cloud, etc.

[0030] In some embodiments, the generation module and the training module may be implemented by a processor.

[0031] For further explanation of the above content, please refer to [link / reference]. Figures 2-4 And its related descriptions.

[0032] The intelligent AI scenario simulation training system generates a target training library through a generation module, enabling targeted training for different users. The training module trains target users and provides real-time feedback and evaluation of training results, improving the efficiency and quality of training.

[0033] It should be understood that Figure 1 The intelligent AI scene simulation training system and its modules shown can be implemented in various ways. It should be noted that the above description of the intelligent AI scene simulation training system and its modules is for convenience only and should not limit this specification to the scope of the illustrated embodiments. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. In some embodiments, Figure 1 The generation module 110 and training module 120 disclosed herein can be different modules within the same system, or a single module can implement the functions of two or more of the aforementioned modules. For example, the modules can share a single storage module, or each module can have its own separate storage module. Such variations are all within the scope of protection of this specification.

[0034] Figure 2 This is an exemplary flowchart of an intelligent AI scene simulation training method according to some embodiments of this specification. In some embodiments, process 200 is executed by a processor. Figure 2 As shown, process 200 includes the following steps:

[0035] Step 210: Retrieve target training data and generate a target training library.

[0036] Target training data refers to data used for training. In some embodiments, training includes dialogue training, etc. Dialogue training refers to training where the processor poses questions, which are then answered by the target user.

[0037] In some embodiments, the target training data includes multiple training sub-data items. A training sub-data item refers to a dialogue statement needed for dialogue training of the target user.

[0038] Target users refer to those who require dialogue training. Examples include at least one of the following: customer service representatives, bank tellers, sales personnel, and e-commerce sellers.

[0039] A target training library is a database used to store target training data.

[0040] In some embodiments, the processor generates the target training library using various methods. For example, the processor obtains target training data corresponding to the target user through a storage device, adds the target training data to the target training library corresponding to the target user, and stores the target training library in the storage device for use by the target user in training.

[0041] In some embodiments, target training data for different target users can be pre-set based on experience and stored in a storage device. A target training library can correspond to one or more target users.

[0042] In some embodiments, the processor may determine reference difficulty data based on multiple historical training results of the target user's multiple historical training sessions, and retrieve target training data based on the reference difficulty data.

[0043] Historical training results refer to the overall training outcomes corresponding to previous training sessions. See below for an explanation of overall training outcomes and related descriptions.

[0044] Reference difficulty data refers to the reference value of the difficulty data of the training sub-data to be retrieved.

[0045] In some embodiments, the processor calculates the average of multiple historical training results based on the target user's multiple historical training sessions. If the average meets a preset condition, the processor reduces the initial reference difficulty by a preset adjustment amount to obtain reference difficulty data. If the average does not meet the preset condition, the processor determines the initial reference difficulty as the reference difficulty data.

[0046] The preset conditions include the mean value not exceeding the quality threshold. The quality threshold and preset adjustment amount are pre-set based on historical experience.

[0047] The initial reference difficulty refers to the initial reference difficulty data. In some embodiments, the initial reference difficulty is preset based on historical experience.

[0048] In some embodiments, the preset adjustment amount may be related to the training frequency of the target users. For example, the preset adjustment amount may be negatively correlated with the training frequency of the target users.

[0049] Training frequency refers to how often target users are trained. A higher training frequency indicates that the intervals between multiple training sessions for target users are shorter, the target users' skill level does not change significantly, there is no need to drastically adjust the original reference difficulty data, and therefore the amount of pre-set adjustment is smaller.

[0050] By adjusting the training frequency for target users, a preset adjustment amount can be set to better match the target users. This avoids situations where the preset adjustment amount is too large when the target users are trained frequently, which would cause the target users to undergo multiple training sessions with a large range of difficulty in a short period of time, thus reducing the effectiveness of the training.

[0051] In some embodiments, the processor can remove multiple training sub-data points from the target training data that do not meet preset retrieval conditions, and generate a target training library based on the removed target training data. The preset retrieval conditions include that the difference between the difficulty data of the training sub-data and the reference difficulty data is less than a first difference threshold. The first difference threshold is preset based on historical experience.

[0052] Based on multiple historical training results of the target users, the target users' own level is assessed, and then the retrieved training sub-data is dynamically adjusted to make the target training data more consistent with the target users' current capabilities, thereby improving the training effect.

[0053] Step 220: In response to the target user being trained based on the target training data in the target training library, the training includes multiple rounds of dialogue, and steps 221-225 are executed for each round of dialogue.

[0054] Understandably, training is equivalent to simulating real-life dialogue scenarios, where the two parties engage in multiple rounds of dialogue to complete the communication.

[0055] Step 221: Obtain the target user's round-by-round dialogue information.

[0056] Round-based dialogue information refers to the content of the target user's response in the current round of dialogue. In some embodiments, round-based dialogue information includes one or more dialogue sub-data and dialogue sub-features corresponding to each dialogue sub-data. Dialogue sub-data refers to a statement by the target user in response to training sub-data. Dialogue sub-features refer to the relevant information and characteristics of the dialogue sub-data.

[0057] In some embodiments, dialogue sub-features include response waiting time, response speed, and response volume corresponding to the dialogue sub-data.

[0058] In some embodiments, the processor can record the target user's dialogue sub-data, response waiting time, response speed, and response volume in real time during the dialogue to obtain round-by-round dialogue information.

[0059] Step 222: Based on the training data and dialogue information from each round, generate quality data corresponding to the dialogue information from each round.

[0060] In some embodiments, round training data refers to the training data used in the current round of dialogue. For example, the training sub-data used by the processor in the current round of dialogue.

[0061] In some embodiments, round-based dialogue information corresponds to round-based training data. It is understood that multiple training sub-data points in the target training data have a certain word order, and the processor selects a training sub-data point from the target training library according to the word order as the round-based training data during training. Accordingly, the target user responds based on the round-based training data, resulting in one or more dialogue sub-data points.

[0062] For example, if the target training data is used to train target users to provide after-sales service, and the after-sales service may include product descriptions, refunds or replacements, and service ratings, then the processor can set the training sub-data used in the dialogue according to the order of the product descriptions, refunds or replacements, and service ratings.

[0063] In some embodiments, after generating the target training library, the processor can construct a response graph based on the target training data in the target training library.

[0064] A response graph is a graph structure that reflects the relationship between target training data and corresponding responses. A graph structure is a data structure composed of nodes and edges, with edges connecting nodes; nodes and edges can have features.

[0065] In some embodiments, the response graph includes question nodes and answer nodes.

[0066] A question node represents a training sub-data set. The node characteristics of a question node include difficulty data, etc. Difficulty data refers to data used to characterize the level of difficulty of the training sub-data set. In some embodiments, the difficulty data is pre-set based on historical experience.

[0067] A response node refers to the response statement corresponding to the training sub-data. The node characteristics of a response node include the response waiting time, response speed, and response volume. The response waiting time refers to the time interval between receiving the training sub-data and making a response.

[0068] In some embodiments, the processor queries a preset question-and-answer table for multiple question-and-answer statements corresponding to the training sub-data, and uses these multiple question-and-answer statements as answer nodes corresponding to the training sub-data. The processor then uses the answer waiting time, answer speed, and answer volume corresponding to each question-and-answer statement as node features of the answer node. The preset question-and-answer table is pre-set based on historical experience and includes multiple training sub-data and multiple answer statements corresponding to different training sub-data, as well as the answer waiting time, answer speed, and answer volume corresponding to each answer statement.

[0069] In some embodiments, there is an edge between the question node and the corresponding answer node, pointing from the question node to the answer node.

[0070] In some embodiments, since the multiple training sub-data in the target training data have a certain word order, after responding to a training sub-data, the processor can select the next training sub-data to be proposed based on the response. Therefore, some response nodes also have outgoing edges, which point to one or more questioning nodes used in subsequent questioning.

[0071] Quality data refers to data used to characterize the quality of a target user's responses in a current round of dialogue. In some embodiments, the processor determines initial quality data based on the similarity between one or more dialogue sub-data in the round-dialogue information and their corresponding multiple answer nodes. For example, the processor calculates one or more dialogue sub-data in the round-dialogue information, concatenates them into a long sentence, calculates the average similarity between the long sentence and its corresponding multiple answer nodes, and uses this average similarity as the initial quality data. The multiple answer nodes corresponding to the long sentence refer to the multiple answer nodes pointed to by the question nodes in the response graph that represent the training sub-data corresponding to the round-dialogue information.

[0072] In some embodiments, the processor can calculate the similarity between long sentences and corresponding answer nodes using BERT models, SentenceBERT models, SimCSE models, etc.

[0073] In some embodiments, the processor can also adjust the initial quality data based on multiple dialogue sub-features in the round-by-round dialogue information and the mean values ​​of the node features of the corresponding multiple answer nodes to obtain the final quality data. For example, the processor calculates the mean value of each data item in the multiple dialogue sub-features and calculates the ratio of the mean value of each data item to the corresponding data item in the mean value of the node features. Based on the multiple ratios and the mean value of the node features, the initial quality data is adjusted using a preset algorithm. For example, the preset algorithm is shown in the following formula (1):

[0074] B=A×[1 - (k1 × C1 / D1 + ……+ kn × Cn / Dn) / (k1 ×D1 + …… + kn ×Dn)] (1)

[0075] Where A represents the initial quality data, B represents the final quality data, C1, ..., Cn represent the mean of the first to the nth data items among multiple dialogue sub-features, D1, ..., Dn represent the mean of the first to the nth data items among the node feature mean values ​​of multiple response nodes, and k1, ..., kn represent the weights of the first to the nth data items. The weights of different data items can be preset based on experience. n is the number of data items included in the dialogue sub-features, for example, 3.

[0076] In some embodiments, the processor can also determine quality data based on multiple sub-quality data corresponding to multiple dialogue sub-data. See more details. Figure 3 And its related descriptions.

[0077] Step 223: Generate multiple candidate training data based on the round-by-round dialogue information.

[0078] Candidate training data refers to the training sub-data to be used in the next round of dialogue. It is understood that there may be multiple responses to a single training sub-data, and each response corresponds to a different subsequent question. Therefore, the processor can select a subset of training sub-data (i.e., subsequent question statements) from multiple training sub-data sets for the next round of dialogue based on the current round's dialogue information (i.e., one or more current response statements), and determine the final training sub-data to be used in the next round of dialogue from among the candidate training sub-data.

[0079] In some embodiments, the processor can acquire candidate training data in various ways. For example, the processor determines the answer node that meets the candidate criteria based on the similarity between the turn-based dialogue information and the corresponding multiple answer nodes, and uses the multiple question nodes pointed to by the outgoing edges of the answer node as multiple candidate training data. The candidate criteria include the highest similarity between the turn-based dialogue information and the answer node.

[0080] In some embodiments, the processor may also determine standard difficulty data based on the variation characteristics corresponding to multiple quality data and the difficulty data corresponding to the round training data, and generate multiple candidate training data based on the standard difficulty data and the round dialogue information.

[0081] It is important to note that at least one round of dialogue is required before the processor can determine the variation characteristics corresponding to multiple quality data points, and thus generate multiple candidate training data points.

[0082] Variation features refer to data used to characterize changes in quality data. In some embodiments, variation features include variation curves, etc. The x-axis of the variation curve represents training duration, and the y-axis represents quality data.

[0083] In some embodiments, the processor can record multiple training time points during multi-turn dialogues and calculate the quality data corresponding to each training time point, constructing a change curve based on the multiple training time points and the quality data corresponding to different training time points. Here, a training time point refers to the time point at the end of a round of dialogue.

[0084] Standard difficulty data refers to the reference value of the difficulty data of candidate sub-data. In some embodiments, the processor can take the first derivative of multiple sampling points on the change curve. If the first derivative of N consecutive sampling points from the current time point satisfies a first adjustment condition, the difficulty data of the training sub-data in this round of dialogue is increased by a difficulty adjustment amount to become the standard difficulty data. If the first derivative of N consecutive sampling points from the current time point satisfies a second adjustment condition, the difficulty data of the training sub-data in this round of dialogue is decreased by a difficulty adjustment amount to become the standard difficulty data. The difficulty adjustment amount and multiple sampling points can be preset based on historical experience. The difficulty data of the training sub-data is obtained through the response graph. N is a preset number.

[0085] The first adjustment condition includes all first derivatives being greater than 0, and the second adjustment condition includes all first derivatives being less than 0. If the first derivatives of N consecutive sampling points from the current time point do not meet the first and second adjustment conditions, the processor will use the difficulty data of the training sub-data from this round of dialogue as the standard difficulty data.

[0086] In some embodiments, the N value can also be correlated with the number of reversals of the first derivative at multiple sampling points on the change curve. For example, the N value is positively correlated with the number of reversals. The number of reversals refers to the number of times the first derivative changes from positive to negative. The larger the number of reversals, the more difficult it is to determine whether the quality data is in an upward or downward trend. In this case, a larger N value is needed to improve the accuracy of determining whether the quality data is in an upward or downward trend.

[0087] In some embodiments, the processor may select answer nodes that meet the candidate conditions in the response graph, determine multiple question nodes pointed to by the outgoing edges of the answer nodes, and select multiple question nodes whose difficulty data and standard difficulty data do not exceed a second difference threshold as multiple candidate training data. The second difference threshold can be preset based on experience.

[0088] By analyzing the changes in quality data from previous training sessions and the difficulty data of current training sub-data, we can determine more suitable standard difficulty data, thereby generating multiple candidate training data that are more suitable for the current target user level and improving training effectiveness.

[0089] Step 224: Based on the difficulty and quality data corresponding to multiple candidate training data, determine the training data for the next round of dialogue.

[0090] In some embodiments, the processor can calculate the ratio of the quality data to the pass / fail value in the current round of dialogue, and calculate the average of the difficulty data of multiple candidate training data. The processor multiplies the obtained ratio by the average of the difficulty data to obtain the comprehensive difficulty data, and selects the candidate training data with the smallest absolute value of the difference between the difficulty data and the comprehensive difficulty data from among the multiple candidate training data, and uses this candidate training data as the training data for the next round of dialogue. The pass / fail value can be preset based on experience.

[0091] Step 225: Send the training data and quality data for the next round of dialogue to the user terminal.

[0092] In some embodiments, the processor can send training data for the next round of dialogue and quality data for the current round of dialogue to the user terminal, allowing the target user to view and conduct the next round of dialogue. The processor can complete multi-round dialogue training by sending training data to the target user multiple times, recording the corresponding round of dialogue information, and determining the quality data.

[0093] Overall training results refer to data used to characterize the quality of training completed by target users.

[0094] In some embodiments, in response to a target user completing training, the processor can generate a comprehensive training result based on multiple quality data points corresponding to multiple rounds of dialogue. For example, the processor can determine the comprehensive training result based on the mean of multiple quality data points. Alternatively, the processor can determine the comprehensive training result based on a weighted sum of multiple quality data points. The weights of different rounds of dialogue are pre-set based on historical experience.

[0095] By integrating the quality data from multiple rounds of dialogue with target users, intuitive and comprehensive training results are generated after the training is completed. This facilitates a comprehensive evaluation of the training effectiveness, provides growth feedback to target users, and helps to determine target training data that is more suitable for the level of target users in subsequent training.

[0096] By building a corresponding target training library for target users and conducting training, more intuitive and effective training can be provided to target users through simulating real dialogue scenarios. At the same time, the training results can be effectively monitored and statistically analyzed, providing real-time feedback and evaluation, thereby improving the efficiency and quality of training.

[0097] It should be noted that the above description of process 200 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0098] Figure 3This is an exemplary schematic diagram illustrating the determination of quality data according to some embodiments of this specification.

[0099] In some embodiments, round-based dialogue information includes multiple dialogue sub-data and multiple dialogue sub-features. Target training data also includes target application scenarios and simulated dialogue information. Each training sub-data may correspond to one target application scenario and one simulated dialogue information. For further explanation of round-based dialogue information, see [link to relevant documentation]. Figure 2 And its related descriptions.

[0100] In some embodiments, for a single dialogue sub-data piece among multiple dialogue sub-data pieces, the processor can generate sub-quality data corresponding to the dialogue sub-data piece based on the dialogue sub-data piece (e.g., dialogue sub-data piece 311-1, ..., dialogue sub-data piece 311-n, where n is the number of dialogue sub-data pieces), the dialogue sub-features corresponding to the dialogue sub-data piece (e.g., dialogue sub-features 312-1, ..., dialogue sub-features 312-n), the target application scenario (e.g., target application scenario 313), and simulated dialogue information (e.g., simulated dialogue information 314-1, ..., simulated dialogue information 314-n). The processor determines quality data 330 based on multiple sub-quality data pieces (e.g., sub-quality data 320-1, ..., sub-quality data 320-n) corresponding to multiple dialogue sub-data pieces. For a description of target training data, dialogue sub-data pieces, dialogue sub-features, and quality data, see [link to documentation]. Figure 2 And its related descriptions.

[0101] A target application scenario refers to the application scenario corresponding to the training sub-data. In some embodiments, the target application scenario includes the simulated customer's dialogue environment, mood state, and / or customer profile corresponding to the training sub-data. Here, "customer" refers to a simulated person engaging in dialogue with the target user. One target application scenario corresponds to one or more training sub-data sets.

[0102] In some embodiments, the conversation setting includes home or office, etc. Mood status includes good or neutral, etc. A customer profile refers to information used to describe a customer. A customer profile includes the customer's gender, age, occupation, etc.

[0103] In some embodiments, the processor may pre-store multiple different target application scenarios, randomly select one target application scenario to incorporate into the training sub-data before sending the training sub-data to the target user's user terminal, and then send the adjusted training sub-data to the target user's user terminal for training.

[0104] In some embodiments, integrating the target application scenario into the training sub-data includes: the processor integrating one or more question keywords corresponding to the target application scenario into the training sub-data. In some embodiments, the processor obtains question keywords, etc., corresponding to the target application scenario through a preset scenario table.

[0105] In some embodiments, the preset scenario table is pre-set based on historical experience and includes multiple target application scenarios and one or more question keywords corresponding to different target application scenarios. The one or more question keywords corresponding to the target application scenarios are determined by technical personnel based on experience.

[0106] Simulated dialogue information refers to information related to simulated responses to training sub-data. In some embodiments, simulated dialogue information includes one or more simulated sub-data and simulated sub-features corresponding to different simulated sub-data. Simulated sub-data refers to simulated dialogue sub-data. Simulated sub-features refer to dialogue sub-features corresponding to simulated sub-data.

[0107] One training subdata set corresponds to one or more simulation subdata sets.

[0108] In some embodiments, the processor generates simulated dialogue information in various ways. For example, the processor can acquire simulated speech segments input by a technician, break down the simulated speech segments into multiple more colloquial or easy-to-read simulated sentences as multiple simulated sub-data, and determine the simulated sub-features corresponding to the simulated sub-data based on the waiting time, speech rate, and volume of the simulated speech segments input by the technician.

[0109] In some embodiments, the processor can also generate simulated dialogue information based on the target user's historical training data and historical quality scores.

[0110] Historical training data refers to relevant data from the historical training process. In some embodiments, historical training data includes multiple historical training sub-data used by the target user and corresponding dialogue information from multiple historical rounds.

[0111] Historical quality data refers to the quality data corresponding to historical dialogue sub-data. In some embodiments, historical quality data includes quality data corresponding to dialogue information from multiple historical rounds in historical training data.

[0112] In some embodiments, the processor retrieves historical training data and historical quality data from a storage device.

[0113] In some embodiments, the processor can insert dialogue defects included in historical rounds of dialogue information into the initial simulated dialogue information based on the degree of obviousness and the number of defects, thereby generating simulated dialogue information.

[0114] Dialogue defects refer to flaws in the target user's responses. Examples of dialogue defects include inappropriate word choice, speaking too slowly, speaking too quickly, semantic redundancy, and logical inconsistencies. Dialogue defects included in historical dialogue information are identified either through manual annotation of the historical dialogue information or through automatic detection by the processor.

[0115] The degree of obviousness of a defect refers to data used to characterize the prominence of defects in a dialogue. The degree of obviousness of a defect can be expressed numerically, with a higher numerical value indicating a greater degree of obviousness of the defect.

[0116] In some embodiments, the processor calculates the average of multiple quality data in historical quality data and determines the severity and number of defects by querying a preset defect table.

[0117] In some embodiments, the preset defect table is pre-set based on historical experience, including multiple quality data averages and the degree of defect prominence and the number of defects corresponding to different quality data averages. The degree of defect prominence and the number of defects can be determined manually by annotating historical rounds of dialogue information corresponding to the quality data averages.

[0118] In some embodiments, for each historical training sub-data set, the processor can also search for the corresponding question node in the response graph based on the historical training sub-data set. The processor randomly selects a target application scenario, integrates one or more question keywords corresponding to the target application scenario into multiple answer nodes corresponding to the question node, and generates initial simulated dialogue information based on the multiple answer nodes and their corresponding node features. For more information on the response graph, see [link to relevant documentation]. Figure 2 And related explanations.

[0119] In some embodiments, the processor inserts the same number of dialogue defects from previous rounds of dialogue information into the initial simulated dialogue information, based on the number of defects. The processor can also insert corresponding dialogue defects into the initial simulated dialogue information based on the severity of the defects. For example, a higher severity of a defect might result in inappropriate word choice or logical contradictions, while a lower severity might result in speech rates that are too slow or too fast.

[0120] By aggregating historical training data and historical quality data of target users, and simulating the dialogue deficiencies of target users, it is possible to generate realistic simulated dialogue information, which is beneficial for more accurate subsequent evaluation of the training quality of target users.

[0121] Sub-quality data refers to the quality data corresponding to the dialogue sub-data.

[0122] In some embodiments, the processor acquires sub-quality data in various ways. For example, the processor adjusts the initial sub-quality data based on the scene fit between the dialogue sub-data and the target application scenario, as well as the degree to which the dialogue sub-data compensates for deficiencies in the simulated dialogue information, to obtain sub-quality data.

[0123] In some embodiments, the processor calculates the similarity between dialogue sub-data and its corresponding multiple response nodes, and uses the highest similarity as the initial sub-quality data of the dialogue sub-data. See [link to similarity calculation instructions] for details. Figure 2 And related explanations.

[0124] Scene fit is used to characterize the degree of fit between dialogue sub-data and the target application scene. In some embodiments, the processor calculates the overlap between the dialogue sub-data and one or more answer keywords corresponding to the target application scene, and determines the obtained overlap as the scene fit. Specifically, the processor determines the overlap as the ratio of the number of answer keywords contained in the dialogue sub-data to the total number of answer keywords corresponding to the target application scene.

[0125] The method for determining keywords in answers is similar to the method for determining keywords in questions.

[0126] The defect compensation degree characterizes the extent to which dialogue sub-data compensates for simulated dialogue information. In some embodiments, the processor compares dialogue sub-data, dialogue sub-features, and simulated dialogue information, and determines the defect compensation degree as the ratio of the number of dialogue defects in the simulated dialogue information compensated by the dialogue sub-data or dialogue sub-features to the total number of dialogue defects in the simulated dialogue information. For example, if the simulated dialogue information lacks the term "workday," but the dialogue sub-data contains "workday," then the dialogue sub-data is determined to have compensated for a defect in the simulated dialogue information. If the simulated dialogue information has a fast response speed, but the dialogue sub-features have a moderate response speed, then the dialogue sub-features are determined to have compensated for a defect in the simulated dialogue information.

[0127] In some embodiments, the processor can calculate the product of scene fit and defect compensation, add 1 to the product, and multiply it by the initial sub-mass data to obtain the sub-mass data.

[0128] In some embodiments, the processor can also generate sub-quality data corresponding to the dialogue sub-data using a quality assessment model. See [link to documentation] for details on this section. Figure 4 And related content.

[0129] In some embodiments, the processor can perform a weighted summation of multiple sub-quality data corresponding to multiple dialogue sub-data to obtain quality data. The weights of different dialogue sub-data are pre-set based on historical experience.

[0130] By conducting in-depth analysis of dialogue sub-data, target application scenarios, and simulated dialogue information, more accurate and comprehensive sub-quality data can be determined, thereby leading to more accurate quality data.

[0131] Figure 4 This is an exemplary schematic diagram of a quality assessment model shown according to some embodiments of this specification.

[0132] In some embodiments, the processor can generate sub-quality data corresponding to the dialogue sub-data based on the dialogue sub-data, the dialogue sub-features corresponding to the dialogue sub-data, the target application scenario, the simulated dialogue information, and the target training data corresponding to the dialogue sub-data (i.e., the training sub-data corresponding to the dialogue sub-data) through a quality assessment model. For a description of the dialogue sub-data, the dialogue sub-features corresponding to the dialogue sub-data, the target application scenario, the simulated dialogue information, and the training sub-data, please refer to [link to documentation]. Figures 2-3 And its related descriptions.

[0133] Quality assessment model 420 refers to a model used to evaluate sub-quality data. In some embodiments, the quality assessment model may be a machine learning model. For example, the quality assessment model may be a combination of one or more of the following: a deep neural network (DNN) model or a custom model.

[0134] In some embodiments, the inputs to the quality assessment model include training sub-data, target application scenarios, simulated dialogue information, dialogue sub-data, and dialogue sub-features, and the output includes sub-quality data corresponding to the dialogue sub-data.

[0135] In some embodiments, the quality assessment model can be obtained by training samples and corresponding labels from a training sample set. The training samples include sample training sub-data, sample target application scenarios and sample simulated dialogue information, sample dialogue sub-data, and sample dialogue sub-features. The labels corresponding to the training samples include the actual sub-quality data of the sample dialogue sub-data. In some embodiments, the training samples are obtained based on historical data. The labels are pre-determined by technical personnel.

[0136] In some embodiments, the quality assessment model includes an information generation layer 421 and a quality assessment layer 422. For example, the information generation layer and the quality assessment layer may include one or more combinations of deep neural network (DNN) models or custom models.

[0137] In some embodiments, the input to the information generation layer 421 may include training sub-data 411, a target application scenario (such as target application scenario B412), and simulated dialogue information (such as simulated dialogue information B413), and the output may include standard dialogue sub-data 431 and standard dialogue sub-features 432. Standard dialogue sub-data refers to the standard answers generated by the information generation layer. Standard dialogue sub-features refer to the dialogue sub-features corresponding to the standard answers generated by the information generation layer.

[0138] In some embodiments, the input to the quality assessment layer 422 may include standard dialogue sub-data 431, standard dialogue sub-features 432, dialogue sub-data (such as dialogue sub-data B441) and dialogue sub-features (such as dialogue sub-features B442), and the output may include sub-quality data (such as sub-quality data B450) corresponding to the dialogue sub-data.

[0139] In some embodiments, the information generation layer and the quality assessment layer can be obtained by training samples and corresponding labels in the training sample set, respectively.

[0140] In some embodiments, the training sample set includes a first training sample for the training information generation layer and a first label corresponding to the first training sample. The first training sample includes sample training sub-data, sample target application scenarios, and sample simulated dialogue information. The first label includes standard dialogue sub-data and standard dialogue sub-features corresponding to the sample training sub-data.

[0141] In some embodiments, the first training sample is obtained based on historical data. The processor can select multiple historical dialogue sub-data corresponding to the sample training sub-data, calculate the scenario fit of each historical dialogue sub-data with the target application scenario of the sample, and the defect compensation degree for the simulated dialogue information of the sample. The historical dialogue sub-data with the highest weighted sum of defect compensation degree and scenario fit is used as the standard dialogue sub-data corresponding to the sample training sub-data, and the historical dialogue sub-features corresponding to this historical dialogue sub-data are used as the standard dialogue sub-features corresponding to the sample training sub-data. The weights of scenario fit and defect compensation degree are pre-set based on historical experience. For an explanation of scenario fit and defect compensation, see [link to documentation]. Figure 3 And its related descriptions.

[0142] In some embodiments, the training sample set includes second training samples for the training quality evaluation layer and second labels corresponding to the second training samples. The second training samples include sample standard dialogue sub-data and sample standard dialogue sub-features output by the information generation layer, sample dialogue sub-data, and sample dialogue sub-features. The second label includes the actual sub-quality data of the sample dialogue sub-data. In some embodiments, the second training samples are obtained based on historical data and the information generation layer. The second label is pre-determined by a technician.

[0143] In some embodiments, a first training sample corresponds to a second training sample.

[0144] In some embodiments, the information generation layer and the quality assessment layer can be trained jointly as follows: Multiple first training samples with a first label are input into the initial information generation layer, and the output of the initial information generation layer and second training samples with a second label are input into the initial quality assessment layer. A loss function is constructed using the second label and the prediction results of the initial quality assessment layer. The initial information generation layer and the initial quality assessment layer are updated synchronously based on the iteration of the loss function. The quality assessment model training is complete when the loss function meets a preset iteration condition. The preset iteration condition can be loss function convergence, the number of iterations reaching a set value, etc.

[0145] In some embodiments, the labels of multiple training samples corresponding to different application scenario classifications in the training sample set meet preset training conditions.

[0146] Application scenario classification refers to the categorization of target application scenarios. In some embodiments, application scenario classification can be pre-set based on historical experience. For example, for multiple target application scenarios, the processor can classify the multiple target application scenarios into different application scenario categories based on at least one of the following: dialogue environment, mood state, and / or customer profile. For example, the processor classifies multiple target application scenarios into three categories: home, office, and street, based on the dialogue environment.

[0147] The training sample corresponding to the application scenario classification refers to the first training sample containing the target application scenario corresponding to the application scenario classification.

[0148] The preset training conditions include that the standard deviation of the second label of the second training sample corresponding to the first training sample is greater than a preset distribution threshold. In some embodiments, the preset distribution threshold is related to the target user's historical quality data. For example, the larger the range of multiple quality data in the historical quality data, the larger the preset score threshold. See [link to documentation] for an explanation of historical quality data. Figure 3 And its related descriptions.

[0149] Since a larger range in the quality data of target users indicates potentially greater fluctuations in the quality of their responses, the accuracy of the quality assessment model's output will decrease accordingly. By introducing training samples with significant differences across different application scenario classifications, the model's generalization performance and adaptability can be enhanced, thereby optimizing training results and improving the accuracy of identifying sub-quality data.

[0150] By employing machine learning models, the impact of multi-dimensional data on sub-quality data can be evaluated more meticulously, improving the accuracy of sub-quality data determination. Simultaneously, a hierarchical structure allows for targeted processing of different input data, thereby improving the accuracy of the final model output. Joint training addresses issues such as insufficient training samples and difficulty in obtaining labels.

[0151] Some embodiments of this specification also provide an intelligent AI scene simulation training apparatus, the apparatus including at least one processor and at least one memory. The at least one memory is used to store computer instructions. The at least one processor is used to execute at least a portion of the computer instructions to implement the intelligent AI scene simulation training method described in any of the above embodiments.

[0152] Some embodiments of this specification also provide a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the intelligent AI scene simulation training method described in any of the above embodiments.

[0153] Furthermore, certain features, structures, or characteristics in one or more embodiments of this specification may be appropriately combined.

[0154] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples by the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%.

[0155] If there is any inconsistency or conflict between the descriptions, definitions, and / or terms used in the materials referenced in this specification and the content described in this specification, the descriptions, definitions, and / or terms used in this specification shall prevail.

Claims

1. A method for evaluating the quality of dialogue training, characterized in that, The method is executed by a processor, and the method includes: Retrieve target training data and generate a target training database; In response to target users undergoing training based on target training data in the target training library, the target training data including target application scenarios and simulated dialogue information, the training includes multiple rounds of dialogue, for each round of dialogue: The round-based dialogue information of the target user is obtained. The round-based dialogue information corresponds to the round-based training data, which is the training data used in this round of dialogue. The round-based dialogue information includes multiple dialogue sub-data and multiple dialogue sub-features. For a single dialogue sub-data in the plurality of dialogue sub-data, sub-quality data corresponding to the dialogue sub-data is generated based on the dialogue sub-data, the dialogue sub-features corresponding to the dialogue sub-data, the target application scenario, and the simulated dialogue information; Based on the multiple sub-quality data corresponding to the multiple dialogue sub-data, determine the quality data corresponding to the round dialogue information; Based on the variation characteristics corresponding to multiple quality data and the difficulty data corresponding to the training round data, standard difficulty data is determined; Based on the standard difficulty data and the round-dialogue information, multiple candidate training data are generated; Based on the difficulty data and quality data corresponding to the multiple candidate training data, the training data for the next round of dialogue is determined. The training data for the next round of dialogue, along with the quality data, will be sent to the user terminal.

2. The method as described in claim 1, characterized in that, The simulated dialogue information is determined based on the target user's historical training data and historical quality data.

3. The method as described in claim 1, characterized in that, The step of generating sub-quality data corresponding to a single dialogue sub-data point among the plurality of dialogue sub-data points, based on the dialogue sub-data point, the dialogue sub-features corresponding to the dialogue sub-data point, the target application scenario, and the simulated dialogue information, includes: Based on the dialogue sub-data, the dialogue sub-features corresponding to the dialogue sub-data, the target application scenario, the simulated dialogue information, and the target training data corresponding to the dialogue sub-data, sub-quality data corresponding to the dialogue sub-data is generated through a quality assessment model, wherein the quality assessment model is a machine learning model.

4. The method as described in claim 3, characterized in that, The quality assessment model is trained using a training sample set, where the labels of multiple training samples corresponding to multiple application scenario classifications in the training sample set meet preset training conditions.

5. The method as described in claim 1, characterized in that, The process of retrieving target training data and generating the target training library includes: Based on the multiple historical training results and preset adjustment amounts of the target user, a reference difficulty data is determined. Based on the reference difficulty data and the preset adjustment amount, the target training data is retrieved.

6. The method as described in claim 5, characterized in that, The preset adjustment amount is determined based on the training frequency of the target user.

7. The method as described in claim 1, characterized in that, The method further includes: In response to the target user completing the training, a comprehensive training result is generated based on the multiple quality data corresponding to the multi-round dialogue.

8. A system for evaluating the quality of dialogue training, characterized in that, The system includes a generation module and a training module; The generation module is configured to retrieve target training data and generate a target training library. The training module is configured to respond to a target user performing training based on target training data in the target training library, the target training data including target application scenarios and simulated dialogue information, the training including multiple rounds of dialogue, for each round of dialogue: The round-based dialogue information of the target user is obtained. The round-based dialogue information corresponds to the round-based training data, which is the training data used in this round of dialogue. The round-based dialogue information includes multiple dialogue sub-data and multiple dialogue sub-features. For a single dialogue sub-data in the plurality of dialogue sub-data, sub-quality data corresponding to the dialogue sub-data is generated based on the dialogue sub-data, the dialogue sub-features corresponding to the dialogue sub-data, the target application scenario, and the simulated dialogue information; Based on the multiple sub-quality data corresponding to the multiple dialogue sub-data, determine the quality data corresponding to the round dialogue information; Based on the variation characteristics corresponding to multiple quality data and the difficulty data corresponding to the training round data, standard difficulty data is determined; Based on the standard difficulty data and the round-dialogue information, multiple candidate training data are generated; Based on the difficulty data and quality data corresponding to the multiple candidate training data, the training data for the next round of dialogue is determined. The training data for the next round of dialogue, along with the quality data, will be sent to the user terminal.

9. A device for evaluating the quality of dialogue training, characterized in that, The device includes at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is configured to execute at least a portion of the computer instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions. When the computer reads the computer instructions from the storage medium, the computer executes the method as described in any one of claims 1 to 7.

Images