Dialogue evaluation method, device and product

Abstract

The invention provides a dialogue evaluation method, dialogue evaluation equipment and a dialogue evaluation product. The method comprises the following steps: receiving a dialogue text, a preset domain information set and a script template; extracting feature vectors contained in the dialogue text; inputting a feature vector extracted from the dialogue text into a trained intention recognition model, and determining an intention recognition result for the dialogue text; constructing an evaluation standard by using the intention recognition result and the domain information set; the evaluation standard and the script template are matched and combined, and an evaluation script is generated; wherein the evaluation script comprises a scoring rule; receiving a reply dialogue of the tested user for the evaluation script; and performing quality evaluation on the reply dialogue by using a scoring rule in the evaluation script.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to dialogue assessment methods, devices, and products. Background Technology

[0002] In dialogue evaluation systems, existing dialogue quality assessments generally rely on manually preset static evaluation indicators (such as fluency, relevance, and information completeness), whose limitations are becoming increasingly apparent. These indicators are typically predefined manually and struggle to adapt to dynamic changes in the dialogue scenario. For example, during user-AI interaction, if a user suddenly raises questions unrelated to their pre-set intent, such as privacy protection or risk disclosure, traditional evaluation systems may fail to effectively assess dialogue quality because they do not cover the corresponding dimensions. Furthermore, static indicators require manual maintenance and updates, which are costly, and the scoring logic lacks real-time adjustment capabilities, making it difficult to respond to immediate changes in dialogue content.

[0003] In existing technologies, the generation and binding of evaluation indicators heavily rely on manually created rule bases, lacking the ability to dynamically perceive domain information and dialogue intent. In cross-domain scenarios (such as migrating from a sales scenario to a financial or medical scenario), evaluation rules need to be redesigned, limiting flexibility and scalability. Furthermore, the scoring process is often based on fixed weights or simple keyword matching, making it difficult to incorporate in-depth semantic analysis, leading to discrepancies between evaluation results and real-world scenario requirements. Summary of the Invention

[0004] This disclosure provides dialogue-based assessment methods, equipment, and products.

[0005] According to a first aspect of this disclosure, a dialogue evaluation method is provided. The method specifically includes: receiving dialogue text, a preset domain information set, and a script template; extracting feature vectors contained in the dialogue text; determining the intent recognition result for the dialogue text based on the feature vectors extracted from the dialogue text; constructing evaluation criteria using the intent recognition result and the domain information set; matching and combining the evaluation criteria with the script template to generate an evaluation script; wherein the evaluation script includes scoring rules; receiving a response dialogue from a user to the evaluation script; and evaluating the quality of the response dialogue using the scoring rules in the evaluation script.

[0006] Based on the above, by successively implementing intent recognition, domain adaptation, and structured evaluation criteria, the system can dynamically generate evaluation indicators that are highly matched to the scenario (such as privacy protection statements and product information accuracy) according to the dialogue content. This breaks through the static bottleneck of traditional preset indicators and significantly improves the accuracy of the evaluation. By combining the intent recognition results with domain information (preset or model-generated), the evaluation direction can be dynamically adjusted (e.g., focusing on "privacy protection" and "explanation of service terms" in the sales domain) to ensure that the evaluation dimensions cover the core needs of the dialogue.

[0007] According to at least one embodiment of this disclosure, a feature vector extracted from the dialogue text is input into a trained intent recognition model to determine the intent recognition result for the dialogue text, including: preprocessing the dialogue text; extracting feature vectors from the preprocessed dialogue text using a large language model; inputting the feature vectors into the intent recognition model; and outputting the intent recognition result for the dialogue text.

[0008] Based on the above, it is clear that keyword extraction and matching can accurately identify domain features in dialogue text. For example, in real estate dialogues, keywords such as "privacy" and "protection" can be extracted, and real estate domain features in the domain information set can be quickly matched to generate evaluation criteria highly relevant to the scenario. This method has low computational cost and is suitable for scenarios with clearly defined domain features.

[0009] According to at least one embodiment of this disclosure, an evaluation standard is constructed using intent recognition results and a domain information set, including: extracting keywords from the dialogue text; inputting the keywords into a domain adaptation module to determine the domain keywords that match the keywords in the domain information set; determining the domain recognition result corresponding to the dialogue text based on the domain keywords; determining the evaluation direction using the intent recognition results and the domain recognition results; and generating the evaluation standard based on the evaluation direction.

[0010] Based on the above, it can be seen that by directly outputting domain recognition results through a pre-trained domain adaptation model, the keyword extraction step is eliminated. Utilizing the semantic understanding capabilities of a large language model, end-to-end domain classification is achieved, making it particularly suitable for handling complex or ambiguous dialogue scenarios, such as hybrid dialogues containing multiple domain features. The combination of these two methods provides a flexible approach to domain recognition, allowing for the selection of appropriate implementation methods based on the complexity of the dialogue scenario and system resources, thus improving the system's adaptability and robustness.

[0011] According to at least one embodiment of this disclosure, constructing evaluation criteria using intent recognition results and domain information sets includes: inputting dialogue text into a trained domain adaptation model and outputting domain recognition results for the dialogue text; determining the evaluation direction using the intent recognition results and domain recognition results; and generating evaluation criteria based on the evaluation direction.

[0012] Based on the above, it can be seen that by directly outputting domain recognition results through a pre-trained domain adaptation model, the keyword extraction step is eliminated. Utilizing the semantic understanding capabilities of a large language model, end-to-end domain classification is achieved, making it particularly suitable for handling complex or ambiguous dialogue scenarios, such as hybrid dialogues containing multiple domain features. The combination of these two methods provides a flexible approach to domain recognition, allowing for the selection of appropriate implementation methods based on the complexity of the dialogue scenario and system resources, thus improving the system's adaptability and robustness.

[0013] According to at least one embodiment of this disclosure, an evaluation direction is determined using intent recognition results and domain recognition results; an evaluation standard is generated based on the evaluation direction, including: combining intent recognition results and domain recognition results to obtain an evaluation direction; splitting the evaluation direction into multiple evaluation dimensions; determining scoring rules corresponding to the multiple evaluation dimensions based on a preset rule base or semantic analysis; and constructing a structured evaluation standard using the evaluation dimensions and the corresponding scoring rules.

[0014] Based on the above, it is evident that by fusing semantic intent and domain recognition results, evaluation directions highly matched to the dialogue scenario are generated, overcoming the limitations of traditional static indicators in adapting to dynamic dialogue scenarios. Breaking down the evaluation directions into multiple evaluation dimensions refines the granularity of the assessment, making dialogue quality evaluation more comprehensive and specific, accurately reflecting multiple aspects of the dialogue. The combination of a pre-defined rule base and semantic analysis to determine scoring rules leverages the efficiency and stability of predefined rules while dynamically adapting to the needs of complex or new scenarios through semantic analysis, enhancing the system's flexibility and robustness. Furthermore, the generated structured evaluation standards (such as JSON or XML formats) are uniform and parsable, facilitating matching with script templates and real-time dialogue monitoring, improving the automation and consistency of the evaluation process.

[0015] According to at least one embodiment of this disclosure, the evaluation criteria are matched and combined with a preset script template to generate an evaluation script, including: receiving a preset script template; and generating an evaluation script by replacing the placeholders in the script template with the evaluation dimensions and scoring rules in the evaluation criteria.

[0016] Based on the above, it can be seen that the dynamic generation mechanism enables real-time compilation of evaluation scripts, dynamically adjusting evaluation dimensions and scoring rules according to real-time changes in the dialogue content (such as homeowners shifting from inquiring about prices to privacy issues). This ensures more accurate evaluation results, promptly reflecting key issues in the dialogue (such as whether the agent effectively responded to privacy concerns), and providing support for real-time feedback and optimization. It effectively overcomes the limitations of traditional static evaluation scripts, achieving precise assessment and real-time monitoring of dialogue quality.

[0017] According to at least one embodiment of this disclosure, receiving a response dialogue from a user under test to an evaluation script; and evaluating the quality of the response dialogue using evaluation criteria, includes: receiving a response dialogue from a user under test to an evaluation script; performing semantic similarity matching between the response dialogue and scoring rules; and executing the corresponding scoring rules based on the matching results to generate a quality evaluation result.

[0018] Based on the above, after obtaining the assessment script, it is possible to receive responses from the users being tested. While engaging in dialogue with the users according to the assessment script, the matching results can be scored using the scoring rules within the script. Different scoring rules can be developed for different assessment scripts, thereby making the quality assessment results more accurate.

[0019] According to at least one embodiment of this disclosure, the intent recognition model is trained based on real dialogue data and manually labeled data as training samples. The training samples include dialogue texts from different domains and their corresponding intent labels.

[0020] Based on the above, by preparing training samples belonging to different domains in advance, the trained intent recognition model can better meet users' intent recognition needs in various domains and obtain more accurate intent recognition results.

[0021] According to a second aspect of this disclosure, a dialogue evaluation system is provided. The system includes: an intent recognition module, configured to receive dynamically generated dialogue text and a preset domain information set; extract feature vectors contained in the dialogue text; input the feature vectors extracted from the dialogue text into a trained intent recognition model to determine an intent recognition result for the dialogue text; a structuring module, configured to construct evaluation criteria using the intent recognition result and the domain information set; a script template compilation module, configured to receive a script template, match and combine the evaluation criteria with the script template to generate an evaluation script; and a dialogue detection module, configured to receive a response dialogue from a user to the evaluation script; and evaluate the quality of the response dialogue using the evaluation criteria.

[0022] According to a third aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs the method described in the first aspect of any embodiment of this disclosure.

[0023] According to a fourth aspect of this disclosure, a readable storage medium is provided that stores executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect of any embodiment of this disclosure.

[0024] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first aspect of any embodiment of this disclosure. Attached Figure Description

[0025] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0026] Figure 1 This is a flowchart illustrating a dialogue evaluation method provided in an embodiment of the present disclosure.

[0027] Figure 2 This is a schematic diagram of a process for constructing evaluation criteria, provided as an embodiment of the present disclosure.

[0028] Figure 3 This is a schematic diagram of another process for constructing evaluation criteria provided in an embodiment of this disclosure.

[0029] Figure 4 This is a schematic block diagram of the structure of a dialogue assessment system according to one embodiment of the present disclosure.

[0030] Figure 5 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0031] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0032] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Figure 1 This is a flowchart illustrating a dialogue assessment method provided in this disclosure. Figure 1 The method shown includes steps 101 to 107. This method can be executed by an electronic device such as a server (local server or cloud server).

[0034] Step 101: Receive the dialogue text, the preset domain information set, and the script template.

[0035] The dialogue text referred to here is the content of a conversation provided by the user being tested. This dialogue text typically contains keywords that clearly define the assessment topic and other relevant information. Intent recognition can be performed through this dialogue text.

[0036] In specific application scenarios, domain information (such as the "real estate" domain) can be pre-defined so that evaluation indicators can focus on specific domain characteristics (such as privacy protection and the accuracy of housing information). In addition, the pre-defined domain information set may be generated based on manual definition or data-driven methods.

[0037] For example, in the real estate sector, the information set might include keywords (such as "house", "ownership", "broker"), sector characteristics (such as privacy protection, transaction process) and typical dialogue patterns.

[0038] Step 102: Extract the feature vectors contained in the dialogue text.

[0039] The dialogue text undergoes preprocessing, including noise removal (such as punctuation and special characters), word segmentation, part-of-speech tagging, and named entity recognition, to better understand its semantic structure. Then, a large-scale language model (LLM) encoder is used to encode the preprocessed dialogue text, extracting semantic features and generating feature vectors. Through a self-attention mechanism, the model focuses on key information (such as keywords and phrases) in the dialogue text, encoding them into semantic feature vectors. It should be noted that this large-scale language model (LLM), such as the GPT series, employs an encoder-decoder architecture (typically a Transformer architecture). This large-scale language model is trained using real-world dialogue data and manually annotated data, including dialogue text and corresponding intent labels; the training samples can cover different domains.

[0040] Specifically, large language models based on the Transformer architecture (such as BERT, GPT series, or other pre-trained models) are used. The Transformer architecture, through multi-layer neural networks and self-attention mechanisms, is able to capture the semantic relationships and contextual dependencies between words in the text. The preprocessed word sequence is input into the encoder of the LLM to generate a semantic representation (embedding vector) for each word or phrase.

[0041] The segmented text is mapped to word embeddings, typically using pre-trained word vectors (such as BERT's WordPiece segmenter). For example, "house" is mapped to a high-dimensional vector (e.g., 768-dimensional, based on BERT). Through the Transformer's attention mechanism, the semantic associations of each word with other words are calculated, generating context-sensitive representations. For example, "house" and "area" are associated in context, enhancing the capture of the overall semantics of "house area." The Transformer calculates weights between words through self-attention, highlighting key information. For example, in "What is the house area?", "area" and "how much" are given higher weights because they are relevant to the query intent. The contextual embedding vectors for each word or phrase are typically high-dimensional feature vectors (e.g., 768-dimensional or higher, depending on the model). The feature vectors of the dialogue text are typically fixed-dimensional (e.g., 768-dimensional or 1024-dimensional) floating-point vectors containing semantic information of the dialogue.

[0042] Step 103: Input the feature vector extracted from the dialogue text into the trained intent recognition model to determine the intent recognition result for the dialogue text.

[0043] After obtaining the feature vectors as described above, the feature vectors can be further input into a pre-trained intent recognition model to identify the intent recognition result of the dialogue text.

[0044] The intent recognition discussed here can be understood as analyzing and classifying the semantic features of dialogue text using an intent recognition model to determine the speaker's core intent (such as inquiry, doubt, request, etc.) and transforming the text content into structured intent tags. The role of intent recognition includes: being the first step in the dynamic indicator generation process, providing a foundation for subsequent domain adaptation, indicator structuring, and evaluation script generation. It helps the system understand the core needs of the dialogue (such as the homeowner's "privacy concerns"), thereby dynamically generating evaluation indicators that match the scenario (such as "privacy protection instructions").

[0045] The intent recognition model mentioned here can be an encoder-decoder model based on the Transformer architecture.

[0046] The intent recognition model was trained in advance. During training, the intent recognition model was trained using real dialogue data and manually labeled data as training samples. The training samples included dialogue texts from different domains and their corresponding intent labels.

[0047] The intent recognition model is trained on real-world dialogue data and manually labeled data. Its training data includes dialogue texts from different domains and their corresponding intent labels. This data is crucial to ensuring the model can accurately identify core intents in diverse dialogue scenarios. Real-world dialogue data comes from actual interaction scenarios, such as dialogues between real estate agents and AI-simulated homeowners in a product context, designed to train the agent's communication skills. These dialogues capture patterns in natural language, including diverse expressions, tones, and domain-specific terminology, such as "house size" or "privacy protection" in the real estate field. To improve dataset quality, human annotators assign specific intent labels to each dialogue, such as "inquiring about house information," "privacy concerns," or "signing intention." These labels provide supervised learning signals for the model, enabling it to map dialogue text to corresponding intents.

[0048] Training data covers multiple domains to ensure the model's generalization ability and adaptability across various dialogue scenarios. For example, in addition to the real estate domain, the data may also include customer service, healthcare, or e-commerce, each with unique intents and linguistic features. The data collection process involves acquiring dialogue records from real-world sources, such as call logs, chat logs, or recorded interactions, and using AI to generate synthetic dialogues to cover edge cases or uncommon intents. Preprocessing steps are performed to prepare the training data, including noise removal (such as removing irrelevant punctuation or filler words), word segmentation (using tools like Jieba to segment Chinese text into words or phrases), and named entity recognition to identify key entities (such as names or locations). These steps ensure the text is structured and clean, facilitating effective feature extraction.

[0049] Intent recognition models are typically based on large language models with a Transformer architecture (such as BERT or the GPT series), fine-tuned on labeled datasets. During training, the model encodes the dialogue text into semantic feature vectors via an encoder, using a self-attention mechanism to capture contextual relationships. For example, in the dialogue "I'm worried about my personal information being leaked," the model identifies "privacy" and "leakage" as key terms, associating them with the intent of "privacy concerns." The training process aligns the feature vectors with the correct intent labels by optimizing the loss function (such as cross-entropy), leveraging labeled data to improve the model's ability to distinguish intents, such as differentiating between "asking about prices" and "signing an agreement." The diversity of the dataset ensures that the model can handle domain-specific nuances, such as the term "title" in the real estate field, while remaining robust to various dialogue styles.

[0050] To improve performance, training data is augmented using data augmentation techniques, such as generating synthetic dialogues simulating rare scenarios or employing adversarial training to enhance robustness to noisy or ambiguous inputs. The model is periodically updated with new dialogue data to adapt to evolving language patterns or emerging intents, ensuring applicability in dynamic applications such as real estate agent training. The labeled dataset is carefully curated to balance positive and negative samples for each intent, preventing bias towards common intents. For example, in the real estate domain, the dataset includes sufficient samples of “privacy concerns” to ensure the model accurately detects privacy-related intents, crucial for generating metrics such as “privacy protection instructions.” This comprehensive training approach enables the intent recognition model to provide accurate and context-sensitive intent predictions, laying the foundation for subsequent dynamic metric generation and dialogue quality assessment.

[0051] Step 104: Input the intent recognition results and domain information set into the domain adaptation module to determine the evaluation criteria.

[0052] It should be noted that the domain adaptation module mentioned here can be a domain adaptation model or a thesaurus capable of keyword matching.

[0053] The evaluation criteria mentioned here are the basis for scoring the responses of the tested users. In this disclosed solution, the evaluation criteria are dynamically adjusted based on the intent recognition results and the domain information set, thereby making the evaluation results more accurate. The specific implementation scheme will be described in detail in subsequent embodiments, and will not be repeated here.

[0054] Step 105: Match and combine the assessment criteria with the preset script template to generate an assessment script; the assessment script includes scoring rules.

[0055] The script templates mentioned here are predefined structured frameworks. For example, they might be designed using the XML-based PlayScriptML language to organize the general structure of assessment scripts. The script templates include placeholders and general logic for dynamically inserting assessment criteria (assessment dimensions and scoring rules) and contextual information related to the dialogue scenario.

[0056] To better meet the needs of different users and scenarios, the script templates support dynamically binding assessment criteria to specific dialogue scenarios. For example, in the real estate sector, templates can be populated with indicators such as "privacy protection statement"; in the medical sector, they can be populated with indicators such as "patient care". This dynamism enables the system to generate highly matched assessment scripts based on assessment criteria and domain information.

[0057] The generated assessment script contains assessment criteria and corresponding scoring rules, which can be used to conduct quality assessments on the responses of the tested users.

[0058] Step 106: Receive the response dialogue from the tested user regarding the evaluation script.

[0059] After generating the script, further effective interactions can be performed with the tested user, such as asking questions to the tested user and receiving the tested user's responses in real time.

[0060] Step 107: Use the evaluation criteria to evaluate the quality of the response dialogue.

[0061] In practical applications, when evaluating the quality of a user's responses, a dynamically generated evaluation script is used. Through semantic analysis and scoring rule matching, the system assesses whether the user's (e.g., a real estate agent) responses meet predefined evaluation dimensions and scoring criteria, ultimately generating a quality score.

[0062] Based on the publicly available solutions described above, by utilizing models to sequentially perform intent recognition, domain adaptation, and structured evaluation criteria, the system can dynamically generate evaluation indicators (such as privacy protection statements and product information accuracy) that are highly matched to the scenario based on the dialogue content. This breaks through the static bottleneck of traditional preset indicators and significantly improves the accuracy of the evaluation. By combining the intent recognition results with domain information (preset or model-generated), the evaluation direction can be dynamically adjusted (e.g., focusing on "privacy protection" and "explanation of terms of service" in the sales domain) to ensure that the evaluation dimensions cover the core needs of the dialogue.

[0063] In one or more embodiments of this disclosure, feature vectors extracted from the dialogue text are input into a trained intent recognition model to determine the intent recognition result for the dialogue text, including: preprocessing the dialogue text; extracting feature vectors from the preprocessed dialogue text using a large language model; inputting the feature vectors into the intent recognition model and outputting the intent recognition result for the dialogue text.

[0064] In practical applications, the dialogue text undergoes preprocessing to transform the raw text into a structured format suitable for model processing, while removing irrelevant information to highlight semantic content. Preprocessing includes several sub-steps: removing noise information from the dialogue text, such as punctuation marks, special characters (e.g., emoticons, line breaks), and irrelevant filler words (e.g., "ah," "um"). For example, the dialogue text "Ms. Wang, hello! What is the area of ​​your house?" can be preprocessed to "Ms. Wang, hello, what is the area of ​​your house?". Next, a word segmentation tool is used to segment the text into words or phrases, such as "Ms. Wang / hello / house / area / is / how much", so that the model can perform semantic analysis on a word-by-word basis. Furthermore, the system performs part-of-speech tagging (e.g., "house / noun," "is / verb") and named entity recognition (e.g., recognizing "Ms. Wang" as a "person's name" and "house area" as an "entity") on the segmented text. After preprocessing, the dialogue text is ensured to be cleaned into a structured word sequence, providing high-quality input for subsequent feature extraction.

[0065] Subsequently, a large language model based on the Transformer architecture (such as BERT or GPT series) is used to extract feature vectors from the preprocessed dialogue text. The large language model semantically encodes the text through its encoder part, generating feature vectors that capture the dialogue context and semantic relationships. It should be noted that the large language model used to extract feature vectors from the dialogue text is pre-trained. During training, scenario-appropriate training samples (e.g., historical data labeled as positive and negative samples) need to be prepared in advance to enable the large language model to extract feature vectors more effectively.

[0066] The extracted feature vectors are input into the trained intent recognition model, which outputs the intent recognition results of the dialogue text.

[0067] The intent recognition models discussed here are typically based on the classifier portion of large language models (such as fully connected layers plus Softmax layers). During the training phase, they are optimized using real dialogue data and manually labeled data. The training data includes dialogue texts from different domains and their corresponding intent labels (such as "asking about housing information" and "privacy concerns"). For example, the training data could include dialogues labeled "asking about housing information" such as "What is the size of the house?", and dialogues labeled "privacy concerns" such as "I am worried about the leakage of my personal information".

[0068] During use, the feature vector is input into the classifier, which predicts the dialogue intent based on the similarity or probability distribution between the feature vector and the pre-trained intent labels. For example, the feature vector [0.23, -0.12, ..., 0.89] (representing "what is the house area") might be classified as "asking about house information" with a probability of 0.85. The classifier ensures the accuracy of intent prediction by optimizing the cross-entropy loss function. The output is a specific intent label, such as "asking about house information" or "privacy concerns," which is then passed to the subsequent domain adaptation module to generate evaluation criteria that match the dialogue scenario.

[0069] Based on the publicly available solutions mentioned above, it is known that by using a well-trained intent recognition model, combined with real dialogue data and manually labeled data, the system can quickly and accurately predict dialogue intent, support real-time dialogue analysis, and meet the timeliness requirements of dynamic scenarios (such as real estate agent training).

[0070] It should be noted that when performing domain identification, the domain adaptation module used can be either a matching module or a domain adaptation model. Specific examples will be provided below to illustrate these points.

[0071] In one or more embodiments of this disclosure, such as Figure 2 This is a schematic diagram of a process for constructing evaluation criteria according to an embodiment of this disclosure. Step 104, which involves constructing evaluation criteria using intent recognition results and a domain information set, includes: Step 1041: Extracting keywords from the dialogue text. Step 1042: Inputting the keywords into a domain adaptation module to determine the domain keywords that match the keywords in the domain information set. Step 1043: Determining the domain recognition result corresponding to the dialogue text based on the domain keywords. Step 1044: Determining the evaluation direction using the intent recognition results and the domain recognition results. Step 1045: Generating evaluation criteria based on the evaluation direction.

[0072] In practical applications, the dialogue text is real-time content generated by the interaction between users (such as real estate agents) and AI-simulated roles (such as homeowners), for example, "I'm worried about my personal information being leaked; how do you protect my privacy?". The system uses natural language processing techniques (such as keyword extraction algorithms or TF-IDF) to extract keywords from the preprocessed dialogue text, such as "privacy," "leakage," and "protection." These keywords are separated from the text using word segmentation tools and combined with part-of-speech tagging and named entity recognition to ensure accuracy.

[0073] Next, the extracted keywords are input into the domain adaptation module and matched with domain keywords in a predefined domain information set. The domain information set is a predefined structured database containing keywords and features from different domains. For example, keywords in the real estate domain include "house," "property rights," and "brokerage fee," while keywords in the customer service domain might include "complaint" and "return." The matching process determines the domain keywords corresponding to the dialogue text by calculating the semantic similarity or exact match of the keywords. For example, "privacy" and "protection" match the "privacy protection" feature in the real estate domain, thus confirming that the dialogue belongs to the real estate domain. Based on the matched domain keywords, the system generates a domain identification result, such as "real estate domain." Then, the system combines the intent recognition result (e.g., "privacy concerns") and the domain identification result ("real estate domain") to determine the evaluation direction through semantic analysis, such as focusing on "privacy protection related issues."

[0074] Based on the aforementioned publicly available solutions, keyword extraction and matching can accurately identify domain features in dialogue text. For example, in real estate dialogues, keywords such as "privacy" and "protection" can be extracted, and real estate domain features in the domain information set can be quickly matched to generate evaluation criteria highly relevant to the scenario. This method has low computational cost and is suitable for scenarios with clearly defined domain features.

[0075] In one or more embodiments of this disclosure, such as Figure 3 This is a schematic diagram illustrating another process for constructing evaluation criteria according to an embodiment of this disclosure. Step 104, which involves constructing evaluation criteria using intent recognition results and a domain information set, includes: Step 1046: Inputting the dialogue text into a trained domain adaptation model and outputting domain recognition results for the dialogue text. Step 1047: Determining the evaluation direction using the intent recognition results and the domain recognition results. Step 1048: Generating evaluation criteria based on the evaluation direction.

[0076] In practical applications, a domain adaptation model is a classification model based on a large language model (such as BERT or GPT series). It learns the linguistic features and patterns of different domains through training on a large amount of labeled domain text. For example, the model learns semantic patterns of words like "house" and "real estate agent" from training data in the real estate domain, and patterns like "service" and "complaint" in the customer service domain. During application, pre-processed dialogue text (such as "I'm worried about my personal information being leaked") is directly input into the domain adaptation model. The model generates semantic feature vectors of the text through an encoder and outputs domain recognition results based on a pre-trained classifier, such as "real estate domain." The training process uses supervised learning and optimizes the cross-entropy loss function to ensure the model can accurately distinguish between different domains. After outputting the domain recognition results, the system combines the intent recognition results (such as "privacy concerns") for semantic analysis to determine the evaluation direction, such as "privacy protection related assessment."

[0077] Based on the aforementioned publicly available solutions, the domain identification results are directly output through a pre-trained domain adaptation model, eliminating the keyword extraction step. Leveraging the semantic understanding capabilities of large-scale language models, end-to-end domain classification is achieved, making it particularly suitable for handling complex or ambiguous dialogue scenarios, such as hybrid dialogues containing multi-domain features. The combination of these two methods provides a flexible approach to domain identification, allowing for the selection of appropriate implementation methods based on the complexity of the dialogue scenario and system resources, thus improving the system's adaptability and robustness.

[0078] In one or more embodiments of this disclosure, the evaluation direction is determined by using the intent recognition result and the domain recognition result; the evaluation criteria are generated according to the evaluation direction, including: combining the intent recognition result and the domain recognition result to obtain the evaluation direction; splitting the evaluation direction into multiple evaluation dimensions; determining the scoring rules corresponding to the multiple evaluation dimensions based on a preset rule base or semantic analysis; and constructing a structured evaluation criteria using the evaluation dimensions and the corresponding scoring rules.

[0079] In practical applications, semantic analysis techniques, combined with large-scale language models based on the Transformer architecture (such as BERT or GPT series), are used to semantically fuse intent recognition results and domain recognition results to generate assessment directions. Specifically, the system utilizes the self-attention mechanism of a large-scale language model (which can be a model used to analyze dialogue text or a model trained independently using historical data) to analyze the semantic relationship between intent and domain. For example, combining the intent of "privacy concerns" with the "real estate domain" determines the assessment direction as "privacy protection related assessment." This process may involve concatenating or weightedly fusing the feature vectors of intent and domain to generate a comprehensive semantic representation, which guides the generation of subsequent assessment criteria.

[0080] Next, the evaluation direction is broken down into multiple evaluation dimensions to refine the assessment content. For example, based on the characteristics of the dialogue scenario and predefined domain knowledge, it can be decomposed into specific evaluation dimensions, such as "mention of privacy policy," "explanation of privacy risks," and "information transparency." These dimensions are determined by analyzing the semantic needs and domain characteristics of the dialogue scenario, possibly leveraging pre-defined rules in the domain information set or the semantic analysis capabilities of a large language model. For instance, in the real estate domain, the system recognizes that "privacy protection" typically involves a clear statement of the privacy policy and an explanation of the risks, thus generating the aforementioned dimensions. The decomposition process may be implemented using clustering algorithms or semantic segmentation techniques to ensure that each dimension focuses on a specific aspect of dialogue quality while maintaining the independence and complementarity between dimensions.

[0081] Furthermore, the system determines the scoring rules for each evaluation dimension based on a pre-set rule base or semantic analysis. The pre-set rule base is a structured database containing general scoring rules for different domains and intentions. For example, rules for the real estate domain might include "2 points if privacy policy is mentioned; 0 points if not mentioned." The rule base is built based on real-world dialogue data and expert knowledge, covering scoring standards for common scenarios.

[0082] For complex or non-predefined scenarios, the system uses a large language model (trained using historical data as training samples) for semantic analysis and dynamically generates scoring rules. For example, for the "explanation of privacy risks" dimension, the system analyzes whether the dialogue text contains keywords such as "data encryption" or "information security" or semantically similar expressions, and generates the rule "1 point if privacy risks are explained, 0 points if not explained".

[0083] The construction process utilizes template tools or XML generators to ensure that the evaluation criteria are formatted uniformly and parsable. The system verifies the completeness of the evaluation criteria, such as checking whether each dimension corresponds to clear scoring rules and whether it is consistent with the dialogue scenario. If the dialogue content changes (e.g., from "privacy concerns" to "price inquiries"), the system can re-execute the above steps to dynamically update the evaluation criteria.

[0084] Based on the publicly available solutions described above, by fusing semantic intent and domain recognition results, evaluation directions that highly match the dialogue scenario are generated, overcoming the limitations of traditional static indicators in adapting to dynamic dialogue scenarios. Breaking down the evaluation directions into multiple evaluation dimensions refines the granularity of the assessment, making dialogue quality evaluation more comprehensive and specific, accurately reflecting multiple aspects of the dialogue. The combination of a pre-defined rule base and semantic analysis to determine scoring rules leverages the efficiency and stability of predefined rules while dynamically adapting to the needs of complex or new scenarios through semantic analysis, enhancing the system's flexibility and robustness. Furthermore, the generated structured evaluation standards (such as JSON or XML format) are uniform and parsable, facilitating matching with script templates and real-time dialogue monitoring, improving the automation and consistency of the evaluation process.

[0085] In one or more embodiments of this disclosure, the evaluation criteria are matched and combined with the script template to generate an evaluation script, including: receiving a preset script template; and generating an evaluation script by replacing the placeholders in the script template with the evaluation dimensions and scoring rules in the evaluation criteria.

[0086] The script templates mentioned here are pre-defined and typically stored in the system's template library. They predefine the general structure of evaluation scripts, such as dialogue scene descriptions, evaluation dimensions, scoring rules, and scoring logic. The templates are stored in XML format, offering high readability and extensibility, and supporting cross-domain dialogue evaluation needs. During application, the corresponding script template is matched using the evaluation criteria.

[0087] The assessment criteria's dimensions and scoring rules are matched against placeholders in the script template. The matching process includes mapping the semantic content of the assessment dimensions (such as "mentioned in the privacy policy") to placeholders in the template. <evaluationdimension>Placeholder. Bind the rating rules (e.g., "2 points for mentioning the privacy policy") to the corresponding... <scoringrule>Placeholders. Fill in the blanks based on the context of the dialogue (e.g., a real estate contract signing scenario). <scenecontext>Ensure that the script is highly relevant to the dialogue scenes.

[0088] As mentioned earlier, by parsing a PlayScriptML-based script template, structured assessment dimensions and scoring rules can be dynamically populated into placeholders, and an assessment script can be generated by combining the dialogue scenario context. The matching process relies on a rule engine or XML parser to ensure that the assessment criteria are semantically consistent with the current scenario to be assessed.

[0089] Based on the publicly available solutions described above, the dynamic generation mechanism enables real-time compilation of evaluation scripts, dynamically adjusting evaluation dimensions and scoring rules according to real-time changes in the dialogue content (such as a homeowner shifting from inquiring about prices to privacy issues). This ensures more accurate evaluation results, promptly reflecting key issues in the dialogue (such as whether the agent effectively responded to privacy concerns), and providing support for real-time feedback and optimization. It effectively overcomes the limitations of traditional static evaluation scripts, achieving precise assessment and real-time monitoring of dialogue quality.

[0090] In one or more embodiments of this disclosure, receiving a response dialogue from a user under test to an evaluation script; and performing quality evaluation on the response dialogue using the scoring rules in the evaluation script, includes: receiving a response dialogue from a user under test to an evaluation script; performing semantic similarity matching between the response dialogue and the scoring rules; and executing the corresponding scoring rules based on the matching results to generate a quality evaluation result.

[0091] Specifically, it receives responses from users being tested (such as real estate agents) to the evaluation script. These responses are real-time dialogues entered by users when interacting with AI-simulated characters (such as homeowners), such as: "Our company has strict privacy protection measures to ensure the security of your information." The evaluation script is a structured XML document (such as based on PlayScriptML), which includes evaluation dimensions (such as "mention of privacy policy"), scoring rules (such as "2 points for mentioning privacy policy, 0 points for not mentioning it") and dialogue scenario context (such as real estate contract signing scenario).

[0092] The response dialogue is preprocessed to semantically match it with the scoring rules in the evaluation script. Preprocessing mainly includes noise reduction and word segmentation.

[0093] After preprocessing, semantic similarity matching is performed. Specifically, the semantic understanding capabilities of large language models (LLMs, such as BERT or GPT series) are used to perform semantic similarity matching between the preprocessed response dialogue and the scoring rules in the evaluation script.

[0094] The specific steps include: inputting the response dialogue into the LLM encoder to generate a semantic feature vector. For example, "Our company has strict privacy protection measures" generates a 768-dimensional feature vector that captures the semantics related to "privacy protection".

[0095] Key conditions in the scoring rules (such as "mention of privacy policy") are also encoded into feature vectors using LLM. For example, "mention of privacy policy" might correspond to the feature vector of the predefined template statement "The company provides a privacy policy".

[0096] The semantic similarity between the feature vector of the response dialogue and the feature vector of the scoring rule is calculated, typically using cosine similarity or Euclidean distance. For example, if the similarity between the vector of the response dialogue and the vector of the "privacy policy mention" rule is higher than a preset threshold (e.g., 0.8), then the rule is considered to be satisfied.

[0097] Semantic matching may involve multi-dimensional analysis, such as simultaneously checking the matching of two dimensions: "mention in privacy policy" and "explanation of privacy risks".

[0098] Based on the semantic similarity matching results, the scoring rules in the evaluation script are executed to generate a score. For example, if the response "Our company has strict privacy protection measures" matches the "Mentioned in privacy policy" rule, the system assigns 2 points. If the response does not mention "Explanation of privacy risks," then this dimension receives 0 points. Scoring rules may include "2 points for mentioning, 0 points for not mentioning," or "1 point for a complete explanation of privacy risks, 0.5 points for a partial explanation, and 0 points for no explanation."

[0099] A weighted total score is calculated based on the weights of the evaluation dimensions (e.g., "Privacy Policy Mention" weighted at 0.6, "Explanation of Privacy Risks" weighted at 0.4). The scores from all evaluation dimensions are then aggregated to generate a final result for the dialogue quality. This result is categorized as "Acceptable" (total score ≥ 3) or "Unacceptable" (total score < 3). This evaluation result is stored in the system and fed back to the user (e.g., a real estate agent) via API or interface to optimize their dialogue strategy. Based on the above, after obtaining the evaluation script, the system can further receive responses from the tested user. While engaging in dialogue with the tested user according to the evaluation script, the scoring rules within the script can be used to score the matching results. Different scoring rules can be developed for different evaluation scripts, resulting in more accurate quality evaluation results.

[0100] Based on the same idea, this disclosure also proposes a dialogue assessment system. For example... Figure 4 This is a schematic diagram of the dialogue evaluation system provided in this disclosure. The dialogue evaluation system includes an intent recognition module 41, a structuring module 42, a script template compilation module 43, and a dialogue detection module 44.

[0101] The intent recognition module 41 is used to receive dynamically generated dialogue text and a preset domain information set; extract feature vectors contained in the dialogue text; input the extracted feature vectors into the trained intent recognition model to determine the intent recognition result for the dialogue text.

[0102] The structured module 42 is used to construct evaluation criteria using intent recognition results and domain information sets.

[0103] The script template compilation module 43 is used to receive script templates, match and combine the evaluation criteria with the script templates, and generate evaluation scripts.

[0104] The dialogue detection module 44 is used to receive the response dialogue from the tested user to the evaluation script; and to evaluate the quality of the response dialogue using the evaluation criteria.

[0105] The intent recognition module 41 is used to preprocess the dialogue text, extract feature vectors from the preprocessed dialogue text using a large language model, input the feature vectors into the intent recognition model, and output the intent recognition result of the dialogue text.

[0106] The intent recognition module 41 is used to extract keywords from the dialogue text; input the keywords into the domain adaptation module to determine the domain keywords that match the keywords in the domain information set; determine the domain recognition result corresponding to the dialogue text based on the domain keywords; determine the evaluation direction using the intent recognition result and the domain recognition result; and generate evaluation criteria based on the evaluation direction. The intent recognition module 41 is also used to input the dialogue text into the trained domain adaptation model and output the domain recognition result for the dialogue text; determine the evaluation direction using the intent recognition result and the domain recognition result; and generate evaluation criteria based on the evaluation direction.

[0107] The structured module 42 is used to combine the intent recognition result with the domain recognition result to obtain the evaluation direction; to break down the evaluation direction into multiple evaluation dimensions; to determine the scoring rules corresponding to the multiple evaluation dimensions based on the preset rule base or semantic analysis; and to construct a structured evaluation standard using the evaluation dimensions and the corresponding scoring rules.

[0108] The script template compilation module 43 is used to receive preset script templates; after replacing the placeholders in the script templates with the evaluation dimensions and scoring rules in the evaluation criteria, the evaluation script is generated.

[0109] The dialogue detection module 44 is used to receive the response dialogue from the tested user to the evaluation script; perform semantic similarity matching between the response dialogue and the scoring rules; execute the corresponding scoring rules according to the matching results, and generate quality evaluation results.

[0110] The dialogue evaluation system may also include a training module 45. The training module 45 is used to train the intent recognition model based on real dialogue data and manually labeled data. The training data includes dialogue texts from different domains and their corresponding intent labels.

[0111] Figure 5 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure.

[0112] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0113] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0114] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. "Readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of readable storage media include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0115] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0116] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0117] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0122] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.< / scenecontext> < / scoringrule> < / evaluationdimension>

Claims

1. A dialogue assessment method, characterized in that, The method includes: Receive dialogue text, a pre-set set of domain information, and a script template; Extract the feature vectors contained in the dialogue text; Based on the feature vector extracted from the dialogue text, the intent recognition result for the dialogue text is determined; Evaluation criteria are constructed using the intent recognition results and the domain information set; The evaluation criteria are matched and combined with the script template to generate an evaluation script; wherein, the evaluation script includes scoring rules; Receive responses from the tested user regarding the assessment script; The quality of the response dialogue is evaluated using the scoring rules in the assessment script.

2. The dialogue assessment method according to claim 1, characterized in that, Based on the feature vector extracted from the dialogue text, the intent recognition result for the dialogue text is determined, including: The dialogue text is preprocessed; The feature vectors are extracted from the preprocessed dialogue text using a large language model; The feature vector is input into the intent recognition model, and the intent recognition result of the dialogue text is output.

3. The dialogue assessment method according to claim 1, characterized in that, The construction of evaluation criteria using the intent recognition results and the domain information set includes: Extract keywords from the dialogue text; The keyword is input into the domain adaptation module to determine the domain keywords in the domain information set that match the keyword; Determine the domain recognition result corresponding to the dialogue text based on the domain keywords; The evaluation direction is determined using the intent recognition results and the domain recognition results; Evaluation criteria are generated based on the evaluation direction.

4. The dialogue assessment method according to claim 1, characterized in that, The construction of evaluation criteria using the intent recognition results and the domain information set includes: The dialogue text is input into the domain adaptation model, and the domain recognition result for the dialogue text is output. The evaluation direction is determined using the intent recognition results and the domain recognition results; Evaluation criteria are generated based on the evaluation direction.

5. The dialogue assessment method according to claim 3 or 4, characterized in that, The evaluation direction is determined by utilizing the intent recognition results and the domain recognition results; Based on the aforementioned evaluation direction, evaluation criteria are generated, including: The evaluation direction is obtained by combining the intent recognition result with the domain recognition result; The assessment direction is broken down into multiple assessment dimensions; Based on a pre-set rule base or semantic analysis, the scoring rules corresponding to the multiple evaluation dimensions are determined respectively; The structured evaluation criteria are constructed using the evaluation dimensions and the corresponding scoring rules.

6. The dialogue assessment method according to claim 5, characterized in that, The step of matching and combining the assessment criteria with the script template to generate an assessment script includes: Receive the preset script template; The evaluation script is generated by replacing the placeholders in the script template with the evaluation dimensions and scoring rules in the evaluation criteria.

7. The dialogue assessment method according to claim 6, characterized in that, The process of receiving responses from the tested user to the assessment script and evaluating the quality of the responses using the assessment criteria includes: Receive responses from the tested user regarding the assessment script; Perform semantic similarity matching between the response dialogue and the scoring rules; Based on the matching results, the corresponding scoring rules are executed to generate quality assessment results.

8. The dialogue assessment method according to claim 2, characterized in that, The intent recognition model is trained based on real dialogue data and manually labeled data as training samples. The training samples include dialogue texts from different domains and their corresponding intent labels.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images