Conversational virtual assistant tuning method and system

Abstract

The application provides a dialog virtual assistant tuning method and system, and relates to the technical field of artificial intelligence and natural language processing, and the method comprises the following steps: determining a target network structure from a preset neural network set according to feature information of training corpus of the dialog virtual assistant; performing hyperparameter optimization and model training according to the training corpus to obtain a target model under a target hyperparameter combination; extracting a target sample set to be labeled from a log database; incorporating the target sample set labeled by manual labeling into a training data set, and performing model incremental updating of the target model according to the training data set and the target hyperparameter combination. The application improves the universality through adaptive selection of the network structure, improves the tuning efficiency through automatic hyperparameter tuning, guarantees the model performance, forms a complete closed loop of data, model, feedback and updating through the active learning mechanism, realizes the continuous evolution of the model, and thus improves the prediction accuracy of the model.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing technology, and in particular to a method and system for optimizing a conversational virtual assistant. Background Technology

[0002] With the rapid development of artificial intelligence technology, conversational virtual assistants have been widely applied in various fields such as customer service, e-commerce, and healthcare. Natural Language Understanding (NLU), as the core module of virtual assistants, is mainly responsible for intent recognition and entity extraction from user-input text. Its performance directly affects the accuracy and fluency of human-computer interaction. Currently, most mainstream NLU systems adopt solutions based on a single model structure, such as pre-trained language models like Long Short-Term Memory networks (LSTM) or Bidirectional Encoder Representations from Transformers (BERT).

[0003] In existing technologies, the architecture and parameter configuration of these models are typically determined manually before deployment, employing static model parameters and a fixed structure. In practice, developers need to train the model using annotated corpora from a specific domain, manually adjusting hyperparameters to optimize performance metrics. Once the trained model is deployed to the production environment, its internal structure and parameters remain fixed. If the corpus context changes or user expression becomes more diverse, model performance is prone to decline. This static model lacks adaptive adjustment capabilities and has poor fault tolerance when faced with spelling errors, non-standard expressions, or industry-specific terminology in user input.

[0004] Furthermore, existing systems often misjudge intent when faced with complex sentence structures such as negation and transitions due to a lack of contextual understanding. Different industry scenarios have significantly different requirements for model structure; for example, the medical field requires stronger entity recognition capabilities, while customer service scenarios prioritize intent classification accuracy. However, under the current technological framework, adjusting the model structure requires retraining the entire system, resulting in high operational costs. Hyperparameter optimization relies on engineer experience, achieved through repeated trial and error, which is inefficient and difficult to guarantee the optimal solution. The model inference process lacks transparency, posing application risks in fields with strict compliance requirements such as finance and healthcare. More importantly, existing systems lack real-time performance monitoring and feedback mechanisms, making it impossible to dynamically optimize based on changes in online data distribution, causing model performance to gradually decline over time. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for optimizing a conversational virtual assistant, so as to at least solve one of the above-mentioned problems.

[0006] In a first aspect, the present invention provides a method for optimizing a conversational virtual assistant, comprising: Based on the feature information of the training corpus of the conversational virtual assistant, the target network structure is determined from the preset neural network set; Based on the training corpus, the initial model of the target network structure is optimized for hyperparameters and trained using Bayesian optimization algorithm or tree structure parameter estimator algorithm to obtain the target model under the optimal combination of target hyperparameters. A target sample set to be labeled is extracted from a pre-set log database for manual labeling. The log database stores log data obtained by the conversational virtual assistant using the target model for online intent prediction. The target samples in the target sample set include input content, confidence value, and intent label information. The manually labeled target sample set is incorporated into the training dataset, and the target model is incrementally updated based on the training dataset and the target hyperparameter combination.

[0007] In an optional implementation, the conversational virtual assistant tuning method further includes: When a local interpretation request is received, the feature contribution of the single-step model prediction data in the local interpretation request is calculated based on the Shapley value mechanism to obtain the local interpretation result. The single-step model prediction data includes the target model, the input content and its prediction result, and the local interpretation result includes the contribution value of each feature in the input content to the intention prediction. When a global explanation request is received, the feature influence of the training corpus prediction data in the global explanation request is calculated based on the Shapley value mechanism to obtain the global explanation result. The training corpus prediction data includes the target model, the training corpus and its prediction results, and the global explanation result includes the average influence ranking of each feature in the training corpus on the target model and the positive and negative key features corresponding to each intent.

[0008] In optional implementations, the target model is a general multilingual model or multiple language-specific models. The general multilingual model is obtained by uniformly encoding multilingual corpora into a shared vector space for training. The conversational virtual assistant optimization method also includes: Perform language recognition on the current input content from the user to obtain the target language; Based on the target language, the current input content is preprocessed to obtain preprocessed content; the preprocessing includes one or more of the following: multimodal conversion, spell correction processing, and negation expression recognition. The intent prediction of the preprocessed content is obtained by using a general multilingual model or a language-specific model corresponding to the target language.

[0009] In an optional implementation, the target network structure is determined from a preset set of neural networks based on the feature information of the training corpus of the conversational virtual assistant, including: Feature information is extracted from the training corpus to obtain corpus features, which include one or more of sentence length distribution, entity density, intent quantity, and intent entropy. Based on the characteristics of the corpus, the target network structure is selected from the neural network set through a preset rule template or heuristic model.

[0010] In an optional implementation, a target network structure is selected from a neural network ensemble based on corpus features using a preset rule template or heuristic model, including: Based on the corpus features, the task complexity is calculated using the following formula: ; in, C Indicates task complexity. L avg Here, represents the average sample length, Entropy(Intent) represents the entropy of the intent label distribution, and Entity_Density represents the average number of entities per sentence. α 1. α 2. α 3 represents the preset weight; The target network structure is selected from the neural network set based on the complexity of the task.

[0011] In an optional implementation, based on the training corpus, a Bayesian optimization algorithm or a tree structure parameter estimator algorithm is used to optimize the hyperparameters and train the initial model of the target network structure to obtain the target model with the optimal combination of target hyperparameters, including: The training corpus is preprocessed to obtain preprocessed training corpus; the preprocessing includes one or more of the following: multimodal transformation, spelling correction, negation expression recognition, and entity placeholder replacement. Based on the preprocessed training corpus, multiple training tasks are scheduled using Bayesian optimization or tree-structured parameter estimator algorithms, and the model performance of different hyperparameter combinations under a preset hyperparameter search space is evaluated to determine the target model with the optimal target hyperparameter combination.

[0012] In an optional implementation, the target sample set to be labeled is extracted from a preset log database, including: According to the preset scheduling frequency, log data of unlabeled sample sets within a preset time range are extracted from the log database; Based on the log data of the unlabeled sample set, use one or more of the following filtering strategies: maximum entropy, minimum confidence, and minimum margin to select the first sample from the unlabeled sample set and add it to the target sample set. The unlabeled sample set is vector-encoded and clustered. A second sample is selected from each cluster of the clustering results and added to the target sample set.

[0013] In an optional implementation, the target sample set involves multiple language categories; after extracting the target sample set to be labeled from a preset log database, the conversational virtual assistant optimization method further includes: The annotation tasks are distributed to the samples in the target sample set according to language category, so that the annotators can handle the annotation tasks of the corresponding language category samples.

[0014] Secondly, the present invention provides a conversational virtual assistant optimization system, comprising: The model structure selection module is used to determine the target network structure from a preset set of neural networks based on the feature information of the training corpus of the conversational virtual assistant. The hyperparameter optimization and training module is used to optimize the hyperparameters and train the initial model of the target network structure based on the training corpus using the Bayesian optimization algorithm or the tree structure parameter estimator algorithm, so as to obtain the target model under the optimal combination of target hyperparameters. The active learning module is used to extract a set of target samples to be labeled from a pre-set log database for manual labeling. The log database stores log data obtained by the conversational virtual assistant using the target model for online intent prediction. The target samples in the target sample set include input content, confidence values, and intent label information. The manually labeled target sample set is incorporated into the training dataset, and the target model is incrementally updated based on the training dataset and the target hyperparameter combination.

[0015] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the conversational virtual assistant tuning method of any of the foregoing embodiments.

[0016] This invention provides a method and system for optimizing a conversational virtual assistant. Based on the feature information of the training corpus of the conversational virtual assistant, it determines the target network structure from a pre-set neural network set. According to the training corpus, it uses a Bayesian optimization algorithm or a tree-structured parameter estimator algorithm to optimize the hyperparameters and train the initial model of the target network structure, obtaining a target model with the optimal combination of target hyperparameters. It extracts a target sample set to be labeled from a pre-set log database for manual labeling. The log database stores log data obtained by the conversational virtual assistant using the target model for online intent prediction. The target samples in the target sample set include input content, confidence values, and intent label information. The manually labeled target sample set is incorporated into the training dataset, and the target model is incrementally updated based on the training dataset and the target hyperparameter combination. This approach improves versatility through adaptive selection of the network structure, enhances optimization efficiency through automated hyperparameter tuning, ensures model performance, and forms a complete closed loop of data, model, feedback, and update through an active learning mechanism, achieving continuous model evolution and thus improving the model's predictive accuracy. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a conversational virtual assistant optimization method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating another method for optimizing a conversational virtual assistant provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a conversational virtual assistant tuning system provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, there is a need to build a multi-model NLU tuning system with dynamic tuning capabilities, input robustness, scene adaptability, and interpretability to support the intelligent understanding and continuous evolution of conversational virtual assistants. Based on this, the present invention provides a conversational virtual assistant tuning method and system applicable to the optimization and deployment of natural language understanding models under multiple languages, industries, and input modes. It encompasses several key technologies, including adaptive selection of neural network structures, automatic hyperparameter search and optimization, spell correction integration, negative semantic direction recognition, and entity placeholder generalization training. This can improve the accuracy, stability, and generalization ability of conversational virtual assistants in intent recognition and entity extraction.

[0021] This invention provides an intelligent tuning and active learning closed-loop solution for natural language understanding systems (i.e., conversational virtual assistants). By constructing a multi-dimensional semantic understanding optimization process, including spell correction, negation recognition, entity placeholder generalization, adaptive selection of neural network structure, automatic hyperparameter optimization (HPO), interpretable prediction output, and an active learning closed-loop feedback mechanism, it achieves intelligent construction and continuous evolution of the natural language understanding model. User input, after error correction and structural enhancement, enters the model tuning engine. The system automatically selects appropriate model architecture and training parameters based on data features and outputs structured results and word contribution explanations during the prediction phase. When the system identifies low-confidence input, it pushes the data into the active learning module, where it is quickly labeled by humans and added to the training data pool, triggering incremental model retraining and forming a continuously self-optimizing closed-loop system. Compared with existing technologies, this invention significantly improves the model's adaptability to real user corpora, prediction accuracy, and model interpretability.

[0022] The present invention proposes a multi-dimensional model tuning and continuous learning closed-loop scheme for natural language understanding systems. Through automatic structure selection, hyperparameter search, input correction, active learning feedback, multilingual adaptation and predictive interpretation mechanisms, it effectively solves the key technical problems existing in the prior art and significantly improves the intelligence, adaptability and application breadth of conversational virtual assistants.

[0023] The embodiments of the present invention mainly solve the following problems: 1. The problem of inefficiency and reliance on manual tuning of natural language understanding models: In existing NLU systems, model structure and training hyperparameters are usually set based on experience, lacking automated optimization capabilities, resulting in bottlenecks in performance improvement, and are time-consuming and labor-intensive.

[0024] 2. The model is sensitive to spelling errors, negative expressions, and entity differences in real user input, and has insufficient robustness: User input often contains typos, negative sentences, or different entity expressions (such as brand names, card types, etc.). Traditional models are prone to misjudging or missing key information, which affects the accuracy of intent recognition.

[0025] 3. Lack of adaptability to multiple network structures and corpora: Different application scenarios (such as customer service, sales, and finance) have significantly different corpus features, and fixed model structures are difficult to take into account diversity. The most suitable neural network architecture needs to be selected according to the characteristics of the task.

[0026] 4. The problem of natural language understanding models struggling to continuously self-optimize after deployment: Traditional models lack an effective feedback mechanism for user dialogue data after deployment, and cannot collect low-performance samples in a timely manner, perform active learning and incremental updates, resulting in a gradual decline in performance.

[0027] 5. High training costs and complex deployment of models in multilingual environments: Training models separately for each language or using translation intermediate layers can easily lead to performance loss. This invention aims to improve cross-language understanding capabilities through language-independent modeling and multilingual semantic routing mechanisms.

[0028] 6. Lack of high-quality explanation and debugging capabilities: Existing NLU systems have difficulty explaining to business or operations personnel "why the model makes this judgment", which affects debugging efficiency and compliance assessment, especially in high-audit industries such as finance and healthcare.

[0029] To facilitate understanding of this embodiment, a detailed description of a conversational virtual assistant optimization method disclosed in this embodiment of the invention will be provided first.

[0030] This invention provides a method for optimizing a conversational virtual assistant, which can be executed by an electronic device with data processing capabilities. See also... Figure 1 The diagram shows a flowchart of a conversational virtual assistant optimization method, which mainly includes the following steps S110 to S140: Step S110: Determine the target network structure from a preset neural network set based on the feature information of the training corpus of the conversational virtual assistant.

[0031] This invention enables adaptive selection of model structure, automatically choosing a suitable network structure from a pre-defined set of neural networks based on the feature information of the training corpus. The criteria for model structure selection may include the average length of the input sentences, entity density (such as term density), number of intents, and historical training performance. Optionally, the set of neural networks may include at least: Multi-Layer Perceptron (MLP), Long Short-Term Memory (LSTM) network, Transformer, BERT (Bidirectional Encoder Representations from Transformers), and Zero-Shot Embedding.

[0032] This invention provides a model switching mechanism that supports the parallel deployment of multiple network structures for different intent subsets. It can automatically select a suitable neural network structure (such as MLP, LSTM, Transformer, BERT, or Zero-shot) based on features of the training data, such as sentence length, entity density (e.g., domain terminology density), number of intents, and intent entropy.

[0033] Features of the training corpus (such as sentence length distribution, number of intents, and intent entropy) can be extracted; network structures can be recommended based on rule templates or heuristic models; and the structure configuration can be injected into the training process. Based on this, in some possible embodiments, the above step S110 may include: extracting feature information from the training corpus to obtain corpus features, which include one or more of sentence length distribution, entity density, number of intents, and intent entropy; and selecting a target network structure from the neural network set based on the corpus features using a preset rule template or heuristic model.

[0034] In one possible implementation, when selecting the target network structure based on corpus features, the task complexity can be calculated first using the following formula based on the corpus features: ; in, C Indicates task complexity. L avg The average length of the sample (corresponding to the sentence length distribution mentioned above), Entropy(Intent) represents the entropy of the intent label distribution (corresponding to the intent entropy mentioned above), and Entity_Density represents the average number of entities per sentence (corresponding to the entity density mentioned above). α 1. α 2. α3 represents the preset weight; Then, based on the complexity of the task, the target network structure is selected from the neural network set according to the preset rule template or heuristic model.

[0035] In practical implementation, a data-driven neural network selector (also known as a structure recommender or model configuration generator) can be built to perform structured analysis on the training corpus. Based on features such as average sentence length, lexical complexity (e.g., entity density), and the number of intents, the most suitable model type can be called, for example: MLP: Suitable for phrase intent classification; LSTM: Suitable for sequential expressions, such as customer service inquiries; Transformer: Suitable for long texts and complex logical expressions, such as sales lead generation; BERT / Zero-shot model: suitable for low-corpus or cold-start tasks.

[0036] An example of a model selection rule can be shown below: C<1.0: Select MLP; 1.0≤C<2.0: Choose LSTM; C≥2.0: Select Transformer or BERT.

[0037] Step S120: Based on the training corpus, use the Bayesian optimization algorithm or the tree structure parameter estimator algorithm to perform hyperparameter optimization and model training on the initial model of the target network structure, and obtain the target model under the optimal combination of target hyperparameters.

[0038] This invention introduces an automated hyperparameter search engine that intelligently explores combinations of model hyperparameters to find the optimal configuration, reducing the cost of repeated manual trial and error. The automated HPO engine is responsible for intelligently exploring model hyperparameter combinations. It receives the hyperparameter search space and optimization objective specified by the AI ​​engineer, evaluates the model performance of different parameter sets by scheduling multiple training tasks, and ultimately produces the optimal parameter configuration and model snapshot.

[0039] In some possible embodiments, the optimal parameter combination can be searched within a predefined hyperparameter space based on a Bayesian optimization algorithm. These hyperparameters may include the learning rate (lr), number of training epochs, dropout ratio, and hidden layer dimension. Specifically, based on the training corpus and the predefined hyperparameter search space, the initial model of the target network structure can be trained and evaluated multiple times under different hyperparameter combinations to obtain models and their performance metrics under different hyperparameter combinations. The model with the optimal hyperparameter combination is then determined as the target model under the target hyperparameter combination.

[0040] In other possible embodiments, the TPE (Tree-Structured Parzen Estimator) Bayesian optimization algorithm can be used to search for the optimal parameter combination in a predefined hyperparameter space. It supports configuring the objective function (such as accuracy, F1 score) and automatic early stopping. Hyperparameters include, but are not limited to: learning rate, dropout rate, hidden size, embedding dimension, number of epochs, batch size, etc.

[0041] In practical implementation, TPE within the Bayesian Optimization (BO) framework can be used as the primary hyperparameter optimization algorithm. TPE models historical experimental results, fitting hyperparameter distributions for "high-performance configurations" and "low-performance configurations" respectively, and uses these two distributions to infer the next set of parameters worth trying. Compared to grid search or random search, Bayesian methods such as TPE are more efficient in cases of complex parameter spaces or nonlinear relationships between parameters, and can approach the optimum with fewer attempts.

[0042] The technology stack selection can be as follows: integrate open-source hyperparameter optimization libraries (such as Optuna, Hyperopt) to leverage their mature TPE implementations, or choose Skopt (Scikit-Optimize) Gaussian process optimization as a supplement. The tuning console allows AI engineers to select optimization algorithms (TPE, Gaussian Process Bayes Optimization, etc.) and set search parameters (maximum number of iterations, initial number of random samples, and early stopping strategy, etc.).

[0043] The input to Automatic Hyperparameter Search (HPO) can be: the hyperparameter search space definition and the optimization objective function. The search space is specified by the user as the range / set of each hyperparameter (e.g., the learning rate in 1e^(-1 / 2)). -4 to 1e -2 The values ​​can be discrete between a certain range; the number of neurons in the hidden layer can be 50-300, etc. The optimization objective function is defined as the model performance metric on the validation set (such as the F1 score for intent classification or the overall accuracy).

[0044] The output of the Automatic Hyperparameter Search (HPO) can be: the optimal combination of hyperparameters and the corresponding model performance evaluation results. The output may also include parameter-performance relationship data (used to generate hyperparameter sensitivity analysis plots), as well as the top N sets of high-performance alternative parameters for reference.

[0045] The workflow of Automatic Hyperparameter Search (HPO) can be summarized as follows: During HPO engine initialization, several parameter combinations are randomly selected to train the model and obtain performance data as "prior" data. Then, a Bayesian optimization loop is entered. The TPE method, based on the current experimental history, divides the results into two groups according to performance (good or bad), constructs probability distribution estimates for the parameters, and selects the next parameter point that maximizes the difference between the "good" and "bad" distributions for evaluation. This loop continues until the iteration limit is reached or performance convergence stops. Throughout the process, the engine fully utilizes parallel computing resources, allowing multiple models to be trained simultaneously to accelerate evaluation.

[0046] To adapt to multimodal processing and improve the robustness of user input recognition, step S120 may include: preprocessing the training corpus to obtain preprocessed training corpus; wherein, the preprocessing includes one or more of multimodal transformation, spelling correction processing, negation expression recognition, and entity placeholder replacement; based on the preprocessed training corpus, multiple training tasks are scheduled by using a Bayesian optimization algorithm or a tree structure parameter estimator algorithm, and the model performance of different hyperparameter combinations under a preset hyperparameter search space is evaluated to determine the target model under the target hyperparameter combination with the best performance.

[0047] The aforementioned multimodal conversion can include converting image-based input to text and converting speech to text. Embodiments of this invention can achieve multimodal support: in terms of multimodal expansion, a text-visual-speech fusion strategy can be adopted, allowing the model to receive signals other than text.

[0048] Specifically, for image-based inputs: pre-trained visual models (such as OpenAI's 1000 Questions) can be integrated to convert images into semantic vectors shared with text, which can then be used as model input along with the text; alternatively, image classification / detection models can be used to first identify object and scene labels in the image, serving as additional intent slot features. For example, if a user uploads a photo of a damaged product to consult customer service, the system can identify that the photo contains the label "broken phone screen," and combine this with the text description to determine the intent "repair request and claim."

[0049] For voice input: a speech recognition module can be integrated to convert audio into a text sequence, which is then processed by the model, consistent with the pure text approach. Furthermore, metadata such as audio confidence or speaker sentiment can be retained as supplementary features for downstream decision-making (e.g., sentiment analysis influences response tone).

[0050] Fusion decision: When the input contains multimodal information (such as a user message with text description and attached photo), the system's decision engine will fuse the judgments from the text and the image.

[0051] The aforementioned spelling correction processing refers to automatically replacing misspelled words in user input based on dictionary definitions and semantic similarity; it may include: in the preprocessing stage, correcting potential spelling errors by combining edit distance and word vector similarity. Optionally, for w 1. w 2. The edit distance between two words can be calculated. d ( w 1, w 2) Cosine similarity with word vectors The weighted scoring is implemented, and if the weighted score is greater than the set threshold, a replacement is performed.

[0052] The aforementioned negation expression recognition refers to identifying negation expressions in user input and performing semantic direction correction before model prediction. This can include: using syntactic tree analysis to identify negation relation structures, determining the modification scope of negation words based on dependency parsing, marking "negation expression" flags, and adjusting the model's prediction logic according to the intent direction (e.g., changing from "place an order" to "cancel the order"). Syntactic tree analysis, also known as constituent syntactic analysis or phrase structure analysis, identifies different components in a sentence (such as noun phrases, verb phrases, etc.) and organizes these components into a hierarchical tree structure to represent the sentence's grammatical structure. Dependency parsing reveals the sentence structure by identifying the dependency relationships between words.

[0053] The aforementioned entity placeholder replacement refers to replacing named entities in the training corpus with standardized entity labels to improve the model's generalization ability. For example, replacing named entities such as "a certain bank" and "a certain type of bank card" appearing in the training samples with standard placeholder labels @bank and @card_type can improve the model's semantic generalization ability under different data distributions.

[0054] Furthermore, embodiments of the present invention also support model training in a multilingual environment. Based on this, the target model can be a general multilingual model or multiple language-specific models. The general multilingual model is obtained by uniformly encoding multilingual corpora into a shared vector space for training.

[0055] This invention employs a multilingual support module for language detection, cross-lingual semantic representation, and multilingual model routing. Upon user request, the language detector first determines the language, and then the appropriate model is selected via language routing (or a unified, general-purpose multilingual model is invoked). During training, this multilingual support module allows for the unified encoding of multilingual corpora into a shared vector space, or the parallel training of multiple models. For multimodal inputs, it also includes modality recognition and conversion components, such as calling an image recognition service to convert images into text labels or feature vectors, which are then fed into the NLU model along with the text.

[0056] Step S130: Extract the target sample set to be labeled from the preset log database for manual labeling; wherein, the log database stores log data obtained by the conversational virtual assistant using the target model for online intent prediction, and the target samples in the target sample set include input content, confidence value and intent label information.

[0057] In this embodiment of the invention, low-confidence prediction samples can be screened, recommended for labeling, and used for training feedback to achieve a closed-loop model update. This embodiment employs an active learning closed-loop feedback mechanism: prediction samples with confidence levels below a set threshold are clustered and screened, added to a pool of samples to be labeled, and then manually labeled for incremental model training, forming a continuous optimization closed loop.

[0058] In some possible embodiments, step S130 above may include: extracting log data of an unlabeled sample set from the log database according to a preset scheduling frequency; selecting a first sample from the unlabeled sample set and adding it to the target sample set based on one or more screening strategies among maximum entropy, minimum confidence, and minimum margin, according to the log data of the unlabeled sample set; performing vector encoding and clustering on the unlabeled sample set, and selecting a second sample from each cluster of the clustering results and adding it to the target sample set.

[0059] The preset time range mentioned above can be set according to actual needs and is not limited here. The preset time range can be related to the scheduling frequency. For example, if the scheduling frequency is once a day, the preset time range can be the nearest 24 hours to the current time. Sample clustering in active learning can use K-Means or hierarchical clustering algorithms based on semantic embedding.

[0060] To facilitate annotation, the candidate intents of each target sample in the target sample set can be prioritized, and the priority ranking results can be used as reference information for each target sample to be displayed to the annotators during annotation.

[0061] Considering that the target sample set may involve multiple language categories, the above-mentioned dialogue virtual assistant optimization method also includes: distributing annotation tasks to samples in the target sample set according to language categories, so that annotators can handle the annotation tasks of samples of the corresponding language categories.

[0062] Step S140: The manually labeled target sample set is incorporated into the training dataset, and the target model is incrementally updated based on the training dataset and the target hyperparameter combination.

[0063] The embodiments of this invention construct a closed loop for model training, autonomously selecting valuable unlabeled samples from user interactions, guiding manual annotation, continuously enriching training data, and improving model performance.

[0064] In some possible implementations, the active learning and closed-loop feedback scheme includes: collecting user input logs in real time after model deployment; if the model confidence is low or the output is None, it is included in the "sample pool to be labeled"; providing sample clustering and priority scoring, and recommending supplementary labels to the labelers; each round of labeled samples will trigger incremental model updates, forming a feedback loop.

[0065] This invention utilizes an active learning function to monitor unidentified or low-confidence user input online and perform sample screening. Core sub-components include an "uncertain sample detector" (identifying low-confidence predictions based on model confidence or prediction score distribution), a "diversity sampler" (clustering unlabeled sample semantic vectors to ensure selected samples cover multiple types), and a "sample queue manager" (caching samples to be labeled and presenting them on the operations interface).

[0066] A pool-based active learning framework can be adopted, which involves selecting the most informative batch of samples from the accumulated unlabeled dialogue logs and submitting them for manual annotation. The specific strategy combines uncertainty sampling and cluster-based diversity sampling. Uncertainty sampling: Identify the input samples with the greatest uncertainty in the model by using the model's prediction confidence or output distribution. Common strategies include selecting the lowest confidence sample (cases where the model has the lowest probability of predicting the intention but still needs to make a judgment), the minimum marginal sample (the sample with the smallest difference in confidence between the top two intentions), or the maximum entropy sample (the sample with the highest entropy value in the prediction distribution), etc.

[0067] Diversity Sampling: To avoid focusing on only a few ambiguous regions, cluster analysis is performed on unlabeled samples. Clustering can be based on the semantic embedding vectors of the samples (e.g., using the hidden layer output of existing models or sentence vectors generated by Sentence-BERT), applying algorithms such as K-Means++ or hierarchical clustering to divide the samples into clusters. Then, the most representative "uncertain" sample from each cluster (such as the cluster centroid or the sample with the lowest model confidence within the cluster) is selected for labeling. This ensures that the selected samples semantically cover a broader region and are not all concentrated on a single topic.

[0068] Technology stack selection: In terms of implementation, vector databases or efficient clustering libraries will be used to handle the embedding and clustering of large-scale unlabeled corpora (such as FAISS for similarity search, Scikit-learn or ELKI for clustering).

[0069] The inputs for active learning are: the raw, unlabeled data pool (continuously collected dialogue logs) and the prediction result set of the current model. The inputs may also include some configuration information: the threshold for filtering uncertain samples (e.g., a confidence level below 0.6 is considered uncertain), the expected number of labels N per round, etc.

[0070] The output of active learning is the set of samples selected for labeling in this round. For each sample, the output data may also include the original content and its auxiliary information (predicted intent and score, cluster category index, etc.). These outputs will be submitted to the labeling interface and enter the manual processing flow.

[0071] Active learning loop: The above strategy forms an iterative closed loop. After the initial model is trained with a small amount of labeled data, it is deployed to collect a large number of unlabeled queries. The active learning module selects several samples that need to be labeled according to the strategy. The training set is expanded by manual labeling. The model is trained and updated. The new model is deployed to continue collecting data.

[0072] The conversational virtual assistant optimization method provided in this invention improves versatility through adaptive selection of network structure, enhances optimization efficiency and ensures model performance through automated hyperparameter optimization, and achieves continuous model evolution by forming a complete closed loop of data, model, feedback and update through an active learning mechanism, thereby improving the model's prediction accuracy.

[0073] To facilitate understanding, the following sections will provide a detailed introduction to hyperparameter optimization, spell correction, negation expression recognition, entity placeholder generalization training, active learning, and feedback loop.

[0074] Considering that most current NLU models rely on manually set hyperparameters, the following problems exist: parameter setting is highly dependent on experience; the combination space is large, resulting in low trial-and-error efficiency; and it is difficult to obtain the optimal structure and training performance. This invention addresses these problems through hyperparameter optimization: based on an automatic search mechanism (e.g., TPE) using Bayesian optimization, it searches for the optimal combination for the validation set performance in a multidimensional parameter space. The technical solution for hyperparameter optimization is as follows: 1. Define the objective function (e.g., the F1 score on the validation set); 2. Construct the hyperparameter space; 3. Apply TPE or Gaussian Process for iterative sampling and evaluation; 4. Automatically generate the optimal training configuration.

[0075] Taking the TPE optimizer as an example, the specific process of hyperparameter optimization is as follows: First, define the objective function: ; in, θ Θ represents the combination of hyperparameters (such as learning rate, dropout, number of epochs); Θ represents the hyperparameter search space; F1 val : F1 score on the validation set.

[0076] Then, the target hyperparameter combination is determined using TPE: 1. Divide the hyperparameter combinations within the hyperparameter search space into a good-performance group l ( x ) and a poor-performance group g ( x ); 2. Calculate the expected improvement rate: ; 3. Select the next candidate point that maximizes the expected improvement rate x *.

[0077] Considering that users are prone to spelling mistakes or colloquial abbreviations when inputting natural language, resulting in model recognition errors: for example, "credit card" should be "credit card"; spelling mistakes directly affect semantic vectors and seriously affect classification performance. In the embodiment of the present invention, a lightweight spelling correction layer is added before the model inference stage to automatically correct the wrong words input by the user. The technical solution for spelling correction is as follows: 1. Tokenize the input text; 2. Match the edit distance of each word with the dictionary and compare the cosine similarity of the word vectors; 3. If the matching degree exceeds the threshold, replace it with the correct word in the dictionary.

[0078] Specifically, for each word w i , find the candidate word set { c j}, and select the one with the highest matching degree score Score c * to replace w i : ; where, represents the wrong word vector; represents the candidate word vector; λ represents the preset spelling weight parameter, such as 0.5; EditDistance() represents the edit distance function; cos() represents the cosine similarity function.

[0079] Considering that users often use negative expressions such as "don't" and "don't need", and traditional models cannot understand the direction of the negative effect, resulting in misclassification. For example, "I don't want to cancel the order" ≠ "I want to cancel the order". In the embodiment of the present invention, dependency syntactic analysis is used to identify negative words, judge the scope of negative modification, and reverse or correct the intention direction. The technical solution for negative expression recognition is as follows: 1. Construct a negative word dictionary; 2. Use dependency syntactic tree analysis to judge the modifier of the negative word; 3. Adjust the semantics direction before model prediction.

[0080] In practical implementation, a negation factor is defined. N ( x If a sentence contains a negative verb, then: ; in, f ( x ) indicates the original model prediction; N ( x `)=1` indicates an explicit negation; the final intent is corrected to `None` or "reverse intent". A "negation detection classifier" can be trained using rules or a lightweight BiLSTM model for location-level labeling.

[0081] Considering that entity values ​​are expressed differently among different users, direct training can lead to the model overfitting entity values ​​as the basis for judgment, reducing generalization ability. This embodiment of the invention replaces entity values ​​with placeholders (such as "@card_type", "@bank_name") in the training data, guiding the model to learn contextual semantics rather than the literal meaning of the entity. The technical solution for entity placeholder generalization training is as follows: 1. Use NER (Named Entity Recognition) tools to identify named entities in the training corpus; 2. Replace the original word with a placeholder; 3. The model is trained based on placeholder samples and inference is performed to recover the entity.

[0082] For example, given the input sample "I want to check the credit card statement of bank X," the placeholder processing would result in "I want to check the statement of @bank_name with @card_type." This way, model training focuses on sentence structures like "check @X statement," rather than specific words like "bank X" or "credit card."

[0083] The objective function for model training in entity placeholder generalization training can be: max ; in, x This represents the original input sample, i.e., the user's original expression; entities represent the set of named entities identified from the original sample, which can be automatically extracted using NER tools; Replace() represents the placeholder operation function, used to replace entity values ​​in the original input with generic placeholders; This indicates the sample after placeholder processing; y This represents the target intent label, i.e., the classification result that the model expects to predict; max This represents the objective function of the model's learning, i.e., the input given a placeholder. Under the condition of maximizing the prediction intention y The probability of.

[0084] Considering that trained models struggle to adapt to new data or rare representations, manual annotation is costly, has an unclear scope, and yields insignificant benefits. This invention introduces an active learning strategy: automatically selecting samples with high prediction uncertainty but strong representativeness for manual annotation, then returning them to the training process to form a closed loop. The technical solution for the active learning and feedback closed loop is as follows: 1. Regularly scan online user logs; 2. Calculate the confidence level for each prediction; 3. Use strategies such as maximum entropy, minimum confidence, and minimum margin to select samples; 4. Ensure sample diversity through clustering; 5. Labeling → Incremental training → Model update.

[0085] In practice, the maximum entropy selection strategy is as follows: ; in, The selected sample is the "most uncertain" sample chosen from the unlabeled sample set; argmax represents the input that maximizes the objective function. Indicates sample x Belongs to the unlabeled sample set U H() represents the entropy function, i.e., the uncertainty of the probability distribution; This indicates that the model is based on the input. x In this case, predict which intent category it belongs to. y The probability of; The minimum confidence difference strategy maximizes the sample with the smallest confidence difference between the first two predicted classes. ; in, This indicates the selected sample, i.e., the "most ambiguous" sample to be labeled; The model represents the input x The probability of the highest-confidence category (the probability of the category with the highest confidence). The model represents the input x The second highest classification probability (the probability of the second highest confidence class); argmin represents the input that minimizes the objective function.

[0086] Cluster center diversity assurance strategies: ; in, This indicates that one of the samples ultimately selected for manual annotation is preferred, with cluster centers selected to ensure that diverse and representative samples are annotated; BERT ( U ) indicates that BERT is used to perform vector encoding (i.e. semantic representation) on the text in the unlabeled sample set U; K-means() indicates that the K-means clustering algorithm is used to cluster the data into K semantic clusters; ClusterCenter() indicates the center sample of each cluster in the clustering results, representing the "core" or "representative" sample of each semantic distribution.

[0087] All embodiments of this invention address the pain points of existing NLU models by establishing an end-to-end intelligent tuning closed-loop system from input preprocessing (spelling, negation, entity) → model training (structure selection, parameter tuning) → continuous optimization (active learning closed loop). This not only improves model performance but also enhances the system's self-evolution capability, interpretability, and cross-industry adaptability.

[0088] To facilitate model debugging, the contribution of each word to intent prediction can be output based on the Shapley value mechanism for model debugging and interpretation. Therefore, the aforementioned dialogue-based virtual assistant optimization methods also include: When a local interpretation request is received, the feature contribution of the single-step model prediction data in the local interpretation request is calculated based on the Shapley value mechanism to obtain the local interpretation result. The single-step model prediction data includes the target model, the input content and its prediction result, and the local interpretation result includes the contribution value of each feature in the input content to the intention prediction. When a global explanation request is received, the feature influence of the training corpus prediction data in the global explanation request is calculated based on the Shapley value mechanism to obtain the global explanation result. The training corpus prediction data includes the target model, the training corpus and its prediction results, and the global explanation result includes the average influence ranking of each feature in the training corpus on the target model and the positive and negative key features corresponding to each intent.

[0089] The Shapley value mechanism described above considers all possible combinations of participants (i.e., alliances), calculates the marginal contribution of a participant joining each alliance, and finally averages these marginal contributions according to certain weights to obtain the participant's Shapley value. The contribution of each word to the current predictive intent can be output based on the SHAP (Shapley Additive Explanation) method.

[0090] Furthermore, the model's interpretation can be visualized. The positive and negative impact of highlighted words on intent can be visualized to aid developers and business understanding of model behavior. Optionally, the contribution of words can be visualized as a heatmap in the debug panel.

[0091] This invention provides an interpretation function for model prediction results, including a SHAP value calculator (calculating the contribution score of each word feature to the intent for a single prediction), a global feature importance analyzer (statistically identifying which words or features have the greatest impact on classification based on the training set), and a debugging panel service (integrating and presenting the above interpretation results, supporting user-interactive queries of the model decision-making process, such as viewing each step of the input through word segmentation, lexical reconstruction, vectorization, and output). This functional module can be integrated with an online inference engine to support real-time interpretation queries, or it can perform offline analysis on batch data.

[0092] This invention enhances interpretability by introducing a Shapley value-based Local Interpretation (SHAP) technology stack, coupled with a lexical-level feature visualization and debugging panel, thereby improving the interpretability of model predictions. SHAP utilizes the Shapley value principle from cooperative game theory to fairly calculate the contribution of each input feature to the model output. For text classification models, each word (or word vector feature) can be considered a "feature player," and the SHAP value reflects the positive or negative score and magnitude of the word's "contribution" to obtaining a certain intention prediction for the current input. A positive contribution value indicates that the word pushes the model to predict in that intention direction, while a negative value has the opposite effect; the absolute value represents the magnitude of the influence.

[0093] Technology Stack Selection: In terms of implementation, the SHAP open-source library is used to interpret deep learning models. For neural networks, especially complex models such as the Transformer, the SHAP library provides approximation algorithms (such as sampling-based approximate Shapley values) to efficiently calculate the contribution of each input token. Simultaneously, by referencing tools such as the Language Interpretability Toolkit (LIT), visualization of methods like attention weights and Grad-CAM can be achieved, allowing for the interpretation of the model from different perspectives.

[0094] Enhanced interpretability input: A model prediction that requires interpretation, including the model itself, the input text (and other modal inputs), and the prediction result. Users typically select a dialogue or input in the debug panel for interpretation analysis.

[0095] Interpretability-enhanced Output: Local interpretation results for this input. Taking text intent classification as an example, the output format is a contribution score corresponding to each word, which can be visualized as highlights with different shades of color (for example, red indicates promoting the prediction of the current intent, blue indicates suppressing this intent, and the color saturation indicates the magnitude of influence). In addition, the output includes the overall prediction confidence of the model, the top N other intents and their confidences, which is convenient for comparison. For global interpretation requests, the output includes the average influence ranking of each feature in the entire training corpus on the model, as well as the most representative positive and negative key feature words for each intent, etc.

[0096] Transparent Debugging Panel: Using the above output, the debugging panel will present the model decision-making link in a user-friendly manner. For example, when an operator or an AI engineer inputs a sentence in the panel, the system will display in real time: Intent Scoring Details: This sentence is predicted to be intent A (90% confidence), the second highest is intent B (5%), etc., and at the same time, the original scores (logits) inside the model and the normalized scores after sorting and processing are listed.

[0097] Visualization of Word Contributions: Each word in the sentence is highlighted with a color, intuitively showing which words strongly point to intent A, and which words make the model tend to other intents or None. For example, the word "cancel" has a strong positive contribution to the intent "cancel the order", while the word "no" has a negative contribution to the intent "purchase", indicating that the model believes the user is negating the purchase intent due to "no".

[0098] Feature Processing Process: If certain text preprocessing is enabled, such as synonym replacement and negation word processing, the debugging panel can display the intermediate results (such as the "no" in the annotation "no problem" is recognized as a negation word, and the weight of some intent scores is reduced). It can also show the results of word segmentation and word form reduction, ensuring that developers understand how the input is understood by the model.

[0099] Training Example Reference: The panel provides a list of training example sentences similar to this input in the vector space, helping to determine whether the model's decision is affected by certain training data (for example, the current sentence may only differ from another sentence in the training set by one word, and the model may analogize the label of that sample).

[0100] Through these interpretability functions, users can gain in-depth insights into the model's behavior: understand "why" the model makes a certain judgment. It helps in optimization (finding the reasons for misjudgments, adding training data or adjusting the algorithm针对性), enhancing the trust in the model, and meeting compliance requirements in certain scenarios.

[0101] In some possible embodiments, the target model is a general multilingual model or multiple language-specific models. The general multilingual model is obtained by uniformly encoding multilingual corpora into a shared vector space for training. Based on this, the above-mentioned conversational virtual assistant optimization method further includes: performing language recognition on the current input content of the user to obtain the target language; preprocessing the current input content according to the target language to obtain preprocessed content; wherein, the preprocessing includes one or more of multimodal conversion, spelling correction processing, and negation expression recognition; and using the general multilingual model or the language-specific model corresponding to the target language to predict the intent of the preprocessed content to obtain the target prediction result.

[0102] Replacing named entities in the training corpus or inference input with standardized entity labels can improve the model's generalization ability. The selected model (i.e., a general multilingual model or a language-specific model corresponding to the target language) can be used to perform intent recognition and entity extraction operations on user input, outputting the predicted intent and entities.

[0103] This invention provides multilingual / multimodal support: the extended engine supports model training and inference in multilingual environments, and integrates multimodal inputs such as speech and images on top of text to address a wider range of application scenarios. It also enhances interpretability by providing in-depth model interpretation tools, such as word vector feature contribution analysis, visualization of intent classification decisions, and a transparent debugging panel to help users understand and debug the model.

[0104] For multilingual support, this embodiment of the invention introduces a scheme combining language-agnostic multilingual models and language routing. This embodiment prioritizes the use of language-agnostic (language-agnostic) NLP models. One feasible technique is to obtain a shared semantic space based on multilingual pre-trained models (e.g., mBERT, XLM-R, etc.): the model maps sentences from different languages ​​to the same vector space, making sentences with the same semantics (regardless of language) close to each other in this space.

[0105] Meanwhile, for languages ​​with ample resources (such as English), language-specific models can be combined to form an ensemble: for example, a powerful English-specific model and a general multilingual model can be used to predict the same input separately, and then the results can be fused to balance multilingual coverage and monolingual performance.

[0106] Multilingual inputs include multilingual training corpora and multilingual user input to be predicted. During training, the input corpora are stored separately by language label, but when using a language-independent model, they are uniformly converted to shared vectors for training. During prediction, a user message is input, processed by a language detector, and its language type and normalized text are output.

[0107] Multilingual support outputs include the model's recognition results for the input, such as intent and confidence level, entity identification, and the language / model identifier used for recognition (facilitating log analysis and statistical reporting by language). For multimodal inputs, the output also includes auxiliary information generated by the image / audio model (e.g., identified object labels, speech-to-text results).

[0108] This invention allows users with different roles to participate in the construction and optimization of the virtual assistant's NLU model, achieving automatic improvement and continuous enhancement of the model's performance. This invention is applicable to various user groups who use the platform to build and optimize virtual assistants, primarily including: AI Algorithm Engineers: Technical personnel responsible for model architecture and algorithm optimization. They focus on model performance metrics, aiming to quickly improve model accuracy using automatic hyperparameter optimization and multilingual frameworks, and leverage interpretable tools to gain a deeper understanding of the model's decision-making principles, guiding further improvements.

[0109] Business Operations Personnel: Business managers responsible for the operation and performance management of the virtual assistant product. They typically do not delve into the details of the algorithm but focus on metrics such as the robot's recognition accuracy and user satisfaction. Operations personnel aim to monitor actual dialogue performance through the active learning module, identify model weaknesses (such as frequently missed questions), and coordinate process improvements to continuously enhance the robot's performance.

[0110] Non-technical annotators: Individuals or domain experts engaged in data annotation and validation. They may lack programming and algorithm backgrounds but are familiar with business corpora. This user expects the system to provide an intuitive annotation interface and an efficient sample recommendation mechanism (e.g., automatically selecting challenging dialogues to be annotated) to quickly and effectively improve the training corpus.

[0111] A typical use case is as follows: 1. Rapid Optimization for New Assistant Deployment: When deploying a new conversational virtual assistant in an enterprise, AI engineers use the automatic HPO module to perform hyperparameter tuning on the initial training data to find the optimal model configuration. Subsequently, through active learning, low-confidence samples from early user dialogues are continuously collected and labeled by annotators, enabling rapid improvement in model performance from cold start to maturity.

[0112] 2. Multilingual Customer Support: A multinational company built a multilingual customer service robot. The AI ​​team utilized multilingual support, using a single model to handle queries in multiple languages, including English and Chinese. During operation, the system automatically collected examples of poorly recognized queries in different languages ​​(such as long sentences from Chinese users or requests mixed with emoticons and images), and annotators labeled them in their respective language queues.

[0113] 3. Iterative Analysis of Complex Business Intents: A bot in the financial industry generated a large number of similar intents. Using an interpretive debugging panel, operations personnel discovered that certain keywords significantly impacted the matching of incorrect intents. Based on this, they adjusted the training corpus and dictionary, and actively learned more relevant example sentences for annotation engineers to expand the dataset. After training the new model, the accuracy of classifying these easily confused intents significantly improved.

[0114] For ease of understanding, please refer to the following: Figure 2 An exemplary overall process for the above-mentioned conversational virtual assistant optimization method is described. For example... Figure 2 As shown, the overall process of optimizing a conversational virtual assistant includes: Step S201: The user enters text.

[0115] Step S202, spelling correction.

[0116] Step S203, Negation expression recognition.

[0117] Step S204: Replace entity placeholders.

[0118] Step S205, Model structure selection.

[0119] Step S206: Determine if there are already training results. If not, proceed to step S207; if yes, proceed to step S208.

[0120] Step S207, Automatic hyperparameter optimization.

[0121] Step S208: Use the optimal model structure and parameter combination.

[0122] Step S209: Start the training process.

[0123] Step S210: Deploy the model to the online NLU engine.

[0124] Step S211: User inputs real-time prediction.

[0125] Step S212: Predict the output intent, entity, and confidence level.

[0126] Step S213: Determine if the confidence level is low. If yes, proceed to step S214; otherwise, proceed to step S218.

[0127] Step S214: Add to the active learning sample pool.

[0128] Step S215, clustering and sorting.

[0129] Step S216: Assign manual annotations.

[0130] Step S217: Label the samples and add them to the training set. Then, repeat step S209.

[0131] Step S218: Log the prediction results and return the prediction results.

[0132] Step S219: Perform model interpretation.

[0133] Step S220: Output word contribution and predicted causal chain. Process ends.

[0134] Compared with the prior art, the embodiments of the present invention have the following technical advantages: 1. Adaptive Structure: The multi-network switching mechanism supports modeling of different business scenarios, improving system versatility; 2. Strong prediction tolerance: Integrates spell correction and negation recognition functions to improve the robustness of user input recognition; 3. High tuning efficiency: Automated hyperparameter tuning reduces training time by an average of 50% and improves F1 scores by 5%~12%; 4. Strong generalization ability: The entity placeholder mechanism eliminates entity value interference and improves the consistency of the model across corpora; 5. Easy model debugging: SHAP's visual output enhances model transparency and supports compliance auditing and business optimization; 6. Continuous evolution capability: The active learning mechanism forms a complete closed loop of data → model → feedback → update, supporting self-driven model optimization; 7. Multilingual / Multimodal Adaptation: The system supports selecting the corresponding model route based on language detection, and supports speech-to-text and image-to-label conversion as NLU inputs.

[0135] The data flow design of this invention embodiment is as follows: 1. When creating a new conversational virtual assistant or making a major update to the conversational virtual assistant, AI engineers can upload the initial training data (i.e., the training corpus mentioned above) to the system (input: a set of labeled user dialogues and their intent labels). The tuning console sends the configured hyperparameter search space and training data to the HPO engine. The HPO engine takes the training data and hyperparameter combination as input and performs multiple model training evaluations (during this process, model performance is evaluated through cross-validation or holding a validation set). Each trial outputs model performance metrics (such as accuracy / F1 score) and corresponding hyperparameter values. After multiple iterations, the HPO module outputs the optimal hyperparameter configuration and the optimized model file (i.e., the target model). The entire process realizes the transformation from training data + hyperparameter space to optimized model, outputting a high-performance model for online use. The optimal model parameters can be stored as an IntelliTune configuration file for subsequent model retraining and reuse; after model deployment, it enters the online service stage.

[0136] 2. Online Inference and Log Collection: The deployed NLU engine processes user input (input: the user's natural language request, which may include text and multimedia) during the dialogue prediction phase. The online engine normalizes the input into a feature representation acceptable to the model through language detection and necessary modality preprocessing; subsequently, the model generates intent prediction results (output: intent and confidence score, entity extraction results, etc.). Simultaneously, the engine records each user request and its prediction result in a log database, including: the original input, the identified language / modality, the predicted intent and confidence score, whether a "None intent" (meaning the user's explicit intent was not identified) or fallback was triggered, etc.

[0137] 3. Active Learning Sample Selection: Log data accumulates continuously in the background. The active learning module extracts a set of recently unlabeled samples (input: recent user dialogue logs) from the log database at a certain scheduling frequency (e.g., daily or whenever N dialogues are accumulated). The sample selection process may include: Uncertainty sample detection: For each record, examine its prediction confidence or ranking difference, and mark sentences with confidence scores below a preset threshold or those judged by the model as having "None intent". According to active learning theory, these low-confidence samples are often the most informative.

[0138] Diversity sampling: To avoid overly similar samples, semantic vector representations are computed for candidate samples (using hidden layers of NLU models or pre-trained embeddings), and clustering (such as K-means) is performed in the vector space. One or two of the most representative sentences from each cluster are selected and added to the annotation list to ensure coverage of various types of unresolved inputs.

[0139] Output filtering results: The active learning module outputs a list of samples to be labeled, each of which includes the original input content, model confidence information, and the cluster / category label (for reference).

[0140] 4. Manual Annotation and Feedback: After reviewing the list of samples to be annotated on the monitoring panel, business operations personnel assign the tasks to annotators for processing (e.g., assigning different annotators to samples in different languages). The annotation tool interface obtains the sample list as input, and the annotators review each sample one by one. For text input, select or enter the correct intent label (if it is a new intent or needs to be split into multiple intents, feedback can be given to the operations staff to add intent definitions).

[0141] For multimodal inputs with images, the annotator simultaneously reviews the preliminary information extracted from the image by the system (such as the category of the item) and adds supplementary descriptions or classification labels to the image portion as necessary.

[0142] The annotation interface can also display the model's current prediction and confidence level for the unannotated sentence for reference, but the correct label is ultimately determined by human intervention.

[0143] For each entry processed, the annotator submits the annotation result (output: manually confirmed correct intent / entity label), and the system immediately stores it in the incremental update area of ​​the training data storage and removes it from the annotation queue. After the entire batch is completed, if the operations staff confirms that there are no errors, the newly annotated data is officially merged into the training set.

[0144] 5. Incremental Model Update: When a certain amount of newly labeled data accumulates or the scheduled retraining time expires, the central scheduling service will initiate the model retraining process. Inputs include: the existing complete training corpus (containing data from previous rounds and newly added data in this round) and the optimal hyperparameter configuration obtained from the last training. The training and tuning subsystem loads these inputs and performs model training, producing an updated model. Since stable hyperparameters are used, this training primarily aims to allow the model to learn the new corpus. When the data distribution changes significantly, the system also allows rerunning HPO to fine-tune hyperparameters to adapt to the new data (e.g., periodically re-searching for hyperparameters on new corpora every few months). If the performance improvement meets expectations, the updated model will be automatically deployed to the online system.

[0145] At this point, an active learning loop is complete: online data collection - filtering and labeling - model update - redeployment. After the new model is launched, the cycle begins again with log collection and active learning, iterating continuously.

[0146] The closed-loop data flow is characterized by continuous feedback and self-improvement: the model output (prediction log) of each round becomes the input (samples to be labeled) of the next round; the new labels output by the human are fed back to the model, thereby reducing the probability of the same error occurring in the next round.

[0147] Corresponding to the above-described method for optimizing conversational virtual assistants, this embodiment of the invention also provides a system for optimizing conversational virtual assistants. See also... Figure 3 The diagram shown illustrates the structure of a conversational virtual assistant tuning system, which includes: The model structure selection module 301 is used to determine the target network structure from a preset set of neural networks based on the feature information of the training corpus of the conversational virtual assistant. The hyperparameter optimization training module 302 is used to optimize the hyperparameters and train the initial model of the target network structure based on the training corpus using a Bayesian optimization algorithm or a tree structure parameter estimator algorithm, so as to obtain the target model under the optimal combination of target hyperparameters. The active learning module 303 is used to extract a target sample set to be labeled from a preset log database for manual labeling. The log database stores log data obtained by the conversational virtual assistant using the target model for online intent prediction. The target samples in the target sample set include input content, confidence value and intent label information. The manually labeled target sample set is incorporated into the training dataset, and the target model is incrementally updated based on the training dataset and the target hyperparameter combination.

[0148] The conversational virtual assistant tuning system provided in this invention improves versatility through adaptive selection of network structure, enhances tuning efficiency and ensures model performance through automated hyperparameter tuning, and achieves continuous model evolution by forming a complete closed loop of data, model, feedback and update through an active learning mechanism, thereby improving the model's prediction accuracy.

[0149] Furthermore, the system also includes a model interpretation module, used for: When a local interpretation request is received, the feature contribution of the single-step model prediction data in the local interpretation request is calculated based on the Shapley value mechanism to obtain the local interpretation result. The single-step model prediction data includes the target model, the input content and its prediction result, and the local interpretation result includes the contribution value of each feature in the input content to the intention prediction. When a global explanation request is received, the feature influence of the training corpus prediction data in the global explanation request is calculated based on the Shapley value mechanism to obtain the global explanation result. The training corpus prediction data includes the target model, the training corpus and its prediction results, and the global explanation result includes the average influence ranking of each feature in the training corpus on the target model and the positive and negative key features corresponding to each intent.

[0150] Furthermore, the target model mentioned above is a general multilingual model or multiple language-specific models. The general multilingual model is obtained by uniformly encoding multilingual corpora into a shared vector space for training. The system also includes a model prediction module, used for: Perform language recognition on the current input content from the user to obtain the target language; Based on the target language, the current input content is preprocessed to obtain preprocessed content; the preprocessing includes one or more of the following: multimodal conversion, spell correction processing, and negation expression recognition. The intent prediction of the preprocessed content is obtained by using a general multilingual model or a language-specific model corresponding to the target language.

[0151] Furthermore, the aforementioned model structure selection module 301 is specifically used to: extract feature information from the training corpus to obtain corpus features, which include one or more of sentence length distribution, entity density, intent quantity, and intent entropy; Based on the characteristics of the corpus, the target network structure is selected from the neural network set through a preset rule template or heuristic model.

[0152] Furthermore, the model structure selection module 301 described above is also used to: calculate the task complexity based on the corpus features using the following formula: ; in, C Indicates task complexity. L avg Here, represents the average sample length, Entropy(Intent) represents the entropy of the intent label distribution, and Entity_Density represents the average number of entities per sentence. α 1. α 2. α 3 represents the preset weight; The target network structure is selected from the neural network set based on the complexity of the task.

[0153] Furthermore, the aforementioned hyperparameter optimization training module 302 is specifically used for: preprocessing the training corpus to obtain preprocessed training corpus; wherein, the preprocessing includes one or more of multimodal conversion, spelling correction processing, negation expression recognition, and entity placeholder replacement; based on the preprocessed training corpus, multiple training tasks are scheduled by employing a Bayesian optimization algorithm or a tree structure parameter estimator algorithm, and the model performance of different hyperparameter combinations under a preset hyperparameter search space is evaluated to determine the target model under the target hyperparameter combination with optimal performance.

[0154] Furthermore, the aforementioned active learning module 303 is specifically used to: extract log data of an unlabeled sample set from the log database according to a preset scheduling frequency; based on the log data of the unlabeled sample set, use one or more screening strategies among maximum entropy, minimum confidence, and minimum margin to select a first sample from the unlabeled sample set and add it to the target sample set; perform vector encoding and clustering on the unlabeled sample set, and select a second sample from each cluster of the clustering results and add it to the target sample set.

[0155] Furthermore, the aforementioned target sample set involves multiple language categories; the aforementioned active learning module 303 is also used to: distribute annotation tasks to the samples in the target sample set according to language categories, so that annotators can handle the annotation tasks of the corresponding language category samples.

[0156] The embodiments of this invention address the core challenges of building and maintaining natural language understanding models, aiming to solve the six categories of problems shown in Table 1 below: Table 1

[0157] The technical features and effects of the embodiments of the present invention are shown in Table 2 below.

[0158] Table 2

[0159] The system provided in this embodiment has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0160] like Figure 4 As shown, an electronic device 400 provided in this embodiment of the invention includes: a processor 401, a memory 402 and a bus. The memory 402 stores a computer program that can run on the processor 401. When the electronic device 400 is running, the processor 401 and the memory 402 communicate through the bus, and the processor 401 executes the computer program to implement the above-mentioned conversational virtual assistant tuning method.

[0161] Specifically, the memory 402 and processor 401 mentioned above can be general-purpose memory and processor, without any specific limitations here.

[0162] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the conversational virtual assistant tuning method described in the preceding method embodiments. The computer-readable storage medium includes various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk.

[0163] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0164] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing a conversational virtual assistant, characterized in that, include: Based on the feature information of the training corpus of the conversational virtual assistant, the target network structure is determined from the preset neural network set; Based on the training corpus, the initial model of the target network structure is optimized for hyperparameters and trained using a Bayesian optimization algorithm or a tree structure parameter estimator algorithm to obtain the target model with the optimal combination of target hyperparameters. A target sample set to be labeled is extracted from a preset log database for manual labeling; wherein, the log database stores log data obtained by the conversational virtual assistant using the target model to perform online intent prediction, and the target samples in the target sample set include input content, confidence value and intent label information; The manually labeled target sample set is incorporated into the training dataset, and the target model is incrementally updated based on the training dataset and the target hyperparameter combination. The step of extracting the target sample set to be labeled from the preset log database includes: extracting log data of an unlabeled sample set within a preset time range from the log database according to a preset scheduling frequency; selecting a first sample from the unlabeled sample set and adding it to the target sample set based on the log data of the unlabeled sample set using one or more filtering strategies among maximum entropy, minimum confidence, and minimum margin; performing vector encoding and clustering on the unlabeled sample set, and selecting a second sample from each cluster of the clustering results and adding it to the target sample set.

2. The method for optimizing a conversational virtual assistant according to claim 1, characterized in that, The conversational virtual assistant optimization method also includes: When a local interpretation request is received, the feature contribution of the single model prediction data in the local interpretation request is calculated based on the Shapley value mechanism to obtain the local interpretation result; wherein, the single model prediction data includes the target model, input content and its prediction result, and the local interpretation result includes the contribution value of each feature in the input content to the intention prediction; When a global explanation request is received, the feature influence of the training corpus prediction data in the global explanation request is calculated based on the Shapley value mechanism to obtain the global explanation result. The training corpus prediction data includes the target model, the training corpus and its prediction results, and the global explanation result includes the average influence ranking of each feature in the training corpus on the target model and the positive and negative key features corresponding to each intent.

3. The method for optimizing a conversational virtual assistant according to claim 1, characterized in that, The target model is a general multilingual model or a multilingual-specific model. The general multilingual model is obtained by uniformly encoding multilingual corpora into a shared vector space for training. The conversational virtual assistant optimization method also includes: Perform language recognition on the current input content from the user to obtain the target language; Based on the target language, the current input content is preprocessed to obtain preprocessed content; wherein, the preprocessing includes one or more of multimodal conversion, spell correction processing, and negation expression recognition; The preprocessed content is used to predict intent using the general multilingual model or the language-specific model corresponding to the target language to obtain the target prediction result.

4. The method for optimizing a conversational virtual assistant according to claim 1, characterized in that, The step of determining the target network structure from a preset set of neural networks based on the feature information of the training corpus of the conversational virtual assistant includes: Feature information is extracted from the training corpus to obtain corpus features, which include one or more of sentence length distribution, entity density, intent quantity, and intent entropy. Based on the features of the corpus, the target network structure is selected from the neural network set using a preset rule template or heuristic model.

5. The method for optimizing a conversational virtual assistant according to claim 4, characterized in that, The step of selecting the target network structure from the neural network set based on the corpus features using a preset rule template or heuristic model includes: Based on the corpus features, the task complexity is calculated using the following formula: ； in, C Indicates task complexity. L avg Here, represents the average sample length, Entropy(Intent) represents the entropy of the intent label distribution, and Entity_Density represents the average number of entities per sentence. α 1. α 2. α 3 represents the preset weight; The target network structure is selected from the set of neural networks based on the complexity of the task.

6. The method for optimizing a conversational virtual assistant according to claim 1, characterized in that, The step involves optimizing the hyperparameters and training the initial model of the target network structure using a Bayesian optimization algorithm or a tree structure parameter estimator algorithm based on the training corpus, to obtain the target model with the optimal combination of target hyperparameters, including: The training corpus is preprocessed to obtain preprocessed training corpus; wherein, the preprocessing includes one or more of the following: multimodal conversion, spelling correction, negation expression recognition, and entity placeholder replacement; Based on the preprocessed training corpus, multiple training tasks are scheduled using a Bayesian optimization algorithm or a tree-structured parameter estimator algorithm, and the model performance of different hyperparameter combinations under a preset hyperparameter search space is evaluated to determine the target model with the optimal target hyperparameter combination.

7. The method for optimizing a conversational virtual assistant according to claim 1, characterized in that, The target sample set involves multiple language categories; after extracting the target sample set to be labeled from the preset log database, the conversational virtual assistant optimization method further includes: The annotation tasks are distributed to the samples in the target sample set according to language category, so that the annotators can handle the annotation tasks of the samples of the corresponding language category.

8. A conversational virtual assistant tuning system, characterized in that, include: The model structure selection module is used to determine the target network structure from a preset set of neural networks based on the feature information of the training corpus of the conversational virtual assistant. The hyperparameter optimization and training module is used to optimize the hyperparameters and train the initial model of the target network structure based on the training corpus using a Bayesian optimization algorithm or a tree structure parameter estimator algorithm, so as to obtain the target model under the optimal combination of target hyperparameters. An active learning module is used to extract a target sample set to be labeled from a preset log database for manual labeling; wherein, the log database stores log data obtained by the conversational virtual assistant using the target model to perform online intent prediction, and the target samples in the target sample set include input content, confidence value and intent label information; the manually labeled target sample set is incorporated into the training dataset, and the target model is incrementally updated according to the training dataset and the target hyperparameter combination; The active learning module is specifically used to: extract log data of an unlabeled sample set within a preset time range from the log database according to a preset scheduling frequency; and, based on the log data of the unlabeled sample set, use one or more filtering strategies among maximum entropy, minimum confidence, and minimum margin to select a first sample from the unlabeled sample set and add it to the target sample set. The unlabeled sample set is vector-encoded and clustered, and a second sample is selected from each cluster of the clustering results and added to the target sample set.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the conversational virtual assistant tuning method according to any one of claims 1-7.

Images