Quality control system and method for conversational ai models

Abstract

The application provides a quality control system and method of a conversational AI model, relates to the technical field of artificial intelligence and natural language processing, and the system comprises a three-dimensional coverage analysis module, a supplementary test recommendation module, a threshold dynamic adjustment module, a misjudgment sample analysis module, a training suggestion generation module, a closed-loop training optimization module and a gate control module; the application integrates three-dimensional coverage analysis of semantic testing, supplementary test recommendation, threshold dynamic adjustment, misjudgment sample analysis, structured training suggestion generation, model training closed loop and gate control to form an integrated "testing-analysis-optimization-regression verification-deployment" system, and the development efficiency, diagnosis capability and online safety are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and natural language processing, in particular to a quality control system and method for a conversational AI model. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, conversational AI models (also known as conversational AI assistants / conversational AI helpers, such as customer service robots, voice assistants, intelligent question answering systems, etc.) are increasingly widely used in various industries, and higher requirements are placed on natural language understanding capabilities.

[0003] In the prior art, the evaluation of conversational AI models usually relies on macro indicators such as overall accuracy or F1 value, which can reflect the overall performance of the model, but cannot deeply analyze the specific performance of the model in complex dialogue processes.

[0004] In actual applications, the test process of the conversational AI model is usually independent of the training process. When the model makes a false judgment in the test phase, the false judgment samples need to be manually exported, manually classified, and improvement tasks are generated through Jira and other task management systems. At the same time, the patches in the training phase may be scattered in different systems or code libraries, resulting in low improvement efficiency and slow model iteration speed. In addition, the traditional quality control system lacks complete reasoning link records, making it difficult for developers to quickly trace the root cause of errors when the model prediction is wrong, affecting problem troubleshooting and optimization.

[0005] In the model online phase, the existing technology mostly relies on manual comparison of the performance differences between new and old versions, lacking an automated health check mechanism, which makes the model online have a potential performance regression risk.

[0006] With the increasing complexity of conversational AI models and the expansion of application scenarios, how to achieve fine-grained evaluation, efficient test and training collaboration, traceable reasoning analysis, and automated online verification has become a key challenge to improve the landing effect of the model. SUMMARY

[0007] The purpose of the present application is to provide a quality control system and method for a conversational AI model to at least solve one of the above problems.

[0008] In a first aspect, the present application provides a quality control system for a conversational AI model, comprising:

[0009] a three-dimensional coverage analysis module, configured to generate a three-dimensional coverage calculation result according to input dialogue test data of the conversational AI model, the three-dimensional coverage calculation result comprising a three-dimensional coverage matrix corresponding to intent, process node, and path;

[0010] The supplement test recommendation module is configured to perform test integrity analysis on the three-dimensional coverage calculation result and a dialog flowchart corresponding to the dialog AI model, and generate supplement test suggestions.

[0011] The threshold dynamic adjustment module is configured to adjust the confidence threshold of each intent according to the intent error rate and the intent heat in the dialog test data, and obtain new confidence threshold information of the dialog AI model.

[0012] The misjudgment sample analysis module is configured to perform error classification and confusion cluster detection on the input misjudgment sample basic data through a rule engine and semantic clustering, and obtain semantic type label data.

[0013] The training suggestion generation module is configured to map the semantic type label data into structured training suggestion data.

[0014] The closed-loop training optimization module is configured to generate training patch data according to a target structured suggestion set adopted by the user in the structured training suggestion data, and perform incremental fine-tuning on the current model of the dialog AI model by using the training patch data, to obtain a new model.

[0015] The gatekeeper control module is configured to compare regression test indicators of the new model and the current model, and perform online deployment of the new model.

[0016] In an optional implementation, the system further includes a self-distillation module configured to take a sample set classified as false positive as a negative sample data set.

[0017] The closed-loop training optimization module is further configured to inject the negative sample data set into the training patch data to realize semantic boundary self-correction.

[0018] In an optional implementation, the dialog test data includes test utterances, predicted intents, hit nodes, and path jump records, and the three-dimensional coverage calculation result further includes three-dimensional statistical data; the three-dimensional coverage calculation result is generated according to the input dialog test data of the dialog AI model, and includes:

[0019] A three-tuple composed of an intent, a flow node, and a path sequence corresponding to each test utterance in the dialog test data is extracted, and the extracted three-tuple data is filled into a three-dimensional tensor composed of an intent number, a node number, and a path identifier, to obtain a three-dimensional coverage matrix.

[0020] The three-dimensional coverage matrix is subjected to intent coverage rate, node coverage rate, and path coverage rate calculation, to obtain three-dimensional statistical data.

[0021] In an optional implementation, the supplement test suggestions include an uncovered path list and recommended supplement test cases; the supplement test suggestions are generated according to the test integrity analysis on the three-dimensional coverage calculation result and a dialog flowchart corresponding to the dialog AI model, and include:

[0022] The uncovered path list is obtained by completing recognition of the three-dimensional coverage matrix of the dialogue flowchart through the directed acyclic graph traverser and the path mapping function;

[0023] The recommended test case is generated according to the uncovered path list and the preset sample case template.

[0024] In an optional implementation, the threshold dynamic adjustment module is configured to adjust the confidence threshold value through the following formula:

[0025] ;

[0026] wherein, θ 0 represents the initial confidence threshold value of the i-th intent, i 0 represents the new confidence threshold value of the i-th intent, θ i i i freq i i err i λ γ

[0027] In an optional implementation, the misjudgment sample basic data includes error samples and their corresponding intent labels, confidence scores and entity recognition results; the input misjudgment sample basic data is subjected to error classification and confusion cluster detection through a rule engine and semantic clustering to obtain semantic type label data, including:

[0028] According to the intent labels, confidence scores and entity recognition results in the misjudgment sample basic data, the rule engine is used to label the error types of the corresponding error samples to obtain error label type data; wherein the error types include intent mismatch, entity missing and low confidence;

[0029] The BERT model is used to generate sample semantic vectors of each error sample, and the sample semantic vectors of the error samples are subjected to K-Means clustering to obtain a cluster analysis report, which includes the cluster identification and cluster entropy value corresponding to each error sample;

[0030] According to the error label type data and the cluster analysis report, the semantic type label data is generated.

[0031] In an optional implementation, the current model of the dialogue AI model is incrementally fine-tuned using the training patch data to obtain a new model, including: ​​​​​​​​

[0032] Freeze the backbone model parameters in the current model of the conversational AI model, and train the LoRA layer or the Adapter layer using the training patch data to obtain a new model.

[0033] In an optional implementation, the online deployment of the new model is performed by comparing the regression test indicators of the new model and the current model, including:

[0034] According to the regression test results of the new model and the current model, the F1 score increase and the rollback rate increase are obtained;

[0035] Determine whether the F1 score increase and the rollback rate increase pass the preset goalkeeper strategy to obtain a goalkeeper judgment result;

[0036] If the goalkeeper judgment result is yes, the online deployment of the new model is performed.

[0037] In a second aspect, the present application provides a quality control method of a conversational AI model, including:

[0038] The three-dimensional coverage analysis module generates a three-dimensional coverage calculation result according to the input conversational AI model test data, and the three-dimensional coverage calculation result includes a three-dimensional coverage matrix corresponding to the intent, the flow node and the path;

[0039] The retest recommendation module performs test integrity analysis on the three-dimensional coverage calculation result and the conversational flowchart corresponding to the conversational AI model to generate a retest suggestion;

[0040] The threshold dynamic adjustment module adjusts the confidence threshold of each intent according to the intent error rate and the intent heat in the conversational test data to obtain new confidence threshold information of the conversational AI model;

[0041] The misjudgment sample analysis module performs error classification and confusion cluster detection on the input misjudgment sample basic data through a rule engine and semantic clustering to obtain semantic type label data;

[0042] The training suggestion generation module maps the semantic type label data into structured training suggestion data;

[0043] The closed-loop training optimization module generates training patch data according to the target structured suggestion set adopted by the user in the structured training suggestion data, and performs incremental fine-tuning on the current model of the conversational AI model using the training patch data to obtain a new model;

[0044] The goalkeeper control module performs online deployment of the new model by comparing the regression test indicators of the new model and the current model.

[0045] In an optional implementation, the above method further includes:

[0046] The self-distillation module classifies the sample set that is false positive as a negative sample data set;

[0047] The closed-loop training optimization module also injects the negative sample data set into the training patch data to achieve semantic boundary self-correction.

[0048] The quality control system and method of the dialog AI model provided by the application, the system comprises: a three-dimensional coverage analysis module, which is used for generating a three-dimensional coverage calculation result according to input dialog test data of the dialog AI model, the three-dimensional coverage calculation result comprising a three-dimensional coverage matrix corresponding to an intent, a flow node and a path; a supplementary test recommendation module, which is used for performing test integrity analysis on a dialog flowchart corresponding to the dialog AI model according to the three-dimensional coverage calculation result, and generating a supplementary test suggestion; a threshold dynamic adjustment module, which is used for adjusting a confidence threshold of each intent according to an intent error rate and an intent heat in the dialog test data, and obtaining new confidence threshold information of the dialog AI model; a misjudgment sample analysis module, which is used for performing error classification and confusion cluster detection on input misjudgment sample basic data through a rule engine and semantic clustering, and obtaining semantic type label data; a training suggestion generation module, which is used for mapping the semantic type label data into structured training suggestion data; a closed-loop training optimization module, which is used for generating training patch data according to a target structured suggestion set adopted by a user in the structured training suggestion data, and performing incremental fine-tuning on a current model of the dialog AI model by using the training patch data, to obtain a new model; and a gatekeeper control module, which is used for comparing regression test indexes of the new model and the current model, and performing online deployment of the new model. In this way, the three-dimensional coverage analysis, the supplementary test recommendation, the threshold dynamic adjustment, the misjudgment sample analysis, the structured training suggestion generation, the model training closed loop and the gatekeeper control are fused to form an integrated “test-analysis-optimization-regression verification-deployment” system, which improves the development efficiency, the diagnosis capability and the online safety. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0050] Figure 1 A structural schematic diagram of a quality control system of a dialog AI model provided by the embodiment of the application;

[0051] Figure 2 A general work flow schematic diagram of a quality control system of a dialog AI model provided by the embodiment of the application;

[0052] Figure 3 A flowchart of a quality control method of a conversational AI model provided for an embodiment of the present application is shown in the figure;

[0053] Figure 4 A flowchart of another quality control method of a conversational AI model provided for an embodiment of the present application is shown in the figure.

[0054] Icon: 101-test execution module; 102-three-dimensional coverage analysis module; 103-supplementary test recommendation module; 104-threshold dynamic adjustment module; 105-misjudgment sample analysis module; 106-training suggestion generation module; 107-closed-loop training optimization module; 108-gatekeeper control module; 109-self-distillation module. DETAILED DESCRIPTION

[0055] The technical solutions of the present application will be described in detail below in conjunction with embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0056] The existing quality assurance technologies for conversational artificial intelligence systems have the following problems:

[0057] 1. Coarse evaluation granularity: The model effect is usually measured by "overall accuracy" or "F1", which lacks fine-grained structural coverage analysis and is difficult to find process-level blind spots.

[0058] 2. Test and training are separated: Misjudgment samples need to be manually exported, classified, and then Jira tasks are generated, and training patches are scattered in different systems, which is low in efficiency and slow in iteration.

[0059] 3. Difficulty in error attribution: There is a lack of traceable reasoning links, and the model cannot quickly locate the cause after making a prediction error.

[0060] 4. High risk of model online: Current model version evaluation mostly relies on manual comparison, lacks programmable health "gatekeeper mechanism", and has the risk of performance regression.

[0061] With the increasing application of conversational AI assistants, how to effectively evaluate and improve the natural language understanding capability (NLP (Natural Language Processing) model) of virtual assistants has become a key problem in practical implementation.

[0062] Based on this, the embodiment of the application provides a quality control system and method of a conversational AI model, adopts a mode based on a semantic test and an automatic training closed loop, integrates test execution, structure coverage analysis, false judgment attribution, training suggestion generation and closed loop model optimization, and realizes health monitoring and training feedback of the conversational AI model.

[0063] The target user, core pain points and value of the embodiment of the application of the conversational AI assistant are shown in Table 1.

[0064] Table 1

[0065]

[0066] Wherein, NLU refers to Natural Language Understanding, that is, natural language understanding.

[0067] In order to facilitate the understanding of the embodiment, first, a quality control system of a conversational AI model disclosed by the embodiment of the application is introduced in detail.

[0068] The embodiment of the application proposes an end-to-end intelligent quality control system for a conversational AI assistant, which integrates semantic test, error diagnosis, structured training suggestion generation and model training closed loop, and constitutes an integrated “test-monitoring-training-regression” system. The main innovations include:

[0069] 1. Three-dimensional coverage rate structure design: the test coverage range is expanded from the traditional “intention recognition accuracy” to the three-dimensional structure of “intention x flow node x flow path”, and the structure integrity of the dialogue flow is quantified by a DAG (Directed Acyclic Graph, Directed Acyclic Graph) matching method.

[0070] 2. Compound error recognition method of rule + clustering: combining interpretable rule judgment and deep semantic embedding clustering, automatically classifying FP (False Positive, False Positive) / FN (False Negative, False Negative) false judgment samples, and identifying potential confusion clusters to provide basis for boundary adjustment.

[0071] 3. Structured training suggestion automatic generation mechanism: each type of error is mapped to a standard suggestion template (such as NeedTraining for training, NeedNegativePattern for negative sample mode, etc.), and a training patch set can be generated by one-key adoption, realizing that recommendation is adoption and adoption is training.

[0072] 4. Dynamic confidence threshold adjustment and negative sample self-distillation mechanism: the system dynamically adjusts the model prediction confidence threshold according to the intent heat and error distribution, automatically injects the misjudgment samples as "negative samples" into retraining, and realizes boundary self-correction.

[0073] 5. Low-cost incremental training and health gatekeeper mechanism: by freezing the backbone parameters, only fine-tuning the Adapter / LoRA layer to complete patch training (also known as local training); Introduce a gatekeeper threshold mechanism, if the new model indicators are not up to standard, refuse to go online.

[0074] 6. Visualization quality dashboard + process DAG heat map: support visualizing test paths, uncovered paths and abnormal jumps in the form of flowcharts, and link to view corresponding samples, error details and suggestions.

[0075] The embodiment of the application provides an automatic closed-loop optimization quality control system capable of automatically connecting "test-analysis-optimization-regression verification-release", which improves development efficiency, diagnosis capability and online safety.

[0076] As shown in Figure 1 The quality control system of the dialog AI model provided by the embodiment of the application comprises:

[0077] The test execution module 101 is configured to execute batch / non-batch dialog testing of the dialog AI model to obtain dialog testing data; the dialog testing data can be, but is not limited to, a test log, and the test log can include test utterances, predicted intents, hit nodes and path jump records, etc.

[0078] The three-dimensional coverage analysis module 102 is configured to generate a three-dimensional coverage calculation result according to the input dialog testing data of the dialog AI model, and the three-dimensional coverage calculation result includes a three-dimensional coverage matrix corresponding to intents, process nodes and paths.

[0079] The retest recommendation module 103 is configured to perform test integrity analysis according to the three-dimensional coverage calculation result and the dialog flowchart corresponding to the dialog AI model to generate retest suggestions.

[0080] The threshold dynamic adjustment module 104 is configured to adjust the confidence threshold of each intent according to the intent error rate and intent heat in the dialog testing data to obtain new confidence threshold information of the dialog AI model.

[0081] The misjudgment sample analysis module 105 is configured to perform error classification and confusion cluster detection on the input misjudgment sample basic data through a rule engine and semantic clustering to obtain semantic type label data.

[0082] The training suggestion generation module 106 is configured to map the semantic type label data into structured training suggestion data.

[0083] The closed-loop training optimization module 107 is configured to generate training patch data according to the target set of structured suggestions adopted by the user in the structured training suggestion data, and to perform incremental fine-tuning on the current model of the conversational AI model using the training patch data to obtain a new model;

[0084] The gatekeeper control module 108 is configured to compare the regression test indicators of the new model and the current model, and to perform online deployment of the new model.

[0085] The quality control system for the conversational AI model provided by the embodiments of the present application integrates three-dimensional coverage analysis of semantic testing, supplementary testing recommendation, threshold dynamic adjustment, misjudgment sample analysis, structured training suggestion generation, model training closed loop, and gatekeeper control, to form an integrated "testing-analysis-optimization-regression verification-deployment" system, thereby improving development efficiency, diagnosis capability, and online safety.

[0086] Further optionally, as shown in Figure 1 The system further includes a self-distillation module 109 configured to use the sample set classified as false positives as a negative sample data set; and the closed-loop training optimization module 107 is further configured to inject the negative sample data set into the training patch data to achieve semantic boundary self-correction.

[0087] The goal of the self-distillation module 109 is to improve model boundary recognition capability and suppress fuzzy matching. The self-distillation module 109 can use the model misjudgment to automatically generate negative example samples for boundary convergence training; the input of the self-distillation module 109 can be the sample set classified as FP (i.e., the identified FP samples), and the output of the self-distillation module 109 is a negative sample data set, which is used by the closed-loop training optimization module 107 to merge and construct the training patch data.

[0088] The self-distillation module 109 can mark these FP samples as "non-target intent" samples, automatically inject them into the negative sample queue, and the label structure can be: label = negative, intent ≠ predicted_intent, i.e., the label is a negative sample, and the actual intent does not match the predicted intent. This process forms a "self-distillation" closed loop, which is equivalent to the model teaching itself the boundary. An example of a possible application scenario is that the model mistakenly identifies "cancel" as "return", and the system automatically uses "cancel" as a counterexample for "return" for training.

[0089] The target of the three-dimensional coverage analysis module 102 is to evaluate the NLP process coverage completeness in a structured manner and capture the "unmeasured" blind area. In some possible embodiments, the input of the three-dimensional coverage analysis module 102 can be batch / non-batch dialogue test logs, the format of which includes test utterances, predicted intents, hit nodes and path jump records, the three-dimensional coverage analysis module 102 can extract triples (intent, process node, path sequence) from the dialogue test logs, construct a three-dimensional coverage matrix to locate the process blind area, and the three-dimensional coverage calculation result output by the three-dimensional coverage analysis module 102 can include a three-dimensional coverage matrix and three-dimensional statistical data, the three-dimensional statistical data including coverage rate statistics table of each layer dimension (intent, process node, path). Based on this, the above three-dimensional coverage analysis module 102 is specifically used for: constructing a three-dimensional tensor composed of intent number, node number and path identifier according to the dialogue process graph corresponding to the dialogue AI model, and initializing the three-dimensional tensor; extracting triples composed of intent, process node and path sequence corresponding to each test utterance from the dialogue test data, and filling the extracted triple data into the initialized three-dimensional tensor to obtain a three-dimensional coverage matrix, wherein the triple data is filled into the corresponding coordinate position of the three-dimensional tensor, and 1 represents hit and 0 represents path not covered; calculating the intent coverage rate, node coverage rate and path coverage rate of the three-dimensional coverage matrix according to the dialogue process graph corresponding to the dialogue AI model to obtain three-dimensional statistical data.

[0090] The intent x process node x path constitutes a three-dimensional coverage evaluation system, and the process node and the path part are constructed on the dialogue process DAG, and the completeness of the process coverage, the interrupt point and the uncompleted path can be identified by traversing the DAG.

[0091] In specific implementation, the above three-dimensional tensor is denoted as C[ i|j|k ], i represents the intent number, j represents the node number, k represents the path identifier; before processing the dialogue test logs, the three-dimensional tensor needs to be constructed according to the dialogue process graph corresponding to the dialogue AI model and initialized, and the value of all positions after initialization is 0, wherein the dialogue process graph corresponding to the dialogue AI model is the process DAG definition (including all process nodes and paths); for each log, the predicted intent, hit node and process jump path are extracted from the current log entry, the intent number of the predicted intent is assigned to i , the node number of the hit node is assigned to j , the process jump path of the current dialogue is converted into a hash value, and the hash value is assigned to k ; in the three-dimensional tensor C, according to i , j and kSet the value as 1 on the position as the index. An example of a possible application scenario is that Li Gong submits a new model version, the system automatically performs testing and generates a process coverage graph, and finds that some paths have not been hit, prompting risks.

[0092] The goal of the supplementary test recommendation module 103 is to find uncovered intent-path combinations in the process structure and improve test integrity. In some possible embodiments, the supplementary test recommendation module 103 can detect uncovered intent-path combinations and automatically prompt the creation of supplementary test cases; the input of the supplementary test recommendation module 103 can be a three-dimensional coverage matrix and a dialog flowchart corresponding to a dialog AI model (i.e., a pre-defined flowchart DAG including a complete set of process paths), and the output of the supplementary test recommendation module 103 can include an uncovered path list and recommended supplementary test cases. Based on this, the above-mentioned supplementary test recommendation module 103 is specifically used for: completing and identifying the three-dimensional coverage matrix of the dialog flowchart through a directed acyclic graph (DAG) traverser and a path mapping function to obtain an uncovered path list; and generating recommended supplementary test cases according to the uncovered path list and a pre-set sample case template.

[0093] In specific implementation, the supplementary test recommendation module 103 can traverse all path combinations in the dialog flowchart i,j,k ), if C[ i|j|k ]=0, it is marked as uncovered (i.e., recorded as a "process blind spot"). The DAG traverser and the path mapping function are used for completion and identification, and an uncovered node can generate a template utterance to prompt a test personnel to construct a test set. An example of a possible application scenario is that the system prompts that the path of "return → logistics query" has never been tested, and Li Gong clicks the supplementary test suggestion to automatically generate a case template.

[0094] The goal of the threshold dynamic adjustment module 104 is to set adaptive decision boundaries for different intents (i.e., automatically set appropriate confidence boundaries for each intent) to prevent overfitting / underjudgment and improve accuracy. Intent confidence threshold dynamic adjustment belongs to the pre-training pre-optimization process and is used to adjust the intent prediction decision boundary of the intent recognition model (such as adjusting the confidence threshold of the softmax score). The input of the threshold dynamic adjustment module 104 can include the error rate of each intent (i.e., intent error rate, also known as error sample density) + usage frequency (i.e., intent heat), and the output of the threshold dynamic adjustment module 104 can be a new confidence threshold table for each intent, in which, for high-frequency and high-accuracy intents, the threshold is increased (to reduce false triggering), and for high-frequency and low-accuracy intents, the punishment is increased. In some possible embodiments, the above-mentioned threshold dynamic adjustment module 104 is used to adjust the confidence threshold through the following dynamic adjustment threshold formula:

[0095] ;

[0096] wherein,θ 0 represents the initial confidence threshold of the first i θ i 0 represents the new confidence threshold of the first i freq i 0 represents the intent heat of the first i err i 0 represents the intent error rate of the first i 0 represents the heat weight (the value can be 0.1-0.3), λ 0 represents the misjudgment penalty coefficient (the value can be 1.0-2.0). γ

[0097] One possible application scenario of the threshold dynamic adjustment module 104 is that the system finds that the "repair" intent is misjudged more, and automatically adjusts the threshold from 0.5 to 0.7.

[0098] The goal of the misjudgment sample analysis module 105 is to accurately identify the root cause of the prediction error and find the intent cluster group that is easy to confuse. In some possible embodiments, the misjudgment sample analysis module 105 can combine the interpretable rules and semantic vector clustering to classify the FP / FN samples, find the potential confusion boundary; the input of the misjudgment sample analysis module 105 can include all samples predicted as FP / FN (which can be FP / FN error samples in the dialogue test log) and the intent prediction log, which can include the original input, the predicted intent, the confidence score and the entity recognition result, etc. Based on this, the above misjudgment sample basic data includes the error sample and its corresponding intent label, confidence score and entity recognition result; the above misjudgment sample analysis module 105 is specifically used for: according to the intent label, confidence score and entity recognition result in the misjudgment sample basic data, using a rule engine to label the error type of the corresponding error sample, to obtain error label type data; wherein the error type includes intent mismatch, entity missing and low confidence; using a BERT (Bidirectional Encoder Representations from Transformers) model to generate a sample semantic vector of each error sample, and performing K-Means clustering on the sample semantic vectors of the error samples to obtain a clustering cluster analysis report, the clustering cluster analysis report includes the corresponding belonging clustering cluster identifier and clustering entropy value of each error sample; according to the error label type data and the clustering cluster analysis report, generate semantic type label data.

[0099] ​​​​The misjudgment sample analysis module 105 can first use a rule engine to preliminarily label the error type, including: intent mismatch, entity missing, and low confidence; and then perform semantic clustering, including: using a BERT model to process the input text to obtain a vector v i , using a K-Means algorithm to perform clustering analysis on a series of vectors v i and divide them into K clusters. The output of the misjudgment sample analysis module 105 can include an error classification label (i.e., the error label type data described above) of each error sample and a clustering cluster analysis report, which can include a confused cluster center intent group. The error classification label is a sample-level error label, and the error type of the error sample is accurately labeled by the error classification label. The purpose is to identify different types of recognition failures and behavior biases in the dialog AI and guide the subsequent automatic training patch generation and quality optimization process. The error type not only covers traditional intent recognition errors, but also refines to entity recognition, feature extraction, rule conflict, and other dimensions. The misjudgment sample analysis module 105 can output semantic type label data by combining error statistical attributes (whether FN / FP / TP, etc.), error label type data, and clustering cluster analysis reports.

[0100] In a specific implementation, the rule judgment of the misjudgment sample analysis module 105 in the first stage can include: if the predicted intent ≠ the expected intent, the error type is intent mismatch; if there is no hit entity, the error type is entity missing; and if the predicted confidence < the preset threshold, the error type is low confidence. The clustering induction in the second stage can include: using BERT to generate a sample semantic vector v i , performing K-Means clustering on FP / FN samples, and if the clustering cluster entropy is high, it is judged as a confused cluster. The clustering uses cosine distance to measure semantic similarity, and the center vector after clustering represents a "potential confused intent representative". A possible application scenario example is that Li Gong sees that "cancel order" is often misjudged as "return", and the system identifies it as a "semantic confused cluster" and recommends boundary optimization.

[0101] The goal of the training suggestion generation module 106 is to transform the analysis results of misjudged samples into standardized suggestion templates to support subsequent automatic processing. In some possible embodiments, the training suggestion generation module 106 can automatically map misjudged samples to standard structured suggestions (recommendations, reasons, related samples). The input of the training suggestion generation module 106 can include erroneous samples and their classification labels (i.e., semantic type labels). The structured training suggestion data output by the training suggestion generation module 106 can include a suggestion list (structured format) and related sample links (traceable). The structure of each suggestion object can include: sample ID, suggestion type, target intent, triggering rule, and suggestion level, such as a suggestion object of {sample ID, suggestion type, suggestion reason}. The suggestion reason can be generated based on the input and mapping rules, including: target intent, triggering rule, and suggestion level. The triggering rule of the structured suggestion can be determined by combining the error label type with conditional expressions (such as sentence length, confidence level, etc.).

[0102] Error samples are not only labeled with basic statistical labels such as FP and FN, but their semantic features and distribution clustering characteristics are further analyzed to generate more targeted suggestions. Therefore, the classification labels of error samples are more granular semantic type labels than FP / FN, as shown in Table 2 below. Here, FP / FN are statistical attributes, while "Error Label Type" is a semantic attribution.

[0103] Table 2

[0104]

[0105] Table 3 below is a standard mapping example: these rules are defined as mapping templates and can be automatically batch-processed into suggestion sets.

[0106] Table 3

[0107]

[0108] Table 4 shows another mapping rule, as follows:

[0109] Table 4

[0110]

[0111] In most semantic analysis and NLP scenarios (such as English scenarios), sentence length is measured in "tokens". To adapt to Chinese scenarios, this system supports two measurement methods: in English scenarios, the default unit for sentence length is the token, while in Chinese scenarios, the default unit for sentence length is the number of Chinese characters. For example, in Chinese scenarios, "sentence length < 3" usually refers to fewer than 3 Chinese characters (i.e., very short user phrases, such as "cancel" or "express delivery", commonly seen in FP / FN boundary errors).

[0112] A possible example of the application scenario of the training suggestion generation module 106 is as follows: The system marks the error sentence "returned" with a negative sample suggestion. Engineer Li clicks "Adopt All" and enters the patch training process.

[0113] The goal of the closed-loop training optimization module 107 is to quickly convert the adopted suggestions into a small sample training set (that is, automatically construct a small batch data set for fine-tuning after adopting the suggestions), and quickly update the model with extremely low resource costs (that is, quickly and at low cost fine-tune the model and shorten the iteration time). In some possible embodiments, for the construction of the small sample training set, the input of the closed-loop training optimization module 107 includes the adopted structured suggestion set and the original sample content. The closed-loop training optimization module 107 can synthesize the input into training patch data in JSONL format (that is, the patch training set). The structure of each patch data can include a sample, an intent, and a label, such as { "utterance": "returned", "intent": "cancel the order", "label": "negative"}. The data synthesis process can include: for the error samples that need to be trained, positive sample labels can be marked; for the error samples that need negative sample patterns, the same sentence can be set as multiple negative samples of non-intents. A possible example of the application scenario is: Engineer Li adopted 8 suggestions, and the system synthesized a batch of patch training sets and automatically sent them to the fine-tuning engine.

[0114] In some possible embodiments, for the fine-tuning training model, the input of the closed-loop training optimization module 107 includes the training patch data and the old model parameters (that is, the model parameters of the previous version). The output of the closed-loop training optimization module 107 can include a new model and a model metric log. The model metric log is a data file that records the changes in various performance metrics of the model during the training process. Based on this, the closed-loop training optimization module 107 is specifically used to: freeze the backbone model parameters in the current model of the conversational AI model, and use the training patch data to train the LoRA (Low-Rank Adaptation) layer or the Adapter layer to obtain a new model.

[0115] Specifically, the fine-tuning training method of the closed-loop training optimization module 107 can include: freezing the backbone model parameters and only training the LoRA layer or the Adapter layer. The number of parameters can be compressed to 1% - 3%. Among them, LoRA introduces a low-rank matrix, supports the reversibility and lightness of weight fine-tuning, and supports quick online and quick rollback. Exemplarily, BatchSize (batch size) = 32, Epoch (number of epochs) = 3, and the training duration ≤ 10 minutes. A possible example of the application scenario is: The training took 8 minutes. Engineer Li received that the new model v1.1 has been generated, and the system enters the regression verification stage.

[0116] The goal of the gatekeeper control module 108 is to prevent the performance degradation model from going online, protect the production quality, that is, ensure that the performance of the new model version is greater than or equal to the old version, and guarantee the regression stability. The input of the gatekeeper control module 108 is the new model, the old model and the regression test results (including full test indicators such as F1 score, fallback rate, etc.), and the output of the gatekeeper control module 108 is the judgment result of whether to allow online, such as online permission / blocking notification. In some possible embodiments, the above-mentioned gatekeeper control module 108 is specifically used for: obtaining the F1 score increment and the fallback rate increment according to the regression test results of the new model and the current model; judging whether the F1 score increment and the fallback rate increment pass the preset gatekeeper strategy to obtain the gatekeeper judgment result; if the gatekeeper judgment result is yes, the online deployment of the new model is performed; and if the gatekeeper judgment result is no, the backtracking optimization and / or blocking notification can be performed.

[0117] Optionally, the above-mentioned gatekeeper strategy can be:

[0118] ;

[0119] wherein, Δ F 1 represents the F1 score increment, and Δ fallback represents the fallback rate increment. 、 represents the preset increment threshold, and the two can be set according to actual needs, for example, , .

[0120] One possible application scenario of the gatekeeper control module 108 is that the new model F1 is improved by +3%, the fallback is reduced, and the system automatically allows online.

[0121] Further, the above-mentioned system further includes a backtracking suggestion generation module, which is used for generating a new round of training suggestion according to the new model (that is, the model version that does not pass the gatekeeper, including the regression failure indicators) and the corresponding failure samples (which can come from the regression error sample log) when the gatekeeper judgment result is no, wherein the regression error sample log is used for recording those data sets that are originally correctly classified or processed, but become incorrect or error in the new environment.

[0122] The goal of the rollback suggestion generation module is to quickly return to optimization when the model performance deteriorates. The rollback suggestion generation module can automatically generate training suggestions for the next round when the new model fails to go online, achieving a test-diagnosis-optimization-verification closed loop. In specific implementation, the rollback suggestion generation module can extract failed samples, call the misjudgment sample analysis module 105 to reanalyze misjudgment samples, then call the training suggestion generation module 106 to generate structured suggestions again, and then realize closed-loop restart through the closed-loop training optimization module. An example of a possible application scenario is that Li worker receives a system notification "v1.2 model fallback improves 5%, suggests adjusting 'query logistics' intent boundary", and generates new suggestions with one key.

[0123] For ease of understanding, the following refers to Figure 2 The overall workflow of the quality control system of the above-mentioned dialog AI model is introduced.

[0124] First, after the user submits a new model version to trigger testing, the test execution module 101 performs testing and generates a dialog test log; then, the three-dimensional coverage analysis module 102 generates a three-dimensional coverage matrix based on the dialog test log, and the supplementary testing recommendation module 103 uses the three-dimensional coverage matrix to mark un-covered paths for the user to re-use the test execution module 101 for supplementary testing; the threshold dynamic adjustment module 104 adjusts the dynamic threshold of the intent confidence in the model based on the dialog test log, which is then used for subsequent inference services (such as inference services when the closed-loop training optimization module 107 performs incremental model training); the misjudgment sample analysis module 105 identifies the error types of error samples in the dialog test log through rules and clustering, then the training suggestion generation module 106 generates structured training suggestions, and the self-distillation module 109 injects FP samples into negative examples; the closed-loop training optimization module 107 adopts suggestions and generates patch data for incremental model training; the gatekeeper control module 108 verifies the online gatekeeper based on the regression test results, if passed, it is released online; if failed, it is returned to optimization.

[0125] The quality control system of the above-mentioned conversational AI model can be divided into trigger layer, execution layer, calculation layer, diagnosis layer, suggestion layer, collaboration layer, training layer and release layer. Specifically, the trigger layer (Event): This layer is responsible for identifying and triggering the start of the entire process; it can start the tasks in the execution layer through Git Hook, Cron job, etc. The execution layer (Test): In this layer, batch testing / conversation testing will be performed; these tests can be run through Argo or Kubernetes Jobs, and involve the invocation of inference services such as machine learning models, rule engines or Knowledge Graph (KG). The calculation layer (Metrics): After the test is executed, this layer uses technologies such as Spark or Flink for data processing, calculating indicators such as true positive (TP), false positive (FP), false negative (FN) and coverage (Coverage); at the same time, the Trace log aggregator collects and analyzes log information in this layer. The diagnosis layer (Errors): Based on the data and logs obtained from the previous layer, this layer uses rules plus BERT clustering methods to label and cluster errors, in order to better understand the causes and patterns of errors. The suggestion layer (Rec): According to the diagnosis results, this layer provides suggestions on whether to train, adjust the model or other operations through template matching and Gradient Boosting Decision Tree (GBDT) algorithms. The collaboration layer (Tasks): Convert the problems or suggestions found into specific tasks, such as integrating RedisStream with Jira to manage the state changes of tasks (Pending→Done, i.e. pending→completed). The training layer (AutoTrain): For the training needs proposed by the suggestion layer, this layer uses tools such as Kubeflow or LoRA for partial fine-tuning to achieve automated model updating. The release layer (Gate): Finally, after regression testing ensures the quality of the new version, if certain threshold conditions are met, the upgrade or release is performed through Helm.

[0126] One possible system hierarchical structure of the above-mentioned conversational AI model quality control system can be shown in Table 5.

[0127] Table 5

[0128]

[0129] The following is an example of user behavior scenarios under the above-mentioned conversational AI model quality control system, taking user Xiaowang as an engineer.

[0130] 1. Xiaowang submits model v1.1→ system automatic testing→ coverage analysis finds two processes not tested;

[0131] 2. Error samples are clustered into two confused clusters → "boundary tuning is needed" suggestion is generated;

[0132] 3. Xiao Wang adopts the suggestion → patch data is automatically generated and fine-tuned → takes 8 minutes;

[0133] 4. The system adjusts the "complaint / return" intention confidence according to the frequency, and the FP sentence "returned" is injected as a negative example;

[0134] 5. Regression test passes, F1 value increases by 2%, fallback decreases by 1%, and the model is successfully released;

[0135] 6. If the regression test fails → the system restarts the suggestion, and the closed-loop restarts.

[0136] In summary, some key innovations of the embodiment of the application are shown in Table 6.

[0137] Table 6

[0138]

[0139] The embodiment of the application has the following advantages:

[0140] 1) Improve recognition accuracy and boundary judgment ability: dynamic threshold adjustment + self-distillation strategy, optimize model boundary behavior;

[0141] 2) Error attribution automation, clear structure: rules + clustering classifier efficiently clusters confused intentions, easy to train and correct;

[0142] 3) Form a complete training closed loop: break through the closed loop process from error recognition → patch training → regression verification;

[0143] 4) Process structure transparency and visualization: support developers to view test coverage status and problem hotspots from the perspective of process diagram.

[0144] Corresponding to the quality control system of the dialog AI model described above, the embodiment of the application also provides a quality control method of a dialog AI model. Referring to Figure 3 the flowchart of the quality control method of the dialog AI model, the method mainly includes the following steps S310 to S370:

[0145] Step S310, the three-dimensional coverage analysis module generates a three-dimensional coverage calculation result according to the input dialog test data of the dialog AI model, and the three-dimensional coverage calculation result includes a three-dimensional coverage matrix corresponding to the intention, the flow node and the path.

[0146] Step S320, the test recommendation module analyzes the test integrity according to the three-dimensional coverage calculation result and the dialog flowchart corresponding to the dialog AI model, and generates a test recommendation.

[0147] Step S330, the threshold dynamic adjustment module adjusts the confidence threshold of each intent according to the intent error rate and the intent heat in the dialog test data, and obtains new confidence threshold information of the dialog AI model.

[0148] Step S340, the misjudgment sample analysis module performs error classification and confusion cluster detection on the input misjudgment sample basic data through a rule engine and semantic clustering, and obtains semantic type label data.

[0149] Step S350, the training suggestion generation module maps the semantic type label data into structured training suggestion data.

[0150] Step S360, the closed-loop training optimization module generates training patch data according to the target structured suggestion set adopted by the user in the structured training suggestion data, and performs incremental fine-tuning on the current model of the dialog AI model using the training patch data, to obtain a new model.

[0151] Step S370, the gatekeeper control module compares the regression test indicators of the new model and the current model, and deploys the new model online.

[0152] The quality control method of the dialog AI model provided by the embodiment of the application integrates three-dimensional coverage analysis, test recommendation, threshold dynamic adjustment, misjudgment sample analysis, structured training suggestion generation, model training closed loop, and gatekeeper control, to form an integrated "test-analysis-optimization-regression verification-deployment" system, thereby improving development efficiency, diagnosis capability, and online safety.

[0153] The embodiment of the application also provides a quality control method of a dialog AI model, wherein a self-distillation mechanism of a self-distillation module is newly added in the method: the self-distillation module takes a sample set classified as false positive as a negative sample data set, and the closed-loop training optimization module also injects the negative sample data set into the training patch data, to realize semantic boundary self-correction. Figure 4 Another flowchart of a quality control method of a dialog AI model is shown in the figure, and the method comprises the following steps:

[0154] Step S410, a three-dimensional coverage analysis module generates a three-dimensional coverage calculation result according to input dialog test data of a dialog AI model.

[0155] Step S420, a test recommendation module analyzes the test integrity according to the three-dimensional coverage calculation result and a dialog flowchart corresponding to the dialog AI model, and generates a test recommendation.

[0156] In step S430, the threshold dynamic adjustment module adjusts the confidence threshold of each intent based on the intent error rate and intent popularity in the dialogue test data to obtain the new confidence threshold information of the conversational AI model.

[0157] In step S440, the misjudged sample analysis module uses a rule engine and semantic clustering to perform misclassification and confusion cluster detection on the input misjudged sample basic data to obtain semantic type label data.

[0158] In step S450, the training suggestion generation module maps the semantic type label data into structured training suggestion data, and the self-distillation module uses the sample set classified as false positives as the negative sample dataset.

[0159] In step S460, the closed-loop training optimization module generates training patch data based on the target structured suggestion set adopted by the user in the structured training suggestion data, injects the negative sample dataset into the training patch data, and uses the training patch data to incrementally fine-tune the current model of the conversational AI model to obtain a new model.

[0160] In step S470, the gate control module compares the regression test metrics of the new model and the current model to deploy the new model online.

[0161] The quality control method for the conversational AI model provided in this embodiment has the same implementation principle and technical effect as the aforementioned quality control system embodiment for the conversational AI model. For the sake of brevity, any parts not mentioned in the quality control method embodiment for the conversational AI model can be referred to the corresponding content in the aforementioned quality control system embodiment for the conversational AI model.

[0162] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0163] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0164] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0165] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A quality control system based on a conversational AI model, characterized in that, include: The 3D coverage analysis module is used to generate 3D coverage calculation results based on the dialogue test data of the input conversational AI model. The 3D coverage calculation results include 3D coverage matrices corresponding to intents, process nodes, and paths. The supplementary test recommendation module is used to perform test integrity analysis based on the 3D coverage calculation results and the dialogue flowchart corresponding to the conversational AI model, and generate supplementary test suggestions. The threshold dynamic adjustment module is used to adjust the confidence threshold of each intent based on the intent error rate and intent popularity in the dialogue test data, so as to obtain the new confidence threshold information of the conversational AI model. The misclassification sample analysis module is used to perform misclassification and confusion cluster detection on the input misclassified sample basic data through rule engine and semantic clustering to obtain semantic type label data; The training suggestion generation module is used to map the semantic type label data into structured training suggestion data; The closed-loop training optimization module is used to generate training patch data based on the target structured suggestion set adopted by the user in the structured training suggestion data, and to incrementally fine-tune the current model of the conversational AI model using the training patch data to obtain a new model. The gatekeeper control module is used to deploy the new model online by comparing the regression test metrics of the new model and the current model. The basic data of the misjudged samples includes the erroneous samples and their corresponding intent labels, confidence scores, and entity recognition results; The process involves using a rule engine and semantic clustering to perform misclassification and confusion cluster detection on the input misclassified sample base data, resulting in semantic type label data, including: Based on the intent labels, confidence scores, and entity recognition results in the basic data of the misjudged samples, the rule engine is used to label the corresponding erroneous samples with error types to obtain error label type data; wherein, the error types include intent mismatch, entity missing, and low confidence. The BERT model is used to generate a sample semantic vector for each of the error samples, and K-Means clustering is performed on the sample semantic vectors of each of the error samples to obtain a cluster analysis report. The cluster analysis report includes the cluster identifier and cluster entropy value corresponding to each error sample. The semantic type label data is generated based on the error label type data and the cluster analysis report.

2. The system according to claim 1, characterized in that, The system also includes a self-distillation module for using the sample set that will be classified as a false positive as a negative sample dataset; The closed-loop training optimization module is also used to inject the negative sample dataset into the training patch data to achieve semantic boundary self-correction.

3. The system according to claim 1, characterized in that, The dialogue test data includes test utterances, predicted intents, hit nodes, and path jump records; the 3D coverage calculation results also include 3D statistical data; the generation of 3D coverage calculation results based on the input dialogue test data of the conversational AI model includes: Extract the triples consisting of intent, flow node, and path sequence corresponding to each test utterance from the dialogue test data, and fill the extracted triple data into a three-dimensional tensor consisting of intent number, node number, and path identifier to obtain a three-dimensional coverage matrix. The intent coverage rate, node coverage rate, and path coverage rate are calculated on the three-dimensional coverage matrix to obtain the three-dimensional statistical data.

4. The system according to claim 1, characterized in that, The supplementary testing recommendations include a list of uncovered paths and recommended supplementary test cases; the step of performing test integrity analysis based on the 3D coverage calculation results and the dialogue flowchart corresponding to the conversational AI model to generate supplementary testing recommendations includes: By using a directed acyclic graph traverser and a path mapping function, the dialogue flowchart is filled with the three-dimensional coverage matrix to obtain a list of uncovered paths. Based on the list of uncovered paths and the preset sample test case template, recommended supplementary test cases are generated.

5. The system according to claim 1, characterized in that, The threshold dynamic adjustment module is used to adjust the confidence threshold using the following formula: ； in, θ 0 indicates the first i The initial confidence threshold for an intent. θ i Indicates the first i A new confidence threshold for an intent freq i Indicates the first i The popularity of an intention. err i Indicates the first i Intent error rate per intent, λ Indicates the weight of popularity. γ This represents the penalty coefficient for misjudgment.

6. The system according to claim 1, characterized in that, The step of incrementally fine-tuning the current model of the conversational AI model using the training patch data to obtain a new model includes: Freeze the backbone model parameters in the current model of the conversational AI model, and train the LoRA layer or Adapter layer using the training patch data to obtain the new model.

7. The system according to claim 1, characterized in that, The step of deploying the new model online by comparing the regression test metrics of the new model and the current model includes: Based on the regression test results of the new model and the current model, the increase in F1 score and the increase in regression rate are obtained; Determine whether the increase in the F1 score and the increase in the backsliding rate pass the preset gatekeeping strategy to obtain the gatekeeping judgment result; If the goalkeeper judgment result is yes, then the new model will be deployed online.

8. A quality control method for a conversational AI model, characterized in that, include: The 3D coverage analysis module generates 3D coverage calculation results based on the dialogue test data of the input conversational AI model. The 3D coverage calculation results include 3D coverage matrices corresponding to intents, process nodes, and paths. The supplementary testing recommendation module performs a test integrity analysis based on the 3D coverage calculation results and the dialogue flowchart corresponding to the conversational AI model, and generates supplementary testing suggestions. The threshold dynamic adjustment module adjusts the confidence threshold of each intent based on the intent error rate and intent popularity in the dialogue test data to obtain the new confidence threshold information of the conversational AI model. The misclassified sample analysis module uses a rule engine and semantic clustering to perform misclassification and confusion cluster detection on the input misclassified sample basic data, and obtain semantic type label data. The training suggestion generation module maps the semantic type label data into structured training suggestion data; The closed-loop training optimization module generates training patch data based on the target structured suggestion set adopted by the user in the structured training suggestion data, and uses the training patch data to incrementally fine-tune the current model of the conversational AI model to obtain a new model. The gate control module compares the regression test metrics of the new model and the current model to deploy the new model online. The basic data of the misjudged samples includes the erroneous samples and their corresponding intent labels, confidence scores, and entity recognition results; The process involves using a rule engine and semantic clustering to perform misclassification and confusion cluster detection on the input misclassified sample base data, resulting in semantic type label data, including: Based on the intent labels, confidence scores, and entity recognition results in the basic data of the misjudged samples, the rule engine is used to label the corresponding erroneous samples with error types to obtain error label type data; wherein, the error types include intent mismatch, entity missing, and low confidence. The BERT model is used to generate a sample semantic vector for each of the error samples, and K-Means clustering is performed on the sample semantic vectors of each of the error samples to obtain a cluster analysis report. The cluster analysis report includes the cluster identifier and cluster entropy value corresponding to each error sample. The semantic type label data is generated based on the error label type data and the cluster analysis report.

9. The method according to claim 8, characterized in that, The method further includes: The self-distillation module will classify the sample set that was classified as a false positive as the negative sample dataset; The closed-loop training optimization module also injects the negative sample dataset into the training patch data to achieve semantic boundary self-correction.

Images