SOP structured data generation and optimization method for driving model evaluation

Abstract

The application discloses a kind of SOP structured data generation and optimization method of driving model evaluation, it is related to model evaluation optimization technical field, it solves the technical problem that existing technology relies on real data and leads to high compliance risk, insufficient scene coverage, and lack of node-level evaluation and closed-loop optimization mechanism.The present application comprises: S1 obtains target scene SOP, analyzes and generates node link and defines state space, obtains node state sequence;S2 constructs a script template library for each node, generates multi-round dialogue text and corresponding real labels based on the template library and node state sequence;S3 inputs the dialogue text into the prediction model to obtain the predicted label, compares it with the real label and calculates the multi-dimensional performance indicators;S4 optimizes the prediction model and the script template library according to the performance indicators, and repeats S1-S4 until the preset conditions are met.The present application has the advantages of not relying on real data, realizing optimization closed loop to improve efficiency and stability, etc.

Description

Technical Field

[0001] This invention belongs to the field of model evaluation and optimization technology, specifically relating to a method for generating and optimizing SOP structured data to drive model evaluation. Background Technology

[0002] Current dialogue understanding systems are widely used in sales, customer service, follow-up calls, and inquiry scenarios. These systems typically rely on large amounts of real audio recordings or text data for training and evaluation of recognition models. However, real data not only contains a large amount of private information, posing difficulties in obtaining it and compliance risks, but also often fails to cover many low-frequency, boundary, or complex process scenarios in actual business, leading to unstable model performance and difficulty in iterative optimization.

[0003] Existing technologies mainly focus on the analysis of real voice data, keyword extraction, and path scoring. For example, by obtaining user reviews of historical products and combining them with the distribution of users in different voice response paths, each voice path is scored. This type of method first collects sales call recordings and converts them into text through Automatic Speech Recognition (ASR). Then, Natural Language Processing (NLP) is used to perform keyword recognition and semantic analysis on the text to build customer profiles. However, its recognition results mainly rely on keyword matching and template extraction, making it difficult to understand the latent semantic relationships in the dialogue context. At the same time, model optimization relies on manual rules and lacks automated feedback and tuning mechanisms.

[0004] Another type of existing technology typically uses speech recognition to convert sales recordings into text, and then uses keyword extraction, rule template matching, or natural language processing to identify key information such as the user's budget, preferences, and intended use to build a customer profile. These methods are all based on real recordings, rely on keywords, templates, or fixed path structures, and do not involve data synthesis, systematic model evaluation, or automatic closed-loop optimization capabilities. Therefore, there is an urgent need to design a SOP-based structured data generation and optimization method to drive model evaluation and address these issues. Summary of the Invention

[0005] This invention discloses a method for generating and optimizing SOP structured data to drive model evaluation, which aims to solve the technical problems of existing technologies that rely on real data, resulting in high compliance risks, insufficient scenario coverage, and a lack of node-level evaluation and closed-loop optimization mechanisms.

[0006] To solve the aforementioned technical problems, the present invention adopts the following technical solution:

[0007] A method for generating and optimizing SOP-based structured data to drive model evaluation includes the following steps:

[0008] S1: Obtain the Standard Operating Procedure (SOP) of the target scenario, parse it to generate node links, define the state space for each node, and obtain the node state sequence;

[0009] S2: Build a dialogue template library for each node, generate multi-turn dialogue text based on the dialogue template library and the node state sequence, and generate corresponding real tags;

[0010] S3: Input the dialogue text into the existing prediction model to obtain the predicted labels of the nodes, compare the predicted labels with the real labels, and calculate the multidimensional performance index of the prediction model.

[0011] S4. Optimize the prediction model and the script template library based on the multidimensional performance indicators, and repeat steps S1-S4 until the preset conditions are met.

[0012] In this invention, the above steps realize the generation, evaluation, and iterative optimization of structured data for a dialogue model based on SOP, completing the entire process from process parsing, data construction, model prediction to indicator optimization. Specifically, step S1 transforms the business SOP into a node link and node state sequence that the algorithm can process, building the logical foundation for subsequent data generation; step S2 generates dialogue text based on the template library and matches it with the final real labels, providing standard data for model evaluation; step S3 inputs the dialogue text into the model to obtain predicted labels, and quantifies the model performance by comparing it with the final real labels; step S4 optimizes the model and template library based on performance indicators, and iteratively executes the process until the preset conditions are met, achieving continuous optimization of the model.

[0013] Further, S1 includes:

[0014] S11: Obtain the standard operating procedure (SOP) for the target scenario from the business side, parse it, and identify and extract the key nodes;

[0015] S12: Assign a unique node ID to each node and build node links based on the order and dependencies between nodes;

[0016] S13: Define a state space for each node, and obtain the node state sequence based on the state space.

[0017] After adopting this technical solution, it should be noted that S11's direct acquisition of original SOPs from the business side ensures the accuracy of process parsing. The SOPs can be in the form of documents, flowcharts, or configuration tables, and are parsed and extracted using LLM or VLLM large models. The parsing rules include key node identification rules: extracting steps with business decision-making significance or semantic transition effects from the SOP materials as key nodes, and abstracting each node to include semantic labels and natural language descriptions. For the same semantic node in different business scenarios, consistent semantic labels and natural language descriptions are used. The output rules also include: outputting all key node sequences and their logical relationships in the form of mind maps, including sequence, branches, loops, or jumps. Both the key node identification and output rules mentioned above are existing technologies. Extracting key nodes can locate the core business steps in the dialogue. These key nodes are steps with business decision-making significance or semantic transition effects, such as opening confirmation, requirement inquiry, identity verification, and objection handling. Each identified step is abstracted as a node. In S12, fixed-length English letter codes, such as "anjc" and "rate," are used as node IDs for reference and reuse across scenarios. Furthermore, based on the Standard Operating Procedure (SOP), the order, dependencies, and possible transitions between nodes are analyzed to construct a flowchart structure with directed edges. For cases involving branches, loops, or transitions, their logical relationships are preserved for subsequent traversal or reduction as needed. In S13, node state sequences are generated through a state space, transforming the process into composable state units and providing a standardized logical basis for subsequent dialogue text generation.

[0018] Preferably, in S13, the state space includes three states: Y state representing affirmative expression, N state representing negative expression, and I state representing ambiguous, unclear, or noisy expression. The node state sequence is formed by enumerating the states of all nodes.

[0019] After adopting this technical solution, it should be noted that the three states Y, N, and I are used to describe the three main types of responses that customers may make at the nodes, which can be adapted to affirmative, negative, ambiguous, and noisy expression scenarios in real dialogues; by enumerating all node states to generate sequences, diverse dialogue logic paths can be constructed to meet the diverse needs of batch dialogue data generation.

[0020] Furthermore, the generation of multi-turn dialogue text in S2 includes:

[0021] S21: Collect historical dialogues and manual templates by node and state, expand diverse dialogues using existing generation models, and build a globally reusable noisy dialogue library for state I.

[0022] S22: Perform text standardization processing on the historical scripts, manual templates, and the diverse scripts, and perform deduplication and merging;

[0023] S23: The script templates for the three states are stored in a structured form of node ID + state type;

[0024] S24: Based on the node state sequence, traverse the node links, and randomly select dialogue text from the structured storage according to the state of each node to generate multi-turn dialogue text;

[0025] S25: Generate initial true labels based on the state sequence, and apply label correction rules to correct the business semantic consistency of the initial true labels to obtain the final true labels.

[0026] After adopting this technical solution, it should be noted that in S21, representative customer response statements are extracted from historical call data using nodes and three states (Y, N, I) as a framework. These statements are then manually filtered, cleaned, and categorized into their corresponding nodes and states. For nodes with scarce data, basic templates can be manually written to supplement the data. The generated model utilizes a large language model, such as DeepSeeK-V3, to paraphrase typical sentence structures and extend their style, generating more dialogue with slightly different tones and wording but consistent semantics, thereby increasing the diversity of dialogue in each state. In S22, text normalization processing includes removing obvious slips of the tongue, garbled characters, and repetitive sentences. It can also impose length limits on excessively long or short statements, which are removed using regular expression filtering. In S23, the structured storage of node ID + status type enables rapid retrieval and extraction of dialogue scripts. For example, using JSON or a table format, the node ID + status type can be used as the key, and the corresponding dialogue script list can be used as the value. In S24, traversing the node link to select dialogue scripts ensures that the dialogue text conforms to the business process logic. For each status sequence, the node link is traversed sequentially, and a corresponding dialogue script is randomly selected from the template library according to the node ID and status type as the customer's response content for that node, thereby generating a complete and well-structured dialogue text. In S25, the initial labels are corrected through label correction rules, which can ensure that the final real labels are consistent with the business semantics, providing an accurate benchmark for model evaluation.

[0027] Preferably, a perturbation mechanism is introduced when generating multi-turn dialogue text in S24. The perturbation mechanism includes: terminating the dialogue prematurely at a certain node with a preset probability to simulate hanging up behavior; and outputting null values ​​or noise content at some nodes.

[0028] By adopting this technical solution, the introduction of a perturbation mechanism can simulate abnormal behaviors such as user hanging up, no response, and noise interference in real dialogues, thereby improving the realism of the synthesized dialogue text and making the generated dialogues closer to actual interaction scenarios.

[0029] Further, S3 includes the following steps:

[0030] S31: Semantic parsing of each node is performed using a prediction model to output the corresponding predicted label;

[0031] S32: Align the predicted labels with the final true labels node by node ID;

[0032] S33: Calculate the multidimensional performance index of the prediction model based on the alignment results.

[0033] After adopting this technical solution, it should be noted that in S31, for dialogues determined to be genuine customer statements, they are input into the prediction model. The model can be a rule engine, machine learning classifier, deep learning model, or large language model, responsible for semantic parsing of each node or the entire dialogue and outputting the corresponding label prediction results. In S32, each node corresponds to three Y / N / I states, that is, the three values ​​of the label. Here, the state labels of the predicted label and the node ID are aligned. After node-by-node alignment, it is saved in a structured format. For example, a record contains the fields of sample ID, node ID, real label, and predicted label.

[0034] Preferably, the multidimensional performance metrics include: accuracy, precision, miss rate, recall rate, and node-level confusion matrix.

[0035] Preferably, S3 further includes:

[0036] S34: Based on multi-dimensional performance metrics and node-level confusion matrices, generate automated evaluation reports containing structured tables and visual charts.

[0037] After adopting this technical solution, it should be noted that S34 specifically includes:

[0038] S341: Based on the final true labels and the predicted labels, construct a benchmark data table by aligning the sample IDs and node IDs;

[0039] S342: Based on the aforementioned baseline data table, count the number of correct and incorrect predictions, calculate the overall accuracy, and break it down by label or scenario dimension to calculate the precision, accuracy, and miss rate under different labels, with the formulas as follows:

[0040] Accuracy = (TP + TN) / (TP + TN + FP + FN)

[0041] Accuracy = TP / (TP + FP)

[0042] Leakage rate = FN / (TP + FN)

[0043] In the formula, TP represents a true positive, TN represents a true negative, FP represents a false positive, and FN represents a false negative.

[0044] S343: Construct a confusion matrix by node ID or tag type.

[0045] In S343, the misclassification flow among categories Y, N, and I is analyzed, such as the proportion of Y being misclassified as N and the proportion of I being misclassified as Y, so as to describe the model's performance at each node and in each state in a more granular way.

[0046] S344: Output the precision, recall, and miss rates in the form of HTML reports, Excel spreadsheets, or JSON files. The reports may include tables, bar charts, line charts, etc., to facilitate developers to quickly browse and compare the performance of different model versions.

[0047] Further, S4 includes:

[0048] S41: Filter bad case samples where the final true label and the predicted label are inconsistent from the evaluation results generated based on multi-dimensional performance indicators and node-level confusion matrix;

[0049] In S41, based on the benchmark data table, all records where the true label and the predicted label are inconsistent are identified, and the records, their original dialogues, and node information are aggregated into a badcase set.

[0050] S42: Classify, aggregate, and cluster badcase samples by node ID, label type, and error type to extract common error features.

[0051] In S42, the badcase set is grouped according to dimensions such as node ID, true label, and predicted label. For example, all samples that should be Y but are predicted as N are aggregated, or samples with noisy expressions under a certain node are easily identified as positive, in order to discover the error concentration pattern. For extracting common error features, text and distribution analysis are performed on each type of badcase to identify common triggering reasons, such as specific sentence patterns, ambiguous expressions, long sentences, short sentence features, or misjudgments caused by the context before and after certain nodes, and from this, directions that can be targeted for correction are extracted.

[0052] Preferably, S4 further includes:

[0053] S43: Develop optimization plans based on the clustering results of bad case samples, and optimize the prediction model and the script template library accordingly;

[0054] In S43, based on the bad case analysis results, targeted optimization measures are formulated, such as emphasizing the processing rules of a certain type of expression in the prompt words, adding supplementary rules to the rule engine, adjusting the labeling of the speech template, or adding typical bad cases as incremental training data to the model training set.

[0055] S44: Repeat S1-S4 until the model performance reaches the preset convergence criterion or the improvement stagnates, mark the abnormal nodes and prompt manual intervention for analysis. S44 specifically includes:

[0056] S441: Based on the optimization scheme, update the model parameters, prompt word templates, rule engine logic or speech template library, and release the new version configuration or model in the system;

[0057] S442: For the same or updated synthetic dataset, perform prediction and evaluation again according to the process of step S3 to obtain a new round of performance indicators, ensuring that the optimization effect can be quantified;

[0058] S443: Compare the evaluation results of each round longitudinally, and plot the change curves of indicators such as accuracy, precision, and missing rate at different iteration rounds to determine whether the overall performance continues to improve and whether there are signs of convergence.

[0059] S444: When certain metrics show limited improvement or even fluctuations after multiple iterations, they can be considered to have reached the performance ceiling under the current configuration. The system marks the relevant nodes or tags as anomalies requiring further manual analysis, prompting R&D or business personnel to make deeper adjustments, thus forming a comprehensive closed loop of automatic optimization and manual decision-making.

[0060] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0061] 1. This application provides a method for generating and optimizing SOP structured data for driving model evaluation. It abstracts the business steps that originally existed as text or flowcharts into a traversable node link and defines a unified Y, N, I multi-state space for each node, thereby achieving an upgrade from sentence-level semantics to process-level semantics.

[0062] 2. This application provides a method for generating and optimizing SOP structured data to drive model evaluation. By enumerating node state combinations and automatically filling them with a script template library of node ID + state, it provides a perturbation mechanism to generate large-scale, highly realistic simulated dialogues, providing a very solid and flexible data foundation for model training and evaluation.

[0063] 3. This application provides a SOP-based structured data generation and optimization method for driving model evaluation. By calculating multi-dimensional indicators such as accuracy, precision, false negative rate, and recall rate, and constructing a node-level confusion matrix, it can quantify and analyze the misjudgment patterns of the model at different nodes and under different states in a fine-grained manner. At the same time, it generates a visual evaluation report in HTML, Excel, or JSON format, transforming abstract performance data into intuitive and readable results, which greatly improves evaluation efficiency. Attached Figure Description

[0064] The present invention will be described by way of example and with reference to the accompanying drawings, wherein:

[0065] Figure 1 This is an overall flowchart of the present invention;

[0066] Figure 2 This is a comparative diagram of Embodiment 2 of the present invention;

[0067] Figure 3 This is the confusion matrix diagram of the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the embodiments and accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and marked in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0069] The following is combined Figures 1-3 The present invention will be described in detail below.

[0070] Example 1

[0071] A method for generating and optimizing SOP-based structured data to drive model evaluation, such as Figure 1 As shown, it includes the following steps:

[0072] S1: Obtain the Standard Operating Procedure (SOP) of the target scenario, parse it to generate node links, define the state space for each node, and obtain the node state sequence;

[0073] S2: Build a dialogue template library for each node, generate multi-turn dialogue text based on the dialogue template library and the node state sequence, and generate corresponding real tags;

[0074] S3: Input the dialogue text into the prediction model to obtain the predicted labels of the nodes, compare the predicted labels with the real labels, and calculate the multidimensional performance index of the prediction model.

[0075] S4. Optimize the prediction model and the script template library based on the multidimensional performance indicators, and repeat steps S1-S4 until the preset conditions are met.

[0076] The above steps achieve structured data generation, evaluation, and iterative optimization of the dialogue model based on SOP, completing the entire process from process parsing, data construction, model prediction to indicator optimization. Specifically, step S1 transforms the business SOP into a node link and node state sequence that the algorithm can process, building the logical foundation for subsequent data generation; step S2 generates dialogue text based on the template library and matches it with the final real labels, providing standard data for model evaluation; step S3 inputs the dialogue text into the model to obtain predicted labels, and quantifies the model performance by comparing it with the final real labels; step S4 optimizes the model and template library based on performance indicators, iteratively executing the process until the preset conditions are met, achieving continuous model optimization.

[0077] In this embodiment, S1 includes:

[0078] S11: Obtain the standard operating procedure (SOP) for the target scenario from the business side, parse it, and identify and extract the key nodes;

[0079] In S11, directly obtaining the original SOP from the business side ensures the accuracy of process parsing. The SOP can be in the form of a document, flowchart, or configuration table, and is parsed and extracted using an LLM or VLLM large model. The parsing rules include key node identification rules: extracting steps with business decision-making significance or semantic transition effects from the SOP material as key nodes, and abstracting each node to include semantic labels and natural language descriptions. For the same semantic node in different business scenarios, consistent semantic labels and natural language descriptions are used. The output rules use a mind map to output the sequence of all key nodes and their logical relationships, including sequence, branches, loops, or jumps. The above key node identification and output rules are existing technologies. Extracting key nodes can locate the core business steps in the dialogue. Key nodes are steps with business decision-making significance or semantic transition effects, such as opening confirmation, requirement inquiry, identity verification, and objection handling. Each identified step is abstracted as a node.

[0080] S12: Assign a unique node ID to each node and build node links based on the order and dependencies between nodes;

[0081] In S12, fixed-length English letter codes, such as anjc and rate, are used as node IDs to facilitate referencing and cross-scenario reuse in the program. In addition, based on SOP, the order, dependencies, and possible jumps between nodes are analyzed to construct a flowchart structure with directed edges. For cases with branches, loops, or jumps, their logical relationships are maintained so that they can be traversed or reduced as needed later.

[0082] S13: Define a state space for each node, and obtain the node state sequence based on the state space;

[0083] In S13, the node state sequence is generated through the state space, which can transform the process into composable state units and provide a standardized logical basis for subsequent dialogue text generation.

[0084] In S13, the state space includes three states: Y state representing affirmative expressions, N state representing negative expressions, and I state representing ambiguous, unclear, or noisy expressions. The node state sequence is formed by enumerating the states of all nodes. The Y, N, and I states describe the three main types of responses a customer might give at a node, adapting to affirmative, negative, ambiguous, and noisy expression scenarios in real-world dialogues. By enumerating the sequence of node states, diverse dialogue logic paths can be constructed, meeting the diverse needs of batch dialogue data generation.

[0085] In this embodiment, generating multi-turn dialogue text in S2 includes:

[0086] S21: Collect historical dialogues and manual templates by node and state, expand diverse dialogues using existing generation models, and build a globally reusable noisy dialogue library for state I.

[0087] In S21, a framework of nodes and three states (Y, N, I) is used. Representative customer responses are extracted from historical call data and manually filtered, cleaned, and categorized into the corresponding nodes and states. For nodes with scarce data, basic templates can be manually written to supplement the data. The generated model utilizes a large language model, such as DeepSeeK-V3, to paraphrase typical sentences and extend their style, generating more dialogues with slightly different tones and wording but consistent semantics, thus increasing the diversity of dialogues in each state. In addition, a global noise dialogue library is built specifically for the I state, collecting various responses without clear business meaning, such as "I can't hear you," "What are you saying," "Who are you?" and "I'm driving." These dialogues are not bound to a specific node but can be reused in the I state of all nodes. When the node state is I, a noise dialogue is randomly extracted from the independent noise library to replace the regular template, so that a large amount of natural language interference that is not directly related to the business is interspersed throughout the dialogue, improving the authenticity and robustness of the data.

[0088] S22: Perform text standardization processing on the historical scripts, manual templates, and the diverse scripts, and perform deduplication and merging;

[0089] In S22, text normalization processing includes removing obvious slips of the tongue, garbled characters, and repetitive sentences. It can also impose length limits on sentences that are too long or too short, and uses regular expression filtering to remove them.

[0090] S23: The script templates for the three states are stored in a structured form of node ID + state type;

[0091] In S23, the structured storage of node ID + status type enables fast retrieval and extraction of dialogue. For example, using JSON or table format, node ID + status type can be used as the key and the corresponding dialogue list as the value.

[0092] S24: Based on the node state sequence, traverse the node links, and randomly select dialogue text from the structured storage according to the state of each node to generate multi-turn dialogue text;

[0093] In S24, traversing the node links and selecting the dialogue text ensures that the dialogue text conforms to the business process logic. For each state sequence, the node links are traversed in turn, and a corresponding dialogue text is randomly selected from the template library according to the node ID and state type as the customer's response content for that node, thereby generating a complete and well-structured dialogue text.

[0094] S25: Generate initial true labels based on the state sequence, and apply label correction rules to correct the business semantic consistency of the initial true labels to obtain the final true labels;

[0095] S25 corrects the initial labels through label correction rules. These rules, derived from business semantic constraints and manually preset logical rules, ensure that the final true labels are consistent with the business semantics, providing an accurate benchmark for model evaluation. Specifically, during dialogue synthesis, although the node state sequence determines the initial true labels, at certain business-sensitive nodes (such as identity verification, risk control confirmation, etc.), the business requires a clear affirmative or negative result, disallowing ambiguous or noisy expressions. Therefore, the system has a built-in rule matrix, which is predefined by business experts according to the mandatory requirements in the SOP. For example, at the "identity verification" node, if noisy expressions appear, such as "cannot hear clearly," "what did you say," or the customer does not answer directly, the label is forcibly corrected to "negative (N)" according to the business rules because the verification failed.

[0096] Furthermore, the rule matrix can be constructed using an if-then rule table, with node ID, initial label, and dialogue features as conditions, to output corrected labels. These rules can be manually configured before the system runs, or dynamically supplemented during iteration based on bad case analysis.

[0097] In step S24, a perturbation mechanism is introduced when generating multi-turn dialogue text. This perturbation mechanism includes: simulating hanging up behavior by prematurely terminating the dialogue at a certain node with a preset probability; and outputting null values ​​or noise content at some nodes. Introducing this perturbation mechanism can simulate abnormal behaviors such as user hanging up, no response, and noise interference in real dialogues, improving the realism of the synthesized dialogue text and making the generated dialogue closer to actual interaction scenarios. Furthermore, in this embodiment, taking a telemarketing scenario as an example, the probability of randomly hanging up at each node is 70%, and the probability of no response from users at each node is 9%. These probabilities can be adjusted according to the actual dialogue quality of the user group.

[0098] In this embodiment, step S3 includes the following steps:

[0099] S31: Semantic parsing of each node is performed using a prediction model to output the corresponding predicted label;

[0100] In S31, dialogues that are determined to be genuine customer statements are input into a prediction model. The model can be a rule engine, a machine learning classifier, a deep learning model, or a large language model. It is responsible for semantic parsing of each node or the entire dialogue and outputting the corresponding label prediction results.

[0101] S32: Align the predicted labels with the final true labels node by node ID;

[0102] In S32, each node corresponds to three Y / N / I states, which are the three values ​​of the label. Here, the state labels of the predicted label and the node ID are aligned. After node-by-node alignment, it is saved in a structured format. For example, a record contains the fields of sample ID, node ID, true label, and predicted label.

[0103] S33: Calculate the multidimensional performance index of the prediction model based on the alignment results.

[0104] The multidimensional performance metrics include: accuracy, precision, missed rate, recall, and node-level confusion matrix.

[0105] In this embodiment, S3 further includes:

[0106] S34: Based on multi-dimensional performance metrics and node-level confusion matrices, generate automated evaluation reports containing structured tables and visual charts.

[0107] S34 specifically includes:

[0108] S341: Based on the final true labels and the predicted labels, a baseline data table is constructed by aligning the sample ID and node ID. In this embodiment, an example of the baseline data table is given as shown in Table 1 below. If there are two dialogue samples numbered 001 and 002, and each sample has three nodes, namely nodes A, B, and C, the following baseline data table is obtained based on the final true labels and predicted labels:

[0109] Table 1. Interpretation of Baseline Data

[0110]

[0111] S342: Based on the aforementioned baseline data table, count the number of correct and incorrect predictions, calculate the overall accuracy, and break it down by label or scenario dimension to calculate the precision, recall, and miss rate under different labels, with the formulas as follows:

[0112] Accuracy = (TP + TN) / (TP + TN + FP + FN);

[0113] Precision rate = TP / (TP + FP);

[0114] Leakage rate = FN / (TP + FN);

[0115] In the formula, TP represents true positives, TN represents true negatives, FP represents false positives, FN represents false negatives, TP+TN is the total number of correctly predicted samples, TP+TN+FP+FN is the total number of samples, TP+FP is the total number of samples predicted as positive, and TP+FN is the total number of samples actually positive.

[0116] Based on the baseline data in Table 1 above, the accuracy can be calculated as 4 / 6 ≈ 66.67%. With Y as the positive class, the precision is 1 / (1+1) = 0.5 = 50%, and the miss rate is 1 / (1+1) = 50%. With N as the positive class, the precision is 2 / (2+1) = 0.6667 = 66.67%, and the miss rate is 0. With I as the positive class, the precision is 1 / (1+0) = 1 = 100%, and the miss rate is 1 / (1+1) = 50%.

[0117] S343: Construct a confusion matrix by node ID or tag type, as shown in the example. Figure 3 As shown.

[0118] In S343, the misclassification flow among categories Y, N, and I is analyzed, such as the proportion of Y being misclassified as N and the proportion of I being misclassified as Y, so as to describe the model's performance at each node and in each state in a more granular way.

[0119] S344: Output the precision, recall, and miss rates in the form of HTML reports, Excel spreadsheets, or JSON files. The reports may include tables, bar charts, line charts, etc., to facilitate developers to quickly browse and compare the performance of different model versions.

[0120] In this embodiment, S4 includes:

[0121] S41: Filter bad case samples where the final true label and the predicted label are inconsistent from the evaluation results generated based on multi-dimensional performance indicators and node-level confusion matrix;

[0122] In S41, based on the benchmark data table, all records where the true label and the predicted label are inconsistent are identified, and the records, their original dialogues, and node information are aggregated into a badcase set.

[0123] S42: Classify, aggregate, and cluster badcase samples by node ID, label type, and error type to extract common error features.

[0124] In S42, the badcase set is grouped according to dimensions such as node ID, true label, and predicted label. For example, all samples that should be Y but are predicted as N are aggregated, or samples with noisy expressions under a certain node are easily identified as positive, in order to discover the error concentration pattern. For extracting common error features, text and distribution analysis are performed on each type of badcase to identify common triggering reasons, such as specific sentence patterns, ambiguous expressions, long sentences, short sentence features, or misjudgments caused by the context before and after certain nodes, from which directions that can be targeted for correction are extracted.

[0125] S4 further includes:

[0126] S43: Develop optimization plans based on the clustering results of bad case samples, and optimize the prediction model and the script template library accordingly;

[0127] In S43, based on the bad case analysis results, targeted optimization measures are formulated, such as emphasizing the processing rules of a certain type of expression in the prompt words, adding supplementary rules to the rule engine, adjusting the labeling of the speech template, or adding typical bad cases as incremental training data to the model training set.

[0128] S44: Repeat S1-S4 until the model performance reaches the preset convergence criterion or the improvement stops, mark the abnormal nodes and prompt manual intervention analysis.

[0129] S44 specifically includes:

[0130] S441: Based on the optimization scheme, update the model parameters, prompt word templates, rule engine logic or speech template library, and release the new version configuration or model in the system;

[0131] S442: For the same or updated synthetic dataset, perform prediction and evaluation again according to the process of step S3 to obtain a new round of performance indicators, ensuring that the optimization effect can be quantified;

[0132] S443: Compare the evaluation results of each round longitudinally, and plot the change curves of indicators such as accuracy, precision, and missing rate at different iteration rounds to determine whether the overall performance continues to improve and whether there are signs of convergence.

[0133] S444: When certain metrics show limited improvement or even fluctuations after multiple iterations, they can be considered to have reached the performance ceiling under the current configuration. The system marks the relevant nodes or tags as anomalies requiring further manual analysis, prompting R&D or business personnel to make deeper adjustments, thus forming a comprehensive closed loop of automatic optimization and manual decision-making.

[0134] Example 2

[0135] In this embodiment, the closed-loop optimization method of Example 1 is used for the two scenarios, dhhf_O2O and dhhf_entreprise, and all are based on the Qwen2.5 prediction model to obtain three index values: accuracy, missing rate, and precision. Figure 2 As shown, it can be seen that with the improvement of the prompt word version, the accuracy and precision of dhhf_O2O have improved after each optimization, while the missing rate has decreased.

[0136] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating and optimizing SOP-based structured data to drive model evaluation, characterized in that, Includes the following steps: S1: Obtain the Standard Operating Procedure (SOP) of the target scenario, parse it to generate node links, define the state space for each node, and obtain the node state sequence; S2: Build a dialogue template library for each node, generate multi-turn dialogue text based on the dialogue template library and the node state sequence, and generate corresponding real tags; S3: Input the dialogue text into the existing prediction model to obtain the predicted labels of the nodes, compare the predicted labels with the real labels, and calculate the multidimensional performance index of the prediction model. S4: Optimize the prediction model and the script template library based on the multidimensional performance indicators, and repeat steps S1-S4 until the preset conditions are met.

2. The method for generating and optimizing SOP structured data for driving model evaluation according to claim 1, characterized in that, S1 includes: S11: Obtain the standard operating procedure (SOP) for the target scenario from the business side, parse it, and identify and extract the key nodes; S12: Assign a unique node ID to each node and build node links based on the order and dependencies between nodes; S13: Define a state space for each node, and obtain the node state sequence based on the state space.

3. The method for generating and optimizing SOP structured data for driving model evaluation according to claim 2, characterized in that, In S13, the state space includes three states: Y state representing affirmative expression, N state representing negative expression, and I state representing ambiguous, unclear, or noisy expression. The node state sequence is formed by enumerating the states of all nodes.

4. The method for generating and optimizing SOP structured data for driving model evaluation according to claim 1, characterized in that, The multi-turn dialogue text generated in S2 includes: S21: Collect historical dialogues and manual templates by node and state, expand diverse dialogues using existing generation models, and build a globally reusable noisy dialogue library for state I. S22: Perform text standardization processing on the historical scripts, manual templates, and the diverse scripts, and perform deduplication and merging; S23: The script templates for the three states are stored in a structured form of node ID + state type; S24: Based on the node state sequence, traverse the node links, and randomly select dialogue text from the structured storage according to the state of each node to generate multi-turn dialogue text; S25: Generate initial true labels based on the state sequence, and apply label correction rules to correct the business semantic consistency of the initial true labels to obtain the final true labels.

5. The method for generating and optimizing SOP structured data for driving model evaluation according to claim 4, characterized in that, In S24, a perturbation mechanism is introduced when generating multi-turn dialogue text. The perturbation mechanism includes: terminating the dialogue prematurely at a certain node with a preset probability to simulate hanging up behavior; and outputting null values ​​or noise content at some nodes.

6. The method for generating and optimizing SOP structured data for driving model evaluation according to claim 1, characterized in that, S3 includes the following steps: S31: Semantic parsing of each node is performed using a prediction model to output the corresponding predicted label; S32: Align the predicted labels with the final true labels node by node ID; S33: Calculate the multidimensional performance index of the prediction model based on the alignment results.

7. The method for generating and optimizing SOP structured data for driving model evaluation according to claim 6, characterized in that, The multidimensional performance metrics include: accuracy, precision, missed rate, recall, and node-level confusion matrix.

8. The method for generating and optimizing SOP structured data for driving model evaluation according to claim 6, characterized in that, S3 further includes: S34: Based on multi-dimensional performance metrics and node-level confusion matrices, generate automated evaluation reports containing structured tables and visual charts.

9. The method for generating and optimizing SOP structured data for driving model evaluation according to claim 1, characterized in that, S4 includes: S41: Filter bad case samples where the final true label and the predicted label are inconsistent from the evaluation results generated based on multi-dimensional performance indicators and node-level confusion matrix; S42: Classify, aggregate, and cluster badcase samples by node ID, label type, and error type to extract common error features.

10. The method according to claim 9, characterized in that, S4 further includes: S43: Develop optimization plans based on the clustering results of bad case samples, and optimize the prediction model and the script template library accordingly; S44: Repeat S1-S4 until the model performance reaches the preset convergence criterion or the improvement stops, mark the abnormal nodes and prompt manual intervention analysis.

Images