Customer service dialogue multi-dimensional automatic scoring system based on large language model

Abstract

The invention discloses a customer service dialogue multi-dimensional automatic scoring system based on a large language model, and the system comprises the following modules: a dialogue processing module which carries out the voice recognition of a customer service voice dialogue, carries out the speaker separation and text standardization based on voice embedding, and generates a dialogue object with a knowledge base version identifier; the evidence retrieval module is used for limiting a knowledge base version and retrieving an evidence set related to the session intention based on a semantic matching model; the dynamic reference generation module is used for calling a large language model to generate a dynamic reference answer conforming to preset structuralization; the state diagram construction and structured scoring module is used for generating actual and ideal session state diagrams and calculating structured scores; the referee scoring module is used for outputting a referee scoring result set by utilizing a plurality of large language model referee instances; and the consistency calculation and score fusion module is used for calculating a consistency index and outputting a structured score result by adopting a self-adaptive score fusion strategy. According to the method, the consistency, the interpretability and the robustness of automatic scoring of the customer service session are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and intelligent customer service systems, and particularly relates to a customer service dialogue multi-dimensional automatic scoring system based on a large language model. BACKGROUND

[0002] In the existing customer service business scenario, customer service dialogue quality inspection mainly relies on manual scoring or rule-based automatic scoring methods. Although manual scoring can comprehensively judge the business process, compliance of the language, and the effect of processing the customer's emotions, it has problems such as low scoring efficiency, strong subjectivity, and high cost. Although the rule-based automatic scoring method can be batch processed, it relies on pre-set keywords or regular rules, and it is difficult to accurately identify complex semantic expressions and variable business processes, and it lacks understanding of unstructured conversation content, resulting in incomplete scoring dimension coverage and low accuracy.

[0003] In recent years, large language models (LLM) have shown high generalization ability in natural language understanding and generation tasks, and have been introduced into the field of customer service dialogue analysis and scoring. However, existing scoring schemes based on large language models usually only rely on a single model to directly generate scoring results, lack multi-judge mechanisms and structured scoring processes, resulting in insufficient explainability of scoring results and unstable consistency. At the same time, the use of deep interaction features such as conversation process structure and inter-node dependency is insufficient, making the scoring unable to fully reflect the completeness of the business process and the accuracy of knowledge reference.

[0004] Therefore, how to provide a customer service dialogue multi-dimensional automatic scoring system based on a large language model is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] One object of the present application is to provide a customer service dialogue multi-dimensional automatic scoring system based on a large language model. The present application provides a customer service dialogue multi-dimensional automatic scoring system based on a large language model, which realizes high-precision automatic scoring of customer service dialogue in the dimensions of business process integrity, knowledge reference accuracy, compliance expression standardization, and emotional disposition effectiveness by introducing state graph analysis, node matching and path consistency calculation, multi-judge scoring mechanism, and adaptive scoring fusion strategy.

[0006] The customer service dialogue multi-dimensional automatic scoring system based on a large language model according to the embodiment of the present application comprises the following modules:

[0007] A conversation processing module for receiving customer service voice conversation data, performing speech recognition, speaker separation based on speech embedding, and text standardization processing, and generating a conversation object with a knowledge base version identifier;

[0008] an evidence retrieval module configured to retrieve, based on the conversation object, a set of evidences related to the conversation intent from a knowledge base, and invoke a large language model to generate a dynamic reference answer conforming to a preset structured pattern;

[0009] a state graph construction module configured to parse the conversation object into an actual conversation state graph constituted by a plurality of key event nodes and time and dependency relationships, and parse the dynamic reference answer into an ideal conversation state graph;

[0010] a structured scoring module configured to calculate a structural similarity of the actual conversation state graph and the ideal conversation state graph based on node matching and path consistency, and calculate a structured score in combination with a node attribute difference;

[0011] a judge scoring module configured to input the conversation object, the set of evidences and the dynamic reference answer into a plurality of independently configured large language model judge instances respectively, and output a set of judge scoring results;

[0012] a consistency calculation and scoring fusion module configured to calculate a consistency index based on the set of judge scoring results, weight the aggregated score and the structured score according to a weight coefficient corresponding to the consistency index, and output a final score of each scoring dimension through a regression model based on multi-dimensional features, and generate a structured scoring result through scoring calibration.

[0013] Further, the modules are implemented through the following methods:

[0014] receiving customer service voice conversation data, performing voice recognition, speaker separation and text standardization processing, and generating a conversation object with a knowledge base version identifier;

[0015] retrieving, based on the conversation object, a set of evidences related to the conversation intent from a knowledge base, and invoking a large language model to generate a dynamic reference answer conforming to a preset structured pattern;

[0016] parsing the conversation object into an actual conversation state graph constituted by a plurality of key event nodes and time and dependency relationships, and parsing the dynamic reference answer into an ideal conversation state graph;

[0017] calculating a structural similarity of the actual conversation state graph and the ideal conversation state graph based on node matching and path consistency, and calculating a structured score in combination with a node attribute difference;

[0018] inputting the conversation object, the set of evidences and the dynamic reference answer into a plurality of independently configured large language model judge instances respectively, and outputting a set of judge scoring results;

[0019] calculating a consistency index based on the set of judge scoring results, triggering an adaptive scoring fusion strategy when the consistency index is lower than a threshold range, and aggregating scores according to a consistency weight when the consistency index is higher than the threshold range, to obtain a structured scoring result.

[0020] Furthermore, generating a session object with a knowledge base version identifier includes: labeling the transcribed text according to agent and user roles using speaker separation technology based on speech embedding, and selecting the version number closest to and no later than the end time from the effective version record of the knowledge base based on the session end time as the knowledge base version identifier.

[0021] Furthermore, the retrieval of evidence set related to the session intent includes: limiting the retrieval scope based on the knowledge base version identifier in the session object, and sorting the knowledge base by similarity based on the intent vector generated by the semantic matching model, and selecting knowledge fragments with similarity higher than a preset threshold as the evidence set.

[0022] Furthermore, the steps for generating the dynamic reference answer include:

[0023] Perform semantic parsing on the session object to generate a session summary;

[0024] Extract key points from the evidence set to form a knowledge point set;

[0025] Based on the correspondence between conversation summaries and knowledge point sets under each scoring dimension, a structured output containing scoring dimension fields, knowledge point coverage fields, and evidence citation fields is generated through a large language model.

[0026] The output results are then mapped and their integrity is verified according to a structured model to obtain a dynamic reference answer.

[0027] Furthermore, the steps for constructing the actual session state diagram and the ideal session state diagram include:

[0028] Perform intent recognition and slot filling on round texts arranged in chronological order to generate a set of event nodes containing event type, occurrence time, speaker role, slot key value and evidence citation identifier;

[0029] Establish time sequence edges based on adjacent time relationships and record time intervals; establish dependency relationship edges based on the pre- and post-constraints of mandatory actions and record trigger conditions; time intervals are in the range of milliseconds to minutes.

[0030] Merge nodes of the same type that have the same content and are adjacent in time; fold nodes that do not carry slot key values; mark processes that are repeated and do not meet the closed-loop conditions by loop counting, with the loop count being a non-negative integer.

[0031] Event nodes include at least process nodes, emotion handling nodes, knowledge citation nodes, and compliance statement nodes. Process nodes are used to represent requirement confirmation, solution provision, and closed-loop actions. Emotion handling nodes are used to represent apologies, reiteration, and reassurance actions. Knowledge citation nodes are used to represent citations of knowledge base entries with source identification. Compliance statement nodes are used to represent pricing standards, compensation conditions, and prohibited language statements.

[0032] Map the mandatory points in the dynamic reference answer to an ordered sequence of process nodes, map the policy references in the dynamic reference answer to a set of compliance statement nodes, map the closed-loop conditions in the dynamic reference answer to termination nodes, and establish dependency edges in the ideal session state diagram according to the pre- and post-relationships between the points.

[0033] Write the knowledge base version identifier into both the actual session state diagram and the ideal session state diagram to ensure that the source entry of the knowledge reference node is consistent with the knowledge base version identifier.

[0034] Write alignment anchors for nodes in both state diagrams. The alignment anchors include entity identifiers and slot key-value pairs. The entity identifiers cover the order number, amount, and time fields. The amount is represented in a uniform currency unit and the numerical precision is in the range of two to four decimal places. The time is represented in a uniform time standard and the alignment time window is in the range of seconds to minutes.

[0035] When a slot is missing, the corresponding node is marked as incomplete and retained.

[0036] Furthermore, the calculation of the consistency index based on the actual session state diagram and the ideal session state diagram includes:

[0037] Based on the alignment anchor point of the node, node matching and missing detection are performed. Required nodes that cannot be matched are recorded as missing nodes. For matched nodes, attribute consistency scores are calculated. The attribute consistency scores are calculated based on the matching degree of slot key-value pairs. Slot matching degree is recorded as consistent when the string similarity is greater than a preset threshold.

[0038] Based on the temporal order edges and dependency edges between matching nodes, an aligned path set is constructed. The consistency score of each path is calculated in terms of node order, node type, and edge attributes. The path consistency score is then obtained by weighting the paths according to their coverage.

[0039] For recurring non-closed-loop processes, a penalty weight is assigned based on the loop count during the consistency calculation, and the penalty weight is located in the range (0,1).

[0040] The structural similarity is calculated by combining the node consistency score and the path consistency score according to a preset ratio. The structural similarity reflects the degree of similarity between the two state graphs in the overall topology.

[0041] In the structural similarity calculation, a node importance weight is introduced. The node importance is based on a preset node type, wherein the weight of process nodes and compliance declaration nodes is higher than that of greeting nodes and emotion handling nodes.

[0042] The consistency index is calculated by combining the node consistency score, path consistency score, and structural similarity using a fusion formula. The consistency index is located in the interval [0,1].

[0043] Furthermore, the steps for generating the set of referee scoring results include:

[0044] The number of the judged instances is between three and seven, and there are differences in at least one of the random initialization parameters, decoding temperature and sampling strategy, prompt word template and evidence window interception strategy to form statistical independence;

[0045] Each judge instance outputs a judge scoring record according to a preset structured pattern. The judge scoring record includes at least the initial score for each scoring dimension, a reference answer key point coverage table, a set of cited evidence identifiers, an index of reasoning fragments, evidence coverage rate, and judge uncertainty index.

[0046] The judge scoring record is subjected to format integrity verification and evidence validity verification. The evidence validity verification requires that the cited evidence falls within the evidence set and that the knowledge base version identifier is consistent with the knowledge base version identifier of the session object.

[0047] The verified referee scoring records are organized into a set of referee scoring results according to the referee instance identifier and the scoring dimension identifier, which serves as the input for the calculation of consistency indicators and subsequent scoring fusion.

[0048] Furthermore, the step of generating the final score includes:

[0049] When the consistency index is lower than the preset threshold, an adaptive scoring fusion strategy is triggered, which dynamically adjusts the proportion of the aggregated score based on the structural similarity and node type weights.

[0050] When the consistency index is higher than the preset threshold, the aggregation score and the structured score are weighted according to the weight coefficient corresponding to the consistency index.

[0051] The fusion control model is a regression model based on multidimensional features. The inputs include the original values ​​of consistency index, structural similarity, node consistency score, path consistency score, and aggregation score and structured score. The output of the regression model is the predicted score and corresponding confidence level of each scoring dimension.

[0052] The scoring dimensions should include at least the completeness of business processes, the accuracy of knowledge citations, the standardization of compliant expression, and the effectiveness of emotional handling.

[0053] The predicted scores are processed by a calibration model to eliminate scaling bias between different dimensions. The calibration model is trained based on historical scoring results and the distribution of standard answers.

[0054] The final output is a structured scoring result, which includes the final score for each scoring dimension, the overall confidence level, the set of cited evidence, and the risk label.

[0055] Furthermore, the adaptive scoring fusion strategy includes:

[0056] When the consistency index is lower than the preset threshold, the aggregation score adjustment coefficient is dynamically calculated based on the structural similarity and node type weight. The adjustment coefficient is located in the (0,1) interval, and the proportion of aggregation score is reduced and the proportion of structured score in the dimension of node type is increased according to the adjustment coefficient.

[0057] The node type weights are set in the order of process node > compliance statement node > knowledge reference node > emotion handling node, and the difference between adjacent weights is in the range of 0.05 to 0.2.

[0058] When the consistency metric is higher than the threshold, the original aggregation score percentage is maintained and a fusion calculation weighted by the consistency metric is performed.

[0059] The beneficial effects of this invention are:

[0060] This invention parses customer service conversations into actual conversation state diagrams and performs node matching and path consistency calculations with dynamically generated ideal conversation state diagrams. This enables interpretable scoring based on structured features, which can not only quantify the completeness of business processes and the accuracy of knowledge references, but also locate specific issues such as missing nodes, incomplete processes, and version inconsistencies. In conjunction with a multi-judge mechanism that introduces multiple independently configured large language model judge instances, the consistency index is calculated using multi-source scoring results, effectively reducing single-model bias and improving the stability and reliability of scoring results.

[0061] Regarding score fusion, this invention designs an adaptive score fusion strategy. When consistency is low, the weight of structured scores is dynamically increased; when consistency is high, scores are aggregated according to consistency weights. A fusion control model and a calibration model are used to generate multi-dimensional final scores and comprehensive confidence levels. The scoring dimensions cover business process completeness, accuracy of knowledge citations, compliance of expression, and effectiveness of emotional handling, comprehensively reflecting the quality of customer service dialogue. Furthermore, each module utilizes existing feasible technologies combined with the invention's unique structured analysis and fusion strategy, demonstrating strong practicality and promotional value. Attached Figure Description

[0062] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0063] Fig. 1 This is a schematic diagram of the structure of the customer service dialogue multi-dimensional automatic scoring system based on a large language model proposed in this invention;

[0064] Fig. 2 The flowchart is a multi-dimensional automatic scoring system for customer service dialogue based on a large language model, as proposed in this invention.

[0065] Fig. 3 This is a flowchart illustrating the consistency index and adaptive fusion calculation process of the multi-dimensional automatic scoring system for customer service dialogue based on a large language model proposed in this invention. Detailed Implementation

[0066] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0067] refer to Figs. 1-3 A multi-dimensional automatic scoring system for customer service dialogues based on a large language model includes the following modules:

[0068] The conversation processing module receives customer service voice conversation data, performs voice activity detection and noise reduction on the original audio, extracts acoustic features, and calls an end-to-end speech recognition model to generate timestamped transcribed text. Speaker vectors are extracted using speaker separation technology based on speech embedding. The transcribed text is labeled according to agent and user roles, and the text undergoes numerical normalization, proper noun standardization, and colloquial phrase replacement to form a standardized text sequence. Based on the conversation end time, the system selects the version number closest to and no later than the end time from the knowledge base version records and appends it as a knowledge base version identifier to the conversation object, thus generating a conversation object with a knowledge base version identifier, which serves as input for subsequent processing.

[0069] The evidence retrieval module, based on the session object, first extracts the session intent vector through a semantic matching model and identifies the relevant business type, core issues, and key entities. During the retrieval phase, the system limits the knowledge base query to entries within the version number range bound to the session object. By calculating the semantic similarity between the intent vector and the knowledge base entries, knowledge fragments with similarity exceeding a preset threshold are selected as the evidence set. The system extracts key points from the evidence set and generates a session summary in conjunction with the session object. Using the session summary and the set of key knowledge points as input, a large language model is invoked to generate a dynamic reference answer conforming to a preset structured pattern. This answer includes scoring dimension fields, key point coverage fields, and evidence citation fields, and undergoes field mapping and integrity verification to ensure its usability for subsequent comparisons.

[0070] After receiving the session object and dynamic reference answer, the state diagram construction module parses the session object into an actual session state diagram and the dynamic reference answer into an ideal session state diagram. During the construction of the actual session state diagram, the system performs intent recognition and slot filling on the session round text in chronological order, generating a set of event nodes containing event type, occurrence time, speaker role, slot key value, and evidence citation identifier. It also establishes time sequence edges based on adjacent time relationships and dependency relationship edges based on business pre- and post-constraints. It merges nodes of the same type with consistent content and adjacent time, collapses greeting nodes that do not carry key slots, and marks loop counts for processes that do not meet the closed-loop conditions. The construction process of the ideal session state diagram maps the mandatory points in the dynamic reference answer into an ordered sequence of process nodes, maps policy citations into a set of compliance statement nodes, maps closed-loop conditions into termination nodes, and establishes dependency relationship edges based on the pre- and post-constraint relationships between points. Both state diagrams are written with the same knowledge base version identifier, and alignment anchors including entity identifiers and slot key value pairs are written for the nodes used for alignment to ensure the accuracy of the subsequent matching process.

[0071] The structured scoring module, based on the alignment results of the actual session state graph and the ideal session state graph, first performs node matching and missing node detection. Unmatched mandatory nodes are marked as missing nodes. For matched nodes, an attribute consistency score is calculated based on the matching degree of slot key-value pairs. When the string similarity is greater than a preset threshold, it is marked as consistent. In the path consistency calculation, the system constructs an aligned path set based on the temporal order edges and dependency edges between matched nodes. It calculates the consistency score of each path in terms of node order, node type, and edge attributes, and obtains the path consistency score by weighting the path coverage. Combining the node consistency score and the path consistency score, and taking into account the importance weight of the node type, the system calculates the structural similarity, which reflects the degree of similarity between the two state graphs in the overall topology, and maps it to a structured score.

[0072] During the scoring process, the judge scoring module inputs the conversation object, evidence set, and dynamic reference answer into multiple independently configured large language model judge instances. These judge instances differ in at least one of the following: random initialization parameters, decoding temperature, sampling strategy, prompt word template, and evidence window truncation strategy, to form statistical independence. Each judge instance outputs judge scoring records according to a unified structured pattern. The records include at least the initial scores for each scoring dimension, the reference answer key point coverage table, the set of cited evidence identifiers, the index of reasoning fragments, evidence coverage rate, and judge uncertainty index. The system performs format integrity checks and evidence validity checks on these scoring records. Records that pass the checks are organized into a judge scoring result set according to the judge instance identifier and the scoring dimension identifier for consistency calculation.

[0073] The consistency calculation and scoring fusion module calculates a consistency index for the set of judges' scoring results. When the consistency index is lower than a preset threshold, an adaptive scoring fusion strategy is triggered, dynamically adjusting the proportion of the aggregated score based on structural similarity and node type weights. When the consistency index is higher than the threshold, the aggregated score and the structured score are weighted according to the weight coefficients corresponding to the consistency index. Subsequently, the system inputs the original values ​​of the consistency index, structural similarity, node consistency score, path consistency score, and aggregated score and structured score into a regression model based on multidimensional features, outputting the predicted scores and corresponding confidence levels for each scoring dimension. After model calibration to eliminate scale bias between different dimensions, the system outputs a structured scoring result containing the final scores for each scoring dimension, comprehensive confidence level, set of cited evidence, and risk labels, realizing fully automated multidimensional quality assessment of customer service dialogues.

[0074] In this embodiment, the modules are interconnected using the following method:

[0075] Receive customer service voice conversation data, perform speech recognition, speaker separation and text standardization processing, and generate conversation objects with knowledge base version identifiers;

[0076] When performing speech recognition, speaker segmentation, and text normalization, the speech recognition process employs an end-to-end automatic speech recognition model based on the Transformer architecture. This model takes a 16kHz sampling rate, 16-bit bit depth PCM speech signal as input. After Mel-frequency cepstral coefficient (MFCC) feature extraction, the signal enters a network consisting of a multi-layer self-attention encoder and decoder, outputting the transcribed text and its corresponding timestamp. Speaker segmentation utilizes an ECAPA-TDNN-structured speech embedding extraction network, converting each speech segment into a 192-dimensional speaker vector. Cosine similarity is used to calculate the similarity matrix between different vectors, and spectral clustering is used to classify speech segments into two categories: agents and users. Text normalization includes: converting all numbers in the text to Arabic numerals, retaining two decimal places; converting currency amounts to RMB units, retaining precision to the centimeters; converting all time expressions to Coordinated Universal Time (UTC) format; replacing colloquial phrases with their corresponding standard written expressions; and adding periods or question marks to the end of sentences lacking punctuation.

[0077] Based on the conversation object, retrieve a set of evidence related to the conversation intent from the knowledge base, and call the large language model to generate a dynamic reference answer that conforms to the preset structured pattern;

[0078] The session object is parsed into an actual session state diagram consisting of multiple key event nodes, time, and dependencies, and the dynamic reference answer is parsed into an ideal session state diagram.

[0079] The structural similarity between the actual session state graph and the ideal session state graph is calculated based on node matching and path consistency, and a structured score is calculated by combining the differences in node attributes.

[0080] Input the conversation object, evidence set, and dynamic reference answer into multiple independently configured large language model judge instances, and output a set of judge scoring results;

[0081] A consistency index is calculated based on the set of referee scores. When the consistency index is below a threshold range, an adaptive score fusion strategy is triggered. When the consistency index is above the threshold range, the scores are aggregated according to the consistency weight to obtain a structured score result.

[0082] In this embodiment, generating a session object with a knowledge base version identifier includes:

[0083] The transcribed text, timestamps of each round of speaking, role tags, file paths of corresponding audio segments, unique session identifiers, session start time, session end time, audio sampling rate, transcription confidence scores, and language identifiers are combined into a JSON-structured session data unit. The version number with the most recent session end time and no later than that time is retrieved from the knowledge base version control table and written into the "kb_version" field of the session data unit. Finally, the JSON object containing all the above fields is saved as a session object for subsequent evidence retrieval and dynamic reference answer generation.

[0084] In this embodiment, the retrieval of the evidence set related to the session intent includes:

[0085] All spoken texts from the conversation object are merged into a conversation text sequence. The [CLS] position vector in the last layer of the BERT-Base Chinese pre-trained model is used as the semantic feature representation. This vector is then input into a normalization layer as the conversation intent vector. The retrieval scope is limited to all knowledge entries under the knowledge base version number corresponding to the conversation object. For each knowledge entry, the same BERT encoding method is used to generate an entry vector. The cosine similarity between the conversation intent vector and the entry vector is calculated. The similarity scores are sorted from high to low, and the top N knowledge entries with similarity greater than 0.85 are selected as the evidence set, where N ranges from 3 to 10.

[0086] When calling a large language model to generate dynamic reference answers that conform to a preset structured pattern, a multi-turn dialogue fine-tuning method is used for training to support multiple inputs and structured outputs. The input format includes three fields: conversation text field, evidence set field, and task instruction field. The task instruction explicitly requires the model to output a JSON object containing a scoring dimension field, a key point coverage field, and an evidence citation field. The output results are parsed using regular expressions and the field integrity is checked to ensure that all field values ​​exist and the format conforms to the preset pattern. If a field value is missing, a rollback mechanism is triggered to regenerate the answer, ultimately obtaining a dynamic reference answer that conforms to the structured pattern.

[0087] In this embodiment, the steps for constructing the actual session state diagram and the ideal session state diagram include:

[0088] In this implementation, intent recognition and slot filling are performed on the sequentially arranged round text. The sequentially arranged round text, labeled with speaker roles, is processed through Chinese word segmentation and pre-trained word vector encoding. This is then input into an intent recognition model composed of a Chinese BERT encoding layer, a bidirectional long short-term memory network layer, and a Softmax classification layer, which outputs the intent category. The same input is then fed into a sequence labeling model composed of BERT and a conditional random field (CRF). The BIO labeling method is used to identify and extract slot key values ​​such as order number, amount, time, and product name. The amount is uniformly converted to RMB and retained to two to four decimal places, and the time is converted to a UTC timestamp with its precision recorded. Finally, the intent category, slot key value, occurrence time, speaker role, and evidence citation identifier are combined to generate event nodes, which serve as the basic data for constructing the subsequent actual conversation state diagram. This generates a set of event nodes containing event type, occurrence time, speaker role, slot key value, and evidence citation identifier.

[0089] Establish time sequence edges based on adjacent time relationships and record time intervals; establish dependency relationship edges based on the pre- and post-constraints of mandatory actions and record trigger conditions; time intervals are in the range of milliseconds to minutes.

[0090] Merge nodes of the same type that have the same content and are adjacent in time; fold nodes that do not carry slot key values; mark processes that are repeated and do not meet the closed-loop conditions by loop counting, with the loop count being a non-negative integer.

[0091] Event nodes include at least process nodes, emotion handling nodes, knowledge citation nodes, and compliance statement nodes. Process nodes are used to represent requirement confirmation, solution provision, and closed-loop actions. Emotion handling nodes are used to represent apologies, reiteration, and reassurance actions. Knowledge citation nodes are used to represent citations of knowledge base entries with source identification. Compliance statement nodes are used to represent pricing standards, compensation conditions, and prohibited language statements.

[0092] Map the mandatory points in the dynamic reference answer to an ordered sequence of process nodes, map the policy references in the dynamic reference answer to a set of compliance statement nodes, map the closed-loop conditions in the dynamic reference answer to termination nodes, and establish dependency edges in the ideal session state diagram according to the pre- and post-relationships between the points.

[0093] Write the knowledge base version identifier into both the actual session state diagram and the ideal session state diagram to ensure that the source entry of the knowledge reference node is consistent with the knowledge base version identifier.

[0094] Write alignment anchors for nodes in both state diagrams. The alignment anchors include entity identifiers and slot key-value pairs. The entity identifiers cover the order number, amount, and time fields. The amount is represented in a uniform currency unit and the numerical precision is in the range of two to four decimal places. The time is represented in a uniform time standard and the alignment time window is in the range of seconds to minutes.

[0095] When a slot is missing, the corresponding node is marked as incomplete and retained.

[0096] In this embodiment, the calculation of the consistency index based on the actual session state diagram and the ideal session state diagram includes:

[0097] Node matching and missing node detection are performed based on the alignment anchor points of the nodes. The alignment anchor points consist of entity identifiers (order number, amount, time, etc.) and slot key-value pairs. Required nodes that cannot be matched are marked as missing nodes. For matched nodes, an attribute consistency score is calculated. The attribute consistency score is determined based on the matching degree of the slot key-value pairs, and the matching degree is calculated using a cosine similarity algorithm.

[0098] ;

[0099] in, This represents the total number of dimensions of the feature vector. This indicates the first node to be matched from the actual session state graph. Each feature component This represents the first node from the corresponding node in the ideal session state graph. Each feature component.

[0100] When the similarity is greater than or equal to the preset threshold (ranging from 0.85 to 0.95), it is considered consistent; otherwise, it is considered inconsistent.

[0101] Based on the temporal order edges and dependency edges between matching nodes, a set of aligned paths is constructed. For each aligned path, the consistency of node order, node type, and edge attributes are calculated respectively:

[0102] Node order consistency is determined by the ratio of the length of the longest common subsequence (LCS) to the total number of nodes in the ideal path.

[0103] The node type consistency ratio is calculated using an exact matching method.

[0104] Edge attribute consistency is calculated based on the time interval difference and the matching rate of the triggering conditions, where the tolerance range of the time interval difference is ±5 seconds to ±60 seconds.

[0105] Weight the three consistency items The consistency score of a single path is obtained by weighted summation (ranging from 0 to 1 and summing to 1); then, the overall path consistency score is obtained by weighted average of path coverage (matching path length / ideal path length).

[0106] For processes that are repeated and do not meet the closed-loop condition, count by loop. Assigning penalty weights ,in The value ranges from (0,1] and is multiplied by this weight in the consistency score calculation.

[0107] Overall node consistency score Path consistency score proportional ( The value ranges from 0.4 to 0.6 (obtained from historical data regression analysis) for calculating structural similarity. Structural similarity reflects the degree of similarity between the two state diagrams in terms of their overall topological structure.

[0108] Introducing node importance weights into structural similarity calculations Examples of node type weights are as follows: process node 1.0, compliance statement node 0.9, knowledge citation node 0.8, emotion handling node 0.6, and small talk node 0.4. Finally, the node consistency score, path consistency score, and structural similarity are fused using the following formula:

[0109] ;

[0110] The consistency index was calculated. ,in The value ranges from 0.2 to 0.5, and The consistency index ranges from [0,1].

[0111] In this embodiment, the steps for generating the set of referee scoring results include:

[0112] The number of the judged instances is between three and seven, and there are differences in at least one of the random initialization parameters, decoding temperature and sampling strategy, prompt word template and evidence window interception strategy to form statistical independence;

[0113] Each judge instance outputs a judge scoring record according to a preset structured pattern. The judge scoring record includes at least the initial score for each scoring dimension, a reference answer key point coverage table, a set of cited evidence identifiers, an index of reasoning fragments, evidence coverage rate, and judge uncertainty index.

[0114] The judge scoring record is subjected to format integrity verification and evidence validity verification. The evidence validity verification requires that the cited evidence falls within the evidence set and that the knowledge base version identifier is consistent with the knowledge base version identifier of the session object.

[0115] The verified referee scoring records are organized into a set of referee scoring results according to the referee instance identifier and the scoring dimension identifier, which serves as the input for the calculation of consistency indicators and subsequent scoring fusion.

[0116] In this embodiment, the steps for generating the set of referee scoring results include:

[0117] Configure multiple independent large language model referee instances, with the number ranging from 3 to 7. To ensure statistical independence, at least one of the following parameters should be differentiated in each referee instance:

[0118] Random initialization parameters: Generate weight perturbation vectors using different random seeds, with the perturbation amplitude ranging from 0.1% to 1% of the absolute value of the original weights;

[0119] Decoding temperature: The value ranges from 0.5 to 1.0, and is randomly selected according to a uniform distribution;

[0120] Sampling strategies include nucleus sampling (p-value range 0.8–0.95) and top-k sampling (k-value range 20–50).

[0121] Prompt word templates: There are differences in command terminology, field order, or sample text.

[0122] Evidence window truncation strategies include three methods: fixed window, sliding window, and truncation based on semantic fragment boundaries.

[0123] Each referee instance outputs referee scoring records according to a preset structured pattern. The structured pattern is a fixed JSON format and includes at least the following fields:

[0124] score_initial (initial scores for each rating dimension, ranging from 0 to 100);

[0125] coverage_table (refer to the answer key coverage table, in two-dimensional Boolean matrix form);

[0126] evidence_ids (a set of reference evidence identifiers, in the form of a list of strings);

[0127] reason_snippets(indices of reason snippets, in the form of an integer list);

[0128] evidence_coverage (evidence coverage rate, range 0 to 1);

[0129] uncertainty (referee uncertainty index, range 0 to 1).

[0130] The completeness of the format and the validity of the evidence in the aforementioned referee scoring records were verified.

[0131] Format integrity verification includes checking whether the number of fields, field names, and data types are consistent with the structured schema.

[0132] Evidence validity verification includes checking whether all identifiers in evidence_ids exist in the evidence set, and verifying whether the knowledge base version identifier of each piece of evidence is consistent with the version identifier of the session object. Version comparison uses an exact string matching method.

[0133] The verified judge scoring records are organized into a two-dimensional index table (score_matrix) based on the judge instance identifier (judge_id) and the scoring dimension identifier (dimension_id). Each cell stores the initial score of the corresponding dimension and its associated coverage table, evidence identifier set, uncertainty and other additional information, forming a set of judge scoring results, which serves as the input for consistency index calculation and subsequent scoring fusion.

[0134] In this embodiment, the step of generating the final score includes:

[0135] When the consistency index is lower than the preset threshold, an adaptive scoring fusion strategy is triggered, which dynamically adjusts the proportion of the aggregated score based on the structural similarity and node type weights.

[0136] When the consistency index is higher than the preset threshold, the aggregation score and the structured score are weighted according to the weight coefficient corresponding to the consistency index.

[0137] The fusion control model is a regression model based on multidimensional features. The inputs include the original values ​​of consistency index, structural similarity, node consistency score, path consistency score, and aggregation score and structured score. The output of the regression model is the predicted score and corresponding confidence level of each scoring dimension.

[0138] The scoring dimensions should include at least the completeness of business processes, the accuracy of knowledge citations, the standardization of compliant expression, and the effectiveness of emotional handling.

[0139] The predicted scores are processed by a calibration model to eliminate scaling bias between different dimensions. The calibration model is trained based on historical scoring results and the distribution of standard answers.

[0140] The final output is a structured scoring result, which includes the final score for each scoring dimension, the overall confidence level, the set of cited evidence, and the risk label.

[0141] In this embodiment, the step of generating the final score includes:

[0142] Determine the fusion strategy based on consistency metrics:

[0143] When the consistency index falls below a preset threshold (threshold range 0.75–0.85), an adaptive scoring fusion strategy is triggered, dynamically adjusting the proportion of aggregated scores using the following formula:

[0144] ;

[0145] in, The combined score obtained after fusion; The aggregate score is obtained by aggregating the output scores of multiple large language model judge instances, ranging from 0 to 100. The structured score is calculated from the state diagram similarity, ranging from 0 to 100; The weighting coefficient for the aggregate score ranges from 0 to 1, and its calculation formula is as follows:

[0146] ;

[0147] in, The structural similarity score ranges from 0 to 1 and is calculated by fusing the node consistency score and the path consistency score. The average weight of node types ranges from 0.4 to 1.0. It is the average of the preset weights corresponding to the node types (such as process nodes and compliance declaration nodes) in the actual session state diagram, with a weight range of 0.4 to 1.0.

[0148] When the consistency index exceeds a preset threshold, the aggregation score and the structured score are weighted according to the weighting coefficient corresponding to the consistency index. It is proportional to the consistency index, ranging from 0.5 to 1.0.

[0149] The fusion results, along with other features, are input into the fusion control model. This fusion control model is a gradient boosting regression tree (GBRT)-based regression model. Its input features include: consistency index, structural similarity, node consistency score, path consistency score, raw aggregate score, and raw structured score. Training data comes from manually annotated customer service dialogue rating results, and the target output is the standard score for each rating dimension. Model training employs 5-fold cross-validation and is optimized using mean squared error (MSE) as the loss function.

[0150] The regression model outputs predicted scores and corresponding confidence levels for each rating dimension. The rating dimensions must include at least:

[0151] Business process integrity (complete process node coverage);

[0152] Knowledge citation accuracy (number of correct citations / total number of citations);

[0153] Compliance and standardization of expression (ratio of the number of violations to the total number of expressions);

[0154] Effectiveness of emotional management (occurrence rate and temporal rationality of emotional relief points).

[0155] Finally, the predicted scores are input into the calibration model for scaling normalization. The calibration model uses the PlattScaling method, trained based on historical scoring results and the distribution of standard answers. It can map scores of each dimension to a uniform range of 0-100, and output the final comprehensive confidence score, the set of cited evidence, and the risk label, forming a complete structured scoring result.

[0156] Example 1:

[0157] To verify the feasibility of this invention in practice, it was applied to real production data from an e-commerce customer service center:

[0158] One thousand Chinese voice customer service calls were collected over a consecutive week (average duration 9.8 minutes; 380 calls for returns / exchanges, 320 calls for billing / invoices, and 300 calls for logistics / receipt). The average score given by three senior quality inspectors was used as the "standard answer." The knowledge base uses a snapshot from August 10, 2025 (denoted as v2025.08.10). The system is deployed on a local inference cluster: ASR uses an end-to-end model based on Transformer (16kHz, word-level timestamps), speaker separation uses ECAPA-TDNN speech embedding + spectral clustering (cosine similarity), and text normalization includes unifying numbers and amounts (RMB, rounded to two decimal places) and time normalization (UTC).

[0159] The retrieval side uses the BERT-Base Chinese model to extract [CLS] vectors for semantic matching, limited to the v2025.08.10 snapshot, with a threshold of 0.88, and the top-5 segments form the evidence set; the large language model is a locally deployed Chinese generative model with hundreds of billions of segments, and the input is "conversation summary, evidence set, task instructions" JSON, and the output is a dynamic reference answer according to the preset structured mode.

[0160] The state graph construction follows the node and edge rules of implementation method S3, with alignment anchors including order number / amount / time. The number of multi-judge instances is 5, with decoding temperatures of 0.6 / 0.7 / 0.8 / 0.9 / 1.0 respectively. The sampling strategy alternates between top-k (k=40) and nullus (p=0.9). The consistency metric uses Kendall's W normalization to [0,1], and the threshold... .

[0161] The fusion phase is performed according to the formula given in the instruction manual (for low consistency). High consistency The fusion control model uses GBRT, and calibration uses PlattScaling. All thresholds are set according to the "Parameter Range" settings in the instruction manual.

[0162] Table 1. Objective alignment results of the present invention on 1,000 voice conversations.

[0163] As shown in Table 1, compared to single-judge LLM, this invention reduces the mean absolute error (MAE) by 35.2%–40.2% across the four dimensions and by 37.9% in the comprehensive dimension, indicating that "state graph structured scoring + multi-judge consistency + adaptive fusion" can stably reduce the deviation from human standards. The process and compliance dimensions show the most significant improvement, benefiting from graph alignment and mandatory / dependency constraints; the emotion dimension shows a smaller improvement, mainly because subjectivity and acoustic emotional cues are more sensitive to ASR noise.

[0164] The mean consistency index C_consist ranges from 0.63 to 0.76, with the lowest value in the sentiment dimension, confirming the aforementioned difficulties; the mean structural similarity index S_struct remains between 0.71 and 0.82, indicating that the topological alignment quality of the actual / ideal state diagram is relatively good.

[0165] Anomaly detection: The recall rate for identifying non-closed-loop processes on the full sample is 0.88% and the precision rate is 0.91%; the accuracy rate for knowledge version inconsistency alarms is 0.98%.

[0166] Performance: End-to-end P95 latency of 1.26s / minute audio (offline batch processing); average GPU utilization of 63%; failure retry rate of 1.7% (mainly due to missing LLM structured fields, which has been reduced by using fallback templates).

[0167] Practical derivation example of a one-way call (excerpt): The scenario is "return and shipping instructions", aggregated score. Structured scores Consistency indicators State diagram similarity (The critical preceding node "identity verification" is missing, and the path order is reversed in one place), Node type weight average Calculate using a low consistency strategy Based on this

[0168]

[0169] After the score was lowered, the system gave a risk label in the "process integrity" dimension of the session: missing node = identity verification, no closed loop = refund confirmation; the reviewer confirmed that the judgment was consistent with the evidence.

[0170] Key points for implementation:

[0171] Data input to the lake: audio 16kHz PCM; session object JSON field set includes session_id, speaker, text, start_ts, end_ts, asr_conf, kb_version;

[0172] Retrieval and Reference: BERT semantic matching threshold 0.88, Top-5; LLM output is validated for field integrity using regular expressions, and missing fields are rolled back and regenerated.

[0173] Graph and Alignment: Node / path consistency and penalty items are calculated according to the formula in the instruction manual, with a time tolerance of ±15s and a monetary tolerance of ±0.01.

[0174] Multiple referees: Heterogeneous configuration of 5 instances ensures statistical independence; consistency is achieved using Kendall's W;

[0175] Fusion and calibration: GBRT regression fusion + PlattScaling calibration, the training set used 400 channels manually scored in the first two days, the validation set used 200 channels, and the test set used 400 channels (results in this table).

[0176] In summary, by adopting the method and parameter settings of this invention, interpretable and robust multi-dimensional automatic scoring is achieved on real production data, which is significantly better than rule baselines and single-judge LLM, and is feasible for implementation in enterprise quality inspection and production environments.

[0177] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-dimensional automatic scoring system for customer service dialogue based on a large language model, characterized in that: Includes the following modules: The conversation processing module is used to receive customer service voice conversation data, perform speech recognition, speaker separation based on speech embedding and text standardization processing, and generate conversation objects with knowledge base version identifiers. The evidence retrieval module is used to retrieve a set of evidence related to the conversation intent from the knowledge base based on the conversation object, and call the large language model to generate a dynamic reference answer that conforms to the preset structured pattern; The state diagram construction module is used to parse a session object into an actual session state diagram consisting of multiple key event nodes, time, and dependencies, and to parse a dynamic reference answer into an ideal session state diagram. The structured scoring module is used to calculate the structural similarity between the actual session state graph and the ideal session state graph based on node matching and path consistency, and to calculate the structured score by combining the differences in node attributes. The judge scoring module is used to input the conversation object, evidence set, and dynamic reference answer into multiple independently configured large language model judge instances, and output a set of judge scoring results. The consistency calculation and scoring fusion module is used to calculate the consistency index based on the set of judges' scoring results, weight the aggregated score and the structured score according to the weight coefficient corresponding to the consistency index, and output the final score of each scoring dimension through a regression model based on multi-dimensional features. After scoring calibration, the structured scoring result is generated.

2. The customer service dialogue multi-dimensional automatic scoring system based on a large language model according to claim 1, characterized in that, The modules are connected in the following way: Receive customer service voice conversation data, perform speech recognition, speaker separation and text standardization processing, and generate conversation objects with knowledge base version identifiers; Based on the conversation object, retrieve a set of evidence related to the conversation intent from the knowledge base, and call the large language model to generate a dynamic reference answer that conforms to the preset structured pattern; The session object is parsed into an actual session state diagram consisting of multiple key event nodes, time, and dependencies, and the dynamic reference answer is parsed into an ideal session state diagram. The structural similarity between the actual session state graph and the ideal session state graph is calculated based on node matching and path consistency, and a structured score is calculated by combining the differences in node attributes. Input the conversation object, evidence set, and dynamic reference answer into multiple independently configured large language model judge instances, and output a set of judge scoring results; A consistency index is calculated based on the set of referee scores. When the consistency index is below a threshold range, an adaptive score fusion strategy is triggered. When the consistency index is above the threshold range, the scores are aggregated according to the consistency weight to obtain a structured score result.

3. The multi-dimensional automatic scoring system for customer service dialogue based on a large language model according to claim 2, characterized in that, The process of generating a session object with a knowledge base version identifier includes: labeling the transcribed text according to agent and user roles using speaker separation technology based on speech embedding; and selecting the version number closest to and no later than the end time from the effective version records of the knowledge base based on the session end time as the knowledge base version identifier.

4. The multi-dimensional automatic scoring system for customer service dialogue based on a large language model according to claim 2, characterized in that, The retrieval of evidence set related to the session intent includes: limiting the retrieval scope based on the knowledge base version identifier in the session object, sorting the knowledge base by similarity based on the intent vector generated by the semantic matching model, and selecting knowledge fragments with similarity higher than a preset threshold as the evidence set.

5. The multi-dimensional automatic scoring system for customer service dialogue based on a large language model according to claim 2, characterized in that, The steps for generating the dynamic reference answer include: Perform semantic parsing on the session object to generate a session summary; Extract key points from the evidence set to form a knowledge point set; Based on the correspondence between conversation summaries and knowledge point sets under each scoring dimension, a structured output containing scoring dimension fields, knowledge point coverage fields, and evidence citation fields is generated through a large language model. The output results are then mapped and their integrity is verified according to a structured model to obtain a dynamic reference answer.

6. The multi-dimensional automatic scoring system for customer service dialogue based on a large language model according to claim 2, characterized in that, The steps for constructing the actual session state diagram and the ideal session state diagram include: Perform intent recognition and slot filling on round texts arranged in chronological order to generate a set of event nodes containing event type, occurrence time, speaker role, slot key value and evidence citation identifier; Establish time sequence edges based on adjacent time relationships and record time intervals; establish dependency relationship edges based on the pre- and post-constraints of mandatory actions and record trigger conditions; time intervals are in the range of milliseconds to minutes. Merge nodes of the same type that have the same content and are adjacent in time; fold nodes that do not carry slot key values; mark processes that are repeated and do not meet the closed-loop conditions by loop counting, with the loop count being a non-negative integer. Event nodes include at least process nodes, emotion handling nodes, knowledge citation nodes, and compliance statement nodes. Process nodes are used to represent requirement confirmation, solution provision, and closed-loop actions. Emotion handling nodes are used to represent apologies, reiteration, and reassurance actions. Knowledge citation nodes are used to represent citations of knowledge base entries with source identification. Compliance statement nodes are used to represent pricing standards, compensation conditions, and prohibited language statements. Map the mandatory points in the dynamic reference answer to an ordered sequence of process nodes, map the policy references in the dynamic reference answer to a set of compliance statement nodes, map the closed-loop conditions in the dynamic reference answer to termination nodes, and establish dependency edges in the ideal session state diagram according to the pre- and post-relationships between the points. Write the knowledge base version identifier into both the actual session state diagram and the ideal session state diagram to ensure that the source entry of the knowledge reference node is consistent with the knowledge base version identifier. Write alignment anchors for nodes in both state diagrams. The alignment anchors include entity identifiers and slot key-value pairs. The entity identifiers cover the order number, amount, and time fields. The amount is represented in a uniform currency unit and the numerical precision is in the range of two to four decimal places. The time is represented in a uniform time standard and the alignment time window is in the range of seconds to minutes. When a slot is missing, the corresponding node is marked as incomplete and retained.

7. The multi-dimensional automatic scoring system for customer service dialogue based on a large language model according to claim 2, characterized in that, The consistency index calculated based on the actual session state diagram and the ideal session state diagram includes: Based on the alignment anchor point of the node, node matching and missing detection are performed. Required nodes that cannot be matched are recorded as missing nodes. For matched nodes, attribute consistency scores are calculated. The attribute consistency scores are calculated based on the matching degree of slot key-value pairs. Slot matching degree is recorded as consistent when the string similarity is greater than a preset threshold. Based on the temporal order edges and dependency edges between matching nodes, an aligned path set is constructed. The consistency score of each path is calculated in terms of node order, node type, and edge attributes. The path consistency score is then obtained by weighting the paths according to their coverage. For recurring non-closed-loop processes, a penalty weight is assigned based on the loop count during the consistency calculation, and the penalty weight is located in the range (0,1). The structural similarity is calculated by combining the node consistency score and the path consistency score according to a preset ratio. The structural similarity reflects the degree of similarity between the two state graphs in the overall topology. In the structural similarity calculation, a node importance weight is introduced. The node importance is based on a preset node type, wherein the weight of process nodes and compliance declaration nodes is higher than that of greeting nodes and emotion handling nodes. The consistency index is calculated by combining the node consistency score, path consistency score, and structural similarity using a fusion formula. The consistency index is located in the interval [0,1].

8. The multi-dimensional automatic scoring system for customer service dialogue based on a large language model according to claim 2, characterized in that, The steps for generating the set of referee scoring results include: The number of the judged instances is between three and seven, and there are differences in at least one of the random initialization parameters, decoding temperature and sampling strategy, prompt word template and evidence window interception strategy to form statistical independence; Each judge instance outputs a judge scoring record according to a preset structured pattern. The judge scoring record includes at least the initial score for each scoring dimension, a reference answer key point coverage table, a set of cited evidence identifiers, an index of reasoning fragments, evidence coverage rate, and judge uncertainty index. The judge scoring record is subjected to format integrity verification and evidence validity verification. The evidence validity verification requires that the cited evidence falls within the evidence set and that the knowledge base version identifier is consistent with the knowledge base version identifier of the session object. The verified referee scoring records are organized into a set of referee scoring results according to the referee instance identifier and the scoring dimension identifier, which serves as the input for consistency index calculation and subsequent scoring fusion.

9. The customer service dialogue multi-dimensional automatic scoring system based on a large language model according to claim 2, characterized in that, The steps for generating the final score include: When the consistency index is lower than the preset threshold, an adaptive scoring fusion strategy is triggered, which dynamically adjusts the proportion of the aggregated score based on the structural similarity and node type weights. When the consistency index is higher than the preset threshold, the aggregation score and the structured score are weighted according to the weight coefficient corresponding to the consistency index. The fusion control model is a regression model based on multidimensional features. The inputs include the original values ​​of consistency index, structural similarity, node consistency score, path consistency score, and aggregation score and structured score. The output of the regression model is the predicted score and corresponding confidence level of each scoring dimension. The scoring dimensions should include at least the completeness of business processes, the accuracy of knowledge citations, the standardization of compliant expression, and the effectiveness of emotional handling. The predicted scores are processed by a calibration model to eliminate scaling bias between different dimensions. The calibration model is trained based on historical scoring results and the distribution of standard answers. The final output is a structured scoring result, which includes the final score for each scoring dimension, the overall confidence level, the set of cited evidence, and the risk label.

10. The multi-dimensional automatic scoring system for customer service dialogue based on a large language model according to claim 9, characterized in that, The adaptive scoring fusion strategy includes: When the consistency index is lower than the preset threshold, the aggregation score adjustment coefficient is dynamically calculated based on the structural similarity and node type weight. The adjustment coefficient is located in the (0,1) interval, and the proportion of aggregation score is reduced and the proportion of structured score in the dimension of node type is increased according to the adjustment coefficient. The node type weights are set in the order of process node > compliance declaration node > knowledge reference node > emotion handling node, and the difference between adjacent weights is in the range of 0.05 to 0.

2. When the consistency metric is higher than the threshold, the original aggregation score percentage is maintained and a fusion calculation weighted by the consistency metric is performed.

Images