SYSTEM AND METHOD FOR INTEGRATED EVALUATION OF INTELLIGENT AGENTS IN PRODUCTION

Description

1 / 13 SYSTEM AND METHOD FOR INTEGRATED EVALUATION OF INTELLIGENT AGENTS IN PRODUCTION FIELD OF THE INVENTION

[001] The present invention falls within the field of computer science, more specifically in the domain of monitoring, evaluation and observability systems of intelligent agents in production, including agents based on large language models (LLM) and machine learning models. More specifically, the invention relates to an integrated system that combines technical observability, automated semantic evaluation and unified analytics on intelligent agents in production. FUNDAMENTALS OF THE INVENTION

[002] Traditional monitoring systems for software applications focus on technical metrics such as latency, error rate, CPU and memory usage, and availability (uptime). While such metrics are essential for operational health, they do not capture the semantic quality of responses provided by intelligent agents, nor the adequacy of tool flows, user experience, security, or regulatory compliance.

[003] On the other hand, approaches to evaluating language models often rely on offline benchmarks with labeled datasets, or on manual labeling of transcript samples. Such approaches are expensive, scale poorly with increasing volumes of real interactions, and are generally not integrated into a continuous production monitoring cycle.

[004] Additionally, existing operational dashboards tend to present tool usage metrics, customer satisfaction indicators (CSAT), and business KPIs in a disaggregated manner, without a unified view that combines these aspects with quality signals lacking true labels. The previous technique does not show solutions that systematically integrate quality detectors, expert judges based on LLM, quality fact tables, and unified analytics by analysis period.

[005] Furthermore, in real-world contexts, many quality checkers are only applicable to subsets of traffic (e.g., only when a particular tool has been invoked or when a journey flow is present). When such inapplicable cases are implicitly treated as OK or removed from the denominator, denominator drift occurs (artificial rate changes caused by coverage changes), impairing comparability over time. Another practical problem is the selection bias introduced by selective evaluation (gating): if only flagged interactions are routed to costly judges, unflagged cases may contain unobserved failures (false negatives). The absence of explicit mechanisms for sentinel sampling and audit trail (prompt / template versions, rules, and slices of Petition 870260045559, dated 05 / 14 / 2026, page 26 / 42 2 / 13 replay) limits governance and reproducibility. STATE OF THE ART

[006] Some prior art documents disclose technologies that have objectives related to those of the present invention, but in which unresolved deficiencies still persist.

[007] Document CN117806980B, entitled “Automatic test case generation device based on large language model”, refers to an automatic test case generation device based on a large language model, in which an LLM agent with memory, tooling, planning and execution components is used to produce test cases from domain data and prompt engineering techniques, focusing on improving the efficiency and quality of electronic testing. A first class of related solutions, therefore, concerns machine learning and LLM model observability platforms and devices or LLM agents for specific tasks, which offer functionalities such as data and model drift monitoring, prediction anomaly detection, execution trace analysis, calculation of aggregate technical metrics and test case generation.

[008] Document CN117633225B, entitled “Alignment evaluation method for Chinese large language model”, proposes a method for measuring alignment for large language models, constructing an alignment dataset with task-oriented questions, rule-calibrated reference responses and evaluation categories (such as general questions and answers, writing, logical reasoning, mathematics and specialized knowledge), and using responses generated by large models as a reference base.

[009] Document US10832002 B2, entitled “System and method for scoring performance of chatbots”, discloses a method, apparatus, system, and computer program product for evaluating the performance of a chatbot by capturing specific node performance metrics based on user interactions with one or more nodes in a chatbot dialogue flow; assigning a score to each of the nodes based on specified node performance metrics; and evaluating a performance measure for at least one part of the chatbot by combining the scores assigned to selected nodes in the dialogue flow.

[010] Document US20240154921 A1, entitled “QA data evaluation apparatus”, proposes a QA data evaluation apparatus that includes: an acquisition unit configured to obtain QA data, including the content of a user's question to a chatbot, the content of the corresponding response, and log information about the user's use of the chatbot; an extraction unit configured to extract a feature value related to the temporal behavior of chatbot usage. Petition 870260045559, dated 05 / 14 / 2026, page 27 / 42 3 / 13 by the user from the registration information; and a generation unit that generates QA data evaluation information, representing whether the QA data is good or bad based on the feature value.

[011] These works demonstrate the potential of language models to act as automatic evaluators in offline benchmarking and alignment scenarios, but, in general, they do not describe a production architecture that systematically integrates quality detectors as a cost filter, “not applicable” aware metrics, an engine of expert judges with calibration and production modes, and a unified analytics layer that operates continuously on real production logs from multiple agents.

[012] There are also market solutions and software projects that provide agent / LLM tracing and observability, offline evaluation and / or application instrumentation. Examples include observability and tracing platforms for LLM and agent applications (e.g., LangSmith, Arize Phoenix, and traceability / experimentation platforms such as MLflow) and evaluation libraries (e.g., RAGAS and TruLens).

[013] In general, such solutions provide trace collection, tool call inspection, prompt and execution logging, and / or mechanisms for evaluating responses through metrics and judges. However, the following combined elements are not systematically identified in a single integrated architecture: (i) detectors with triestated output and metrics consciously sensitive to not applicable and numerical invariants; (ii) explicitly declared judge selection policy with modes (gated and sentinel), quotas, token / latency budgets, and reason codes for auditing; (iii) canonical fact table persisted in analytical media with lineage and reproducibility fields (hash of prompts / models and replay slices); and (iv) a refinement cycle that transforms structured judge recommendations into candidate rules for detector evolution.

[014] Thus, although the prior art and the cited patent documents acknowledge, in a fragmented way, the importance of model observability, the use of LLM as evaluators, LLM-based agents, and conceptual frameworks for agent evaluation and governance, they do not identify solutions that combine, in a single integrated system: (i) an ingestion module that converts heterogeneous logs from different agents into a canonical interaction scheme; (ii) a triestated detector engine with NA-aware metrics and numerical invariants; (iii) a specialized judge engine with distinct modes for calibration and production and explicit gating and cost control mechanisms; (iv) a module for a persisted interaction quality fact table in an analytical environment; and (v) an analytics layer that, from this fact table, provides general analyses, tool-specific analyses with semantic message clustering, in a single dashboard. Petition 870260045559, dated 05 / 14 / 2026, page 28 / 42 4 / 13 satisfaction analyses and observability views guided by detectors and judges, operating on demand or in near real-time mode. BRIEF DESCRIPTION OF THE INVENTION

[015] The invention proposes an integrated system for evaluating intelligent agents in production without true labels, comprising: an ingestion and normalization module configured to convert heterogeneous operational logs into a canonical database of interactions; a detector engine composed of a library of quality detectors, configured to consume the canonical database and to return triestated values ​​(OK, Problem, Not Applicable) for each interaction; a metrics module configured to aggregate the triestated values ​​into canonical metrics for each detector of the detector engine; a gating mechanism configured to select a subset of the interactions for analysis by an engine of expert judges, the subset being selected based on the outputs of the detectors, the selection comprising cost control by sampling and quotas;An expert judge engine based on large-scale language models (LLMs) or machine learning (ML) models, where each judge in the expert judge engine is triggered only on the subset of interactions in the canonical interaction scheme, the expert judge engine being configured to produce enriched outputs containing at least the actual problem, severity, category, justification, and recommendation; and a metrics and quality fact table module configured to produce canonical metrics per detector and a fact table per interaction combining normalized data, detectors, and judges. [01 6] Optionally, the system also includes an analytics and dashboard layer that provides, for selected date ranges or executions, integrated views of tool usage, LLM usage, customer satisfaction indicators (CSAT) and quality observability, optionally with drill-down execution to the session or interaction level.

[017] The present invention further provides an integrated method for evaluating intelligent agents in production without true labels, performed by said system. BRIEF DESCRIPTION OF THE FIGURES

[018] The present invention will be described below, with reference to the accompanying figures which illustrate examples of its embodiment.

[019] Figure 1 is a flowchart illustrating a high-level view of the architecture of the present invention. DETAILED DESCRIPTION OF THE INVENTION [02 0] Specific embodiments of this disclosure are described below. In an effort to provide a concise description of these embodiments, all features of an implementation Petition 870260045559, dated 05 / 14 / 2026, page 29 / 42 5 / 13 actual implementations may not be described in the descriptive report. It should be appreciated that in the development of any actual implementation, as in any engineering or design project, numerous specific implementation decisions must be made to achieve the specific objectives of the developers, such as compliance with system and business-related constraints, which may vary from one implementation to another. Furthermore, it should be appreciated that such a development effort may be complex and time-consuming, but would nevertheless be a routine design and manufacturing undertaking for those of average skill, having the benefit of this disclosure. General Architecture

[021] The general architecture of the invention can be understood from a block diagram, illustrated schematically in Figure 1. In a typical configuration: Operational data sources (analytic databases, message queues, log files) feed into an ingestion and normalization module; The ingestion and normalization module produces a canonical basis of interactions, where each line represents an interaction with standardized fields for input text, output text, invoked tools, state, and metadata; A detector engine is configured to consume this database and write a table of detection flags and scores, returning a value of {ok, problem, not applicable} for each interaction and for each detector. a metrics module configured to aggregate these results into canonical metrics per detector; a gating mechanism configured to select a subset of interactions for analysis by a specialized judge engine; the aforementioned engine of specialized judges, configured to produce enriched outputs containing, at least, an indication of the real problem, its severity, category, justification, and recommendation; A quality fact table module configured to combine normalized, detector, and judge data into an interaction quality fact table; and an analytics layer configured to consume both the canonical metrics and the fact table to provide interactive dashboards.

[022] In a representative embodiment, the integrated execution of the system can be described, at a high level, by the following steps: (i) selection, by a technical or business user, of an analysis period or a sliding time window for evaluation; (ii) reading, by the ingestion and normalization module, of interaction records in the sources of Petition 870260045559, dated 05 / 14 / 2026, pages 30 / 42 6 / 13 operational data corresponding to the selected period and conversion of these records to the canonical interaction scheme; (iii) application, by the detector engine, of a plurality of quality detectors on the normalized interactions, with generation of an intermediate table of flags and aggregated metrics per detector; (iv) selection, by a gating mechanism, of a subset of interactions to be forwarded to one or more expert judges, based on signals from the detectors, probabilistic sampling and configurable maximum limits; (v) evaluation, by expert judges, of the selected interactions, taking into account the session context and the signals from the detectors, and production of diagnoses of the real problem, severity, category, justifications and recommendations; (vi) construction, by the interaction quality fact table module, of a fact table that associates, to each session and interaction pair, the canonical data, the detector results, the judge results and derived aggregate fields; (vii) optional export of the generated fact table to a data lake-type analytical repository, in a columnar format partitioned by period; and (viii) consumption of the metrics and fact table by the analytics and dashboards layer, for the generation of overviews, analyses by tool, satisfaction analyses and observability views. Intake and Normalization Module

[023] The ingestion and normalization module is responsible for connecting to one or more interaction data sources (e.g., tables in a data lake, Athena queries, CSV / Parquet files, streams), for extracting raw fields of interest (messages, current agent state, execution traces, errors, metadata) from one or more interaction data sources, and for applying transformations to produce a canonical schema that contains, for each interaction: Session and interaction identifiers; User input text; Agent output text; List of tools called (tools_called); Structured representation of the state (current_state); Journey result; Error information (error_obj, trace); and Response times and other metadata. [02 4] This canonical schema is designed to be reusable across different agents, simply by parameterizing, on a per-agent basis, how each original data source is mapped to the fields. Petition 870260045559, dated 05 / 14 / 2026, pages 31 / 42 7 / 13 canonical. Tristate Detector Motor

[025] The detector engine consists of a collection of detection modules, each responsible for evaluating a specific quality dimension, for example and without limitation: Execution of transfers or transactions; Handling limits and overflow; Simulation errors; Validation errors; Missed opportunities to check balance; Handling QR codes without value; Hallucinations; Consistency between sessions; Privacy and security; User experience (tone, educational value, etc.). Each detector implements one of the following standard contracts: OK (True): No problem conditions were detected in the interaction; Problem (False): the detector identified a problem or risk; or Not Applicable (NA): the detector does not apply to that interaction.

[026] The metrics module calculates NA-aware metrics for each detector, guaranteeing invariants such as: applicable_count + na_count = rows; coverage_rate = applicable_count / rows; problem_count + ok_count = applicable_count; problem_pct_total + ok_pct_total = coverage_rate.

[027] These metrics allow comparison of detectors with different ranges of applicability, without distortion. Specialized Judges Engine

[028] The judge engine comprises a collection of specialized judges based on language models or machine learning models, for example and without limitation: Judge responsible for executing settlements; Goal quality judge; A judge of precision and hallucination; Judge of limits and overflow; Judge of safety and compliance; Petition 870260045559, dated 05 / 14 / 2026, pages 32 / 42 8 / 13 User experience judge; Judge for agent improvement.

[029] In a specific realization, the judge engine operates in at least two distinct execution modes: Calibration mode, in which all cases flagged by the detectors as potential problems are forwarded to the relevant judges, with the aim of measuring performance, adjusting prompts, and calibrating detection thresholds; or production mode, in which, in addition to the detector filter, additional gating rules are applied by domain, probabilistic sampling by judge, and global and session maximum ceilings, in order to substantially reduce the number of calls to language models without losing coverage in more critical cases.

[030] A resilient pipeline, with checkpoints per judge and per case, ensures that long executions can be resumed after infrastructure or API failures, reusing already computed results. Judges are not applied to all interactions, but only to a subset generated by a gating mechanism that uses at least one of the following: Flags of the detectors (for example, any detector indicating Problem); Judge-based probability sampling; Maximum limits per judge in a trial; Maximum limits per user session.

[031] Each judge receives the current interaction (including input and output texts, status, and tools), the session context (history of associated interactions), and alerts from the relevant detectors. The judge returns, for each evaluated case: a real problem indicator (Boolean), a severity level (critical, high, medium, low, not_applicable), a problem category (domain-dependent), a textual justification, and a structured recommendation for detector adjustment (detector_recommendation).

[032] When the available context is insufficient, ambiguous, or out of scope, the judge may explicitly return a not applicable (NA) indication or abstention for the actual problem field and / or for specific dimensions (e.g., security vs. enforcement), preserving applicability denominators in a manner analogous to the NA-aware metrics of detectors. Detector Refinement Module

[033] A refinement submodule aggregates, over time, the recommendations issued by the judges. From these recommendations, recurring patterns are extracted, such as: Input or output text patterns; Combinations of tools called; Petition 870260045559, dated 05 / 14 / 2026, pp. 33 / 42 9 / 13 Specific state or journey patterns.

[034] Based on these patterns, the module generates candidate rules (for example, in the form of logical expressions or code snippets) that can be incorporated into detectors to: reduce false positives, reduce false negatives, or refine the application of a detector to more relevant cases. Optionally, these rules can be submitted to a human approval step before being activated. Quality Metrics and Fact Sheet Module

[035] Based on the results from the detectors and the judges, the metrics module: calculates metrics per detector, as described previously; It aggregates metrics by judge (e.g., counts of cases indicating a real problem by severity, confidence distribution, etc.); It constructs an interaction quality fact table, in which each row corresponds to a pair (session, interaction), containing: canonical data (texts, state, tools, times), columns of flags and detector scores, columns of judge outputs, aggregated fields (e.g., any_detector_false, any_gold_detector_false, any_judge_real_problem, max_global_severity), indicators by detector family (risk, transaction, UX, security, etc.).

[036] This fact table can be exported to a data lake in columnar format (e.g., Parquet), partitioned by year, month, and day, allowing efficient queries by external analytical tools. Lineage, versioning and reproducibility (audible replay)

[037] In a preferred embodiment, the fact table and / or an associated execution manifest includes lineage fields that allow auditing and controlled replays, such as: Run identifier (run_id) and timestamp; Schema version (schema_version); Detector bundle version identifier (e.g., detector_bundle_version or hash); Judge identifiers (e.g., judge_model_id) and prompt hash (prompt_hash); Selection mode and hash / gating policy configuration (e.g., selection_mode, gating_policy_hash); and Input slice identifier and content hash (e.g., input_slice_hash).

[038] This makes it possible to compare executions on the same slice (same hash) under different configurations, isolating rule / prompt changes from traffic drift. As a non-limiting illustration, Table 1 presents typical categories of fields included in the interaction quality fact table. Petition 870260045559, dated 05 / 14 / 2026, pages 34 / 42 10 / 13 Table 1 Field Category Examples of columns Identification and context session_id, interaction_id, channel, timestamp, agent identifier Canonical data input and output texts, tools_called, current_state, journey_result, response times Detector results values ​​tried by detector, any_detector_false, any_gold_detector_false Judge results real problem indicators, severity, category, textual justification, recommendation by judge Aggregate fields any_judge_real_problem, max_global_severity, aggregate indicators by detector family Analytics and Dashboards Layer

[039] The analytics layer provides, via a web interface or equivalent, a dashboard in which the user explicitly selects a date range or one or more evaluation runs, structured in, for example, tabs: General Analysis: metrics for interaction volume, percentage of interactions using tools, percentage of interactions using LLM, latency distributions, correlations between tool usage and latency, comparison between interactions with and without LLM. These views allow for capacity scaling, evaluating the impact of using language models on service time, and guiding decisions on scaling, fallback, and prompt adjustments to maintain latency and experience within established goals.

[040] Analysis by Tool: for each tool, daily metrics (count, minimum, maximum, average, standard deviation), duration distribution of interactions that used the tool, and an advanced exploratory layer based on clustering of messages that triggered the tool, in which such messages are represented by vectors of semantic embeddings obtained from language models based on transformers, allowing automatic grouping of similar messages in a vector space and interactive adjustment of the number of clusters, as well as performing specific analyses such as use with / without LLM, resolution via different mechanisms (e.g., Bag of Words (BoW) vs LLM) and resolvability of ML models.These analyses, by relating message clusters that triggered each tool with execution time and success metrics, allow for the identification of clear patterns of inadequate or inefficient tool routing and guide adjustments to the agent, prompts, and orchestration rules, with a direct impact on accuracy, latency, and resolution. Petition 870260045559, dated 05 / 14 / 2026, pages 35 / 42 11 / 13

[041] CSAT: counting and temporal trend of satisfaction ratings, analysis of textual justifications, word clouds constructed by term counting (bag-of-words) segmented by rating category (low, medium, high). By combining this information with journey segments and tools used, the CSAT layer allows identifying recurring classes of dissatisfaction and prioritizing interventions in the agent and flows with the greatest impact perceived by the end user.

[042] Observability: view of detector and judge metrics (severity and confidence distribution), exploration of problem interaction identifiers by judge and detector, correlations between problems from different families, and suggestions for improvements derived from judges' recommendations. Together, these capabilities allow for the derivation of a prioritized backlog of technical and product improvements, classified by severity, frequency, and risk family, and support for evidence-based operational and regulatory risk mitigation decisions. Examples of Achievement Example 1: Instant Payment Financial Agent.

[043] As a first example, consider a conversational agent responsible for instant payments. Its production logs contain input and output messages, payment flow status; tools used (balance inquiry, transaction creation, limit validation, QR generation, etc.), and error and latency information. The ingestion and normalization module converts this data to the canonical schema. A detector library is configured for this domain, including, for example and without limitation, detectors for: Transaction successfully executed; Limit overflow in a specific channel; Valueless QR Code; Simulation errors; Input data validation errors; Insufficient funds; Hallucination regarding values ​​and limits; Consistency between sessions regarding stable facts; Privacy and security (e.g., improper disclosure of data); User experience (tone, clarity, educational value). Specialized judges are configured to assess, for example and without limitation, whether: The transaction was successfully executed; The user's objective was achieved; There was a factual hallucination; Limits and redirections were handled appropriately; Petition 870260045559, dated 05 / 14 / 2026, pages 36 / 42 12 / 13 The messages are secure and compliant; The overall experience is adequate.

[044] Detectors are run daily on production data, generating the detected_scored table and aggregate metrics. Then, the gating mechanism selects problematic interactions to be evaluated by judges, who return diagnoses and recommendations. The quality fact table is generated and the dashboard presents integrated metrics for product, engineering, operations, and compliance.

[045] In a specific embodiment, the tool analysis tab of this dashboard uses semantic vector representations of input messages, obtained through models based on transformers for generating sentence embeddings, and applies unsupervised clustering techniques in this vector space to form clusters of messages that triggered each tool, allowing the analyst to interactively explore recurring themes and intents by tool, adjusting the number of clusters. This makes it possible to detect groups of messages in which a particular tool is triggered redundantly, late, or inappropriately, correlate these clusters with latency and success rate of interactions, and, based on these findings, guide adjustments in the agent configuration, tool routing strategies, and prompts, in order to simultaneously improve accuracy and response time.The satisfaction tab, in turn, employs a lighter approach, based on word clouds constructed by counting terms in textual justifications segmented by rating ranges. Example 2: Credit Agent.

[046] The same framework can be applied to a credit agent, with specific tools (credit analysis, document collection, proposal approval, etc.), detectors focused on the correct application of credit rules, on customer guidance and regulatory compliance, and judges specialized in loan flow execution, risk analysis and compliance.

[047] Ingestion and normalization are adapted for the credit agent data schema, but the rest of the architecture (tristate detectors, judges, fact table, dashboards) remains the same. Example 3: General Customer Service Agent.

[048] In a third example, a general support agent uses the system to monitor technical metrics and problem resolution metrics, quality of information provided, support experience, and need for human escalation. Detectors focused on user experience and information compliance are used in conjunction with expert judges in call resolution and appropriate escalation. Petition 870260045559, dated 05 / 14 / 2026, pp. 37 / 42 13 / 13 Advantages and Technical Effects

[049] Among the technical advantages provided by the invention, the following stand out: The ability to measure the quality of intelligent agents in production without the need for actual labels, through a combination of detectors and specialized judges; Significant reduction in assessment costs, thanks to the use of detectors as a filter and the use of sampling and quotas to limit the volume of cases referred to LLM-based judges; Improved interpretability of metrics through the explicit use of non-applicable values ​​and numerical invariants that avoid inconsistencies in statistics; Ease of reuse of the platform across multiple agents and domains, with low incremental customization restricted to ingestion mapping and selection of detectors / judges; Support for exploratory analysis without labels through semantic vector representations of messages and unsupervised clustering techniques, revealing tool usage patterns, recurring intents, and latent problems that are not easily identifiable by traditional term counts, and providing objective input for adjustments to routing, rules, and agent prompts with an impact on accuracy and latency; Consolidation of operational data, quality signals (detectors and judges), and satisfaction feedback into a single analytical layer, allowing for the systematic derivation of prioritized lists of improvement opportunities, risk scenarios, and journey adjustments based on quantitative evidence; The possibility of continuous self-improvement of the detection logic, through rules suggested and refined based on the judges' recommendations.

[050] Although aspects of the present disclosure may be subject to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and have been described in detail in this document. But it should be understood that the invention is not intended to be limited to the particular forms disclosed. Instead, the invention should cover all modifications, equivalents and alternatives that fall within the scope of the invention, as defined by the following appended claims. Petition 870260045559, dated 05 / 14 / 2026, pp. 38 / 42

Claims

1 / 2 CLAIMS 1. A system for integrated evaluation of intelligent agents in production, characterized in that it comprises: an ingestion and normalization module configured to convert heterogeneous operational logs into a canonical database of interactions; a detector engine composed of a library of quality detectors, configured to consume the canonical database and to return triestated values ​​(OK, Problem, Not Applicable) for each interaction; a metrics module configured to aggregate the triestated values ​​into canonical metrics for each detector of the detector engine; a gating mechanism configured to select a subset of the interactions for analysis by an engine of expert judges, the subset being selected based on the outputs of the detectors, the selection comprising cost control by sampling and quotas;An expert judge engine based on large-scale language models (LLMs) or machine learning (ML) models, where each judge in the expert judge engine is triggered only on the subset of interactions in the canonical interaction scheme, the expert judge engine being configured to produce enriched outputs containing at least the actual problem, severity, category, justification, and recommendation; and a metrics and quality fact table module configured to produce canonical metrics per detector and a fact table per interaction combining normalized data, detectors, and judges.

2. System, according to claim 1, characterized in that it further comprises: an analytics and dashboard layer that provides, for selected date ranges or executions, integrated views of tool usage, LLM usage, customer satisfaction indicators (CSAT), and quality observability, optionally with drilldown execution to the session or interaction level.

3. System, according to claim 1 or 2, characterized in that the canonical interaction base comprises multiple lines, each line representing an interaction with standardized fields for input text, output text, invoked tools, state, and metadata.

4. Method for integrated evaluation of intelligent agents in production, characterized by the fact that it comprises the steps of: (i) selection of an analysis period from an operational data source; (ii) reading, by an ingestion and normalization module, of interaction records in the operational data source corresponding to the selected analysis period and conversion of said interaction records to a canonical interaction scheme; (iii) application, by a detector engine, of a plurality of quality detectors on the interactions of the canonical interaction scheme, comprising the generation of an intermediate table of flags and aggregated metrics per detector;(iv) selection, by a gating mechanism, of a subset of interactions to be forwarded to an engine of expert judges, the subset being selected based on the outputs of the detectors, the selection comprising cost control by sampling and quotas; (v) evaluation, by the engine of expert judges, of the selected interactions and production of diagnoses containing, at least, real problem, severity, category, justifications and recommendations; and (vi) construction, by an interaction quality fact table module, of a fact table that aggregates, for each session and interaction pair, canonical metrics per detector and a fact table per interaction combining normalized data, detectors and judges.

5. Method according to claim 4, characterized in that it further comprises the step of: (vii) exporting the interaction quality fact table to an analytical repository.

6. Method according to claim 4 or 5, characterized in that it further comprises the step of: (viii) consumption of interaction quality metrics and fact table by an analytics and dashboard layer. Petition 870260045559, dated 05 / 14 / 2026, pp. 40 / 42

Images