A method for auditing confidence scores of incrementally composable multi-signal traces

Abstract

The application relates to the technical field of artificial intelligence agent control, and discloses a confidence score method for auditing incremental combinable multi-signal trajectories, which comprises the following steps: receiving a complete execution trajectory generated by an intelligent agent during task execution, extracting multiple types of heterogeneous signals from the trajectory and forming a signal set; registering multiple signal contributors and arranging them in a preset order to form a scoring chain, wherein the first contributor is a baseline emission contributor; outputting the signal set and the signal contributor list; receiving the signal set and the baseline emission contributor configuration, accumulating the incremental value to the current confidence to complete a confidence adjustment; obtaining an intermediate confidence after all incremental contributors complete execution; and simultaneously generating an auditable scoring report containing a baseline value, incremental information and execution states of each incremental contributor, a final confidence and an execution decision. The application solves the problems of the prior art, such as black box non-attributability, signal failure interruption and difficulty in expanding new dimensions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence control technology, specifically to a confidence scoring method for auditing incrementally combinable multi-signal trajectories. Background Technology

[0002] The intelligent agent autonomously executes a series of operations, including code writing, compilation, execution, error correction, and testing, forming a complete execution trajectory. At each step of this trajectory, the agent continuously generates various quality signals, such as compilation error types, test results, file artifact verification status, repetitive loop detection signals, and task progress stall indicators. The core issue in improving the reliability of the agent's autonomous execution is how to integrate these heterogeneous signals into a unified confidence score and, based on this score, make decisions such as "continue execution," "escalate to human review," or "terminate execution."

[0003] Existing technologies for evaluating the confidence level of execution trajectories mainly fall into three categories: The first category is based on black-box scoring models. These methods train an independent scoring model, inputting the entire execution trajectory into the model and outputting a confidence score. Their drawback is the complete lack of transparency in the scoring process, making it impossible to answer the question of "why this score was given," and they do not support attribution analysis for individual signal dimensions. When the scoring model makes a misjudgment, it is impossible to pinpoint which specific signal caused the misjudgment. The second category is based on large language model self-evaluation. These methods allow the executing agent itself or an independent judging large language model to read the entire trajectory and output an overall execution quality judgment in natural language. Their drawback is the severe anchoring effect and illusion tendency of large language model self-evaluation; the model tends to give higher scores to its own generated execution results, and the scoring criteria are unstable, with the same trajectory potentially receiving significantly different scores in different contexts. The third category is hard-judgment based on pre-set rules. These methods set a fixed threshold, for example, classifying a task as a failure if more than three compilation errors occur. Its drawback lies in the rigidity of the rules, which makes it unable to capture the complex interactive effects of different signal combinations. For example, a minor warning plus a successful product verification may result in a higher actual quality than a scenario with zero warnings plus zero product verification.

[0004] Therefore, to address the above issues, a confidence scoring method for auditing incremental combinatorial multi-signal trajectories is needed. Summary of the Invention

[0005] The purpose of this invention is to provide a confidence scoring method for auditing incrementally combinable multi-signal trajectories. This invention achieves item-by-item attribution confidence scoring and fine-grained execution decisions through a multi-signal contributor baseline, an incremental chain architecture, a full-scale audit mechanism, a contributor fault-tolerant degradation strategy, and a final clamping decision rule.

[0006] This invention is implemented as follows:

[0007] This invention provides a confidence scoring method for audit incrementally combinable multi-signal trajectories, comprising:

[0008] The system receives the complete execution trajectory generated by the intelligent agent during the execution of the task, extracts multiple heterogeneous signals from the trajectory and forms a signal set; registers multiple signal contributors and arranges them in a preset order to form a scoring chain, where the first contributor is the baseline transmission contributor and the remaining contributors are incremental contributors; and outputs the signal set and the list of signal contributors.

[0009] The received signal set and baseline transmitter contributor configuration are used. The baseline transmitter contributor extracts the corresponding signal from the signal set, calculates the absolute baseline confidence value according to the preset baseline scoring rules, initializes the current confidence value to the absolute baseline confidence value, and outputs the absolute baseline value and the initialized current confidence value.

[0010] The incremental contributors are executed sequentially according to the preset order of the scoring chain. Each incremental contributor extracts its own corresponding subset of signals from the signal set, calculates a signed confidence increment value, and adds the increment value to the current confidence to complete a confidence adjustment. After all incremental contributors have been executed, the intermediate confidence is obtained, and an incremental audit record containing the corresponding increment values ​​of each incremental contributor and the input signal is generated. The intermediate confidence and the incremental audit record are output.

[0011] Force all signal contributors to execute in a pre-defined order, regardless of whether the current confidence level exceeds the pre-defined threshold during execution;

[0012] The intermediate confidence scores, after fault tolerance processing, are clamped to a preset floor threshold and ceiling threshold to obtain the final confidence score; the final confidence score is then output.

[0013] Based on the preset range of the final confidence level, the corresponding execution decision is mapped and output; at the same time, an auditable scoring report containing the baseline value, incremental information and execution status of each incremental contributor, final confidence level, and execution decision is generated; the execution decision and auditable scoring report are output.

[0014] Furthermore, the absolute value of the signed increment value output by each incremental contributor does not exceed the preset maximum adjustment range per step.

[0015] Furthermore, each incremental contributor only outputs a signed relative adjustment amount, without recalculating the absolute baseline.

[0016] Furthermore, if the current confidence level is lower than the preset minimum threshold or higher than the preset maximum threshold, all subsequent signal contributors will still execute in the preset order.

[0017] Furthermore, the fault tolerance and degradation processing includes not adjusting the confidence level when the signal contributor crashes, times out, or the input data is corrupted, while leaving a fault mark in the audit track to prevent interruption of the execution of the entire scoring chain.

[0018] Furthermore, regardless of whether the current confidence level exceeds the preset threshold range during execution, if any signal contributor experiences an execution failure, the incremental value corresponding to that contributor is set to zero, and the contributor is marked as faulty in the audit record; a complete audit trajectory record containing the execution status and corresponding values ​​of all contributors is generated; and the complete audit trajectory record and the intermediate confidence level after fault tolerance processing are output.

[0019] Furthermore, the execution decision is divided into three levels: when the confidence level is higher than the first threshold, the decision to continue execution is output; when the confidence level is between the first threshold and the second threshold, the decision to upgrade and review is output; and when the confidence level is lower than the second threshold, the decision to terminate execution is output.

[0020] Furthermore, the signal set supports incremental expansion: when adding a new signal dimension, it is only necessary to register the corresponding new incremental contributor and append it to the end of the scoring chain, without modifying the scoring rules of existing contributors.

[0021] Furthermore, the present invention provides a confidence scoring system for auditing incremental combinable multi-signal trajectories, including a trajectory input and registration module, which receives the trajectory executed by the agent, extracts heterogeneous signals to form a signal set, and registers and sorts the signal contributors to form a scoring chain;

[0022] The baseline transmission module calculates the absolute baseline confidence level and completes the confidence level initialization through baseline contributors;

[0023] The incremental chain processing module calculates the signed increments sequentially through each incremental contributor and accumulates and adjusts the confidence level to generate incremental audit records.

[0024] The full execution of the audit and fault tolerance module forces all contributors to complete the full execution, performs zero-incremental degradation on faulty contributors, and generates a complete audit trail.

[0025] The confidence clamping module clamps the intermediate confidence scores within a preset threshold range to obtain the final confidence score.

[0026] The decision output module maps the output to the execution decision and generates an auditable scoring report.

[0027] Furthermore, the present invention provides a computer storage medium storing a computer program that can run on a processor, wherein the processor executes the computer program to implement the confidence scoring method for audit incremental combinatorial multi-signal trajectories described in any one of the above-mentioned methods.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] 1. This invention endows each contributor with independent attribution capabilities through a structured scoring chain of baselines and increments. Compared to existing black-box scoring models, this invention can clearly answer the contribution amount and reasons for each signal dimension, supporting item-by-item tracing and manual review. Furthermore, the failure of a single contributor in this invention is downgraded to a zero-increment signal without disrupting the scoring chain. Compared to existing methods where the failure of a single signal source leads to the interruption of the entire scoring process, this invention achieves optimal fault tolerance and availability.

[0030] 2. This invention adds a new signal dimension simply by registering a new contributor and appending it to the end of the scoring chain, without requiring retraining or modification of existing contributors. Compared to existing methods that require redesigning the entire process for each adjustment of the scoring strategy, this invention minimizes the combinatorial cost. Even if the confidence level has fallen below the floor value, all contributors are still forced to complete the process. Compared to the early stopping strategy of existing methods, this invention preserves complete adjustment information for all contributors, avoiding the loss of valuable signals. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort. Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system structure diagram of the present invention; Figure 3 This is a structural diagram of the storage medium of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to describe selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1

[0034] Please see Figure 1 The present invention will be further described in detail below with reference to specific implementation scenarios. This embodiment takes the VibeCoding scenario as an application scenario where an agent executes the task of "implementing a RESTful API service based on FastAPI, including three interfaces: user registration, login, and data query". The confidence score and execution decision of the agent's execution trajectory are performed as follows.

[0035] In this embodiment, specifically, the execution trajectory input and signal contributor registration agent completes code writing through multiple iterations. The execution trajectory includes compilation output, runtime test output, and file output records. The system extracts a signal set Σ from the trajectory, specifically including:

[0036] Misclassification signal This includes one ModuleNotFoundError, two TypeErrors, and one IndentationError.

[0037] cyclic signal The fourth round used the same repair strategy as the second round, which is considered a duplicate attempt.

[0038] Product verification signal The validation results of the three output files (main.py, auth.py, and db.py) all passed type checking, syntax analysis, and dependency integrity checks.

[0039] stall signal The fifth round of execution took 45 seconds, exceeding the preset stagnation threshold of 30 seconds.

[0040] In this embodiment, specifically, the system registers four signal contributors and sorts them to form a scoring chain: C1 is the baseline contributor, based on the product verification signal; C2 is the contributor to error severity, based on error classification signals; C3 is a contributor to the cyclic detection, based on the cyclic signal; C4 is a contributor to stagnation detection, based on stagnation signals.

[0041] In this embodiment, specifically, the baseline transmitter C1 receiver... The signal and all three product files passed verification. The absolute baseline value was calculated using baseline scoring function B. In this embodiment, all products passed verification, and the baseline value c0 = 0.80 was set. The current confidence level c was initialized to:

[0042] In this embodiment, specifically, the incremental chaining process executes the three incremental contributors C2, C3, and C4 sequentially, with each contributor calculating the signed increment Δ. i And accumulate: (Error severity contributor) receives Signal:

[0043] ModuleNotFoundError is a critical error, corresponding to an increment of -0.12;

[0044] The two TypeErrors are considered medium-level errors, each corresponding to an increment of -0.04, for a total of -0.08;

[0045] • IndentationError is a minor error, corresponding to an increment of -0.02; the total increment Δ2 = -0.12 - 0.08 - 0.02 = -0.22.

[0046] The updated confidence level is: c = 0.80 + (-0.22) = 0.58.

[0047] In this embodiment, specifically, C3 (cyclically detecting contributors) receives... Signal: Repeated repair strategy detected, indicating moderate cyclicality, corresponding to an increment Δ3 = -0.08. Post-update confidence: c = 0.58 + (-0.08) = 0.50.

[0048] In this embodiment, specifically, C4 (Stagnation Detection Contributor) receives... Signal:

[0049] A pause was detected, corresponding to an increment Δ4 = -0.05. The updated confidence level is: c = 0.50 + (-0.05) = 0.45. After all increment contributors have completed their execution, the median confidence level is: Simultaneously generate incremental audit records:

[0050] .

[0051] This step follows incremental design constraints, and all Δ i All are signed relative adjustment amounts, and the absolute value of each step increment does not exceed the preset maximum adjustment range of 0.2.

[0052] In this embodiment, specifically, the full execution and fault tolerance protection system forcibly records the execution results of all contributors: • C1: Status: success, baseline value: 0.80; • C2: State is success, increment -0.22; • C3: Status: success, increment: -0.08; • C4: Status: success, increment: -0.05.

[0053] This embodiment verifies the full execution rule: even if the confidence level drops to 0.58 after C2 is executed, which is lower than the subsequent set threshold of 0.6 for continuing execution, the system still forces the execution of C3 and C4, retaining the adjustment information of loop detection and stall detection, and avoiding information loss.

[0054] As an example of a fault-tolerant scenario: If C3 experiences a timeout failure during execution, a fault-tolerant degradation strategy is implemented, setting Δ3 to 0 and marking C3's state as "error". The intermediate confidence level after fault tolerance processing is as follows:

[0055] Complete audit trail It includes all 4 contributor records and their corresponding statuses.

[0056] In this embodiment, specifically, confidence clamping is used in fault-tolerant scenarios. Taking 0.53 as an example, perform a clamping operation:

[0057] This embodiment is set =0.05, =1.0. Since 0.05≤0.53≤1.0, floor or ceiling protection was not triggered, and the final confidence level is... =0.53.

[0058] In this embodiment, specifically, the decision output sets a decision threshold:

[0059] Continue execution threshold =0.6, upgrade threshold =0.2.

[0060] In this embodiment =0.53, which satisfies:

[0061] If it falls into the "upgraded LLM" range, the system outputs... =“Upgraded LLM”.

[0062] Simultaneously, a complete and auditable scoring report is generated: Audit_Report={ baseline: 0.80, deltas:[ (-0.22, subset of misclassification signals, success) (0, cyclic signal subset, error) (-0.05, stagnant signal subset, success) ], final: 0.53, Decision: "Upgrade to LLM" }

[0063] After receiving the "upgrade to LLM" instruction, the agent framework will compare the current execution trajectory with... Both code snippets are submitted to a more robust language model for review. The language model's review process can then determine whether to continue adjusting the code or conduct manual acceptance testing of the deliverables.

[0064] Existing black-box scoring models only return a "low confidence" score, failing to pinpoint whether the score reduction is due to a severe, excessive negative contribution from errors or loop detection. Self-assessment using large language models may result in overestimation due to anchoring effects, ignoring loop risks. This invention, through a progressively controllable incremental chain, provides accurate attribution and flexible decision-making capabilities.

[0065] Example 2

[0066] This invention provides a confidence scoring system for auditing incrementally combinable multi-signal trajectories, such as... Figure 2 It includes a trajectory input and registration module, which receives the trajectory executed by the intelligent agent, extracts heterogeneous signals to form a signal set, and registers and sorts the signal contributors to form a scoring chain;

[0067] The baseline transmission module calculates the absolute baseline confidence level and completes the confidence level initialization through baseline contributors;

[0068] The incremental chain processing module calculates the signed increments sequentially through each incremental contributor and accumulates and adjusts the confidence level to generate incremental audit records.

[0069] The full execution of the audit and fault tolerance module forces all contributors to complete the full execution, performs zero-incremental degradation on faulty contributors, and generates a complete audit trail.

[0070] The confidence clamping module clamps the intermediate confidence scores within a preset threshold range to obtain the final confidence score.

[0071] The decision output module maps the output to the execution decision and generates an auditable scoring report.

[0072] Example 3

[0073] In this embodiment, as Figure 3The present invention provides a computer storage medium storing a computer program that can run on a processor, wherein the processor executes the computer program to implement a confidence scoring method for audit incremental combinatorial multi-signal trajectories as described above.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations will be apparent to those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A confidence scoring method for audit incremental combinatorial multi-signal trajectories, characterized in that: include: The system receives the complete execution trajectory generated by the intelligent agent during the execution of the task, extracts multiple heterogeneous signals from the trajectory and forms a signal set; registers multiple signal contributors and arranges them in a preset order to form a scoring chain, where the first contributor is the baseline transmission contributor and the remaining contributors are incremental contributors; and outputs the signal set and the list of signal contributors. The received signal set and baseline transmission contributor configuration are configured. The baseline transmission contributor extracts the corresponding signal from the signal set, calculates the absolute baseline confidence value according to the preset baseline scoring rules, and initializes the current confidence value to the absolute baseline confidence value. Output the absolute baseline value and the current confidence level after initialization; Each incremental contributor is executed sequentially according to the preset order of the scoring chain. Each incremental contributor extracts its own corresponding subset of signals from the signal set, calculates a signed confidence increment value, and adds the increment value to the current confidence to complete a confidence adjustment. After all incremental contributors have been executed, the intermediate confidence is obtained, and an incremental audit record containing the increment value corresponding to each incremental contributor and the input signal is generated. Output intermediate confidence level and incremental audit records; Force all signal contributors to execute in a pre-defined order, regardless of whether the current confidence level exceeds the pre-defined threshold during execution; The intermediate confidence scores after fault tolerance processing are clamped to be limited between the preset floor threshold and ceiling threshold to obtain the final confidence score. Output the final confidence score; Based on the preset range of the final confidence level, the corresponding execution decision is mapped and output; at the same time, an auditable scoring report containing the baseline value, incremental information and execution status of each incremental contributor, final confidence level, and execution decision is generated. Output execution decisions and auditable scoring reports.

2. The confidence scoring method for a combinable multi-signal trajectory of audit increments according to claim 1, characterized in that, The absolute value of the signed increment output by each incremental contributor does not exceed the preset maximum adjustment range per step.

3. The confidence scoring method for a combinable multi-signal trajectory of audit increments according to claim 1, characterized in that, Each incremental contributor only outputs a signed relative adjustment amount, without recalculating the absolute baseline.

4. The confidence scoring method for a combinable multi-signal trajectory of audit increments according to claim 1, characterized in that, If the current confidence level is lower than the preset minimum threshold or higher than the preset maximum threshold, all subsequent signal contributors will still execute in the preset order.

5. The confidence scoring method for a combinable multi-signal trajectory of audit increments according to claim 1, characterized in that, The fault tolerance and degradation processing includes not adjusting the confidence level when the signal contributor crashes, times out, or the input data is corrupted, while leaving a fault mark in the audit track and not blocking the execution of the entire scoring chain.

6. The confidence scoring method for a combinable multi-signal trajectory of audit increments according to claim 1, characterized in that, Regardless of whether the current confidence level exceeds the preset threshold range during execution, if any signal contributor experiences an execution failure, the incremental value corresponding to that contributor is set to zero, and the contributor is marked as faulty in the audit record. Generate a complete audit trail record containing the execution status and corresponding values ​​of all contributors; Output the complete audit trail record and the intermediate confidence level after fault tolerance processing.

7. The confidence scoring method for a combinable multi-signal trajectory of audit increments according to claim 1, characterized in that: The execution decision is divided into three levels: when the confidence level is higher than the first threshold, the decision to continue execution is output; when the confidence level is between the first and second thresholds, the decision to upgrade and review is output; and when the confidence level is lower than the second threshold, the decision to terminate execution is output.

8. The confidence scoring method for a combinable multi-signal trajectory of audit increments according to claim 1, characterized in that: The signal set supports incremental expansion: when adding a new signal dimension, it is only necessary to register the corresponding new incremental contributor and append it to the end of the scoring chain, without modifying the scoring rules of existing contributors.

9. A confidence scoring system for audit incremental combination of multiple signal trajectories, characterized in that: It includes a trajectory input and registration module, which receives the trajectory executed by the intelligent agent, extracts heterogeneous signals to form a signal set, and registers and sorts the signal contributors to form a scoring chain; The baseline transmission module calculates the absolute baseline confidence level and completes the confidence level initialization through baseline contributors; The incremental chain processing module calculates the signed increments sequentially through each incremental contributor and accumulates and adjusts the confidence level to generate incremental audit records. The full execution of the audit and fault tolerance module forces all contributors to complete the full execution, performs zero-incremental degradation on faulty contributors, and generates a complete audit trail. The confidence clamping module clamps the intermediate confidence scores within a preset threshold range to obtain the final confidence score. The decision output module maps the output to the execution decision and generates an auditable scoring report.

10. A computer storage medium storing a computer program executable on a processor, characterized in that, When the processor executes the computer program, it implements a confidence scoring method for audit incremental combinable multi-signal trajectories as described in any one of claims 1-8.

Images