An agent publishing control method based on playback attribution

Abstract

The present application relates to the technical field of artificial intelligence, in particular to an agent release control method based on playback cause, comprising: collecting execution trajectories of agents in a production environment and determining corresponding execution contexts, generating multi-component version fingerprints according to the execution contexts, solidifying external dependencies in the execution trajectories, and constructing reproducible playback sample packages according to the execution trajectories, the execution contexts, the multi-component version fingerprints and the solidified external dependencies, driving a first release version and a second release version to execute in an isolated environment using the same reproducible playback sample packages, obtaining different node trajectories, calculating node-level degradation contributions of the second release version relative to the first release version after node alignment and performing component attribution, and controlling the release state of the second release version according to the component attribution result. The present application realizes execution condition reconstruction and external dependency reproduction, improves the accuracy of degradation root cause positioning, and improves the efficiency of agent version release control.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to an agent release control method based on playback attribution. Background Technology

[0002] In recent years, large language model intelligent agent technology has made significant progress and has been widely applied in scenarios such as intelligent customer service, knowledge-based question answering, business processing, R&D assistance, and automated operation and maintenance. A production execution of an intelligent agent typically involves multiple components, including prompt words, basic models and inference parameters, workflow topology, knowledge base index, retrieval configuration, external tools, permission policies, tenant data, and external services. These components collectively determine the execution path and final output of the intelligent agent. When any of the prompt words, models, workflows, knowledge base indexes, tools, or policies changes, the output quality of the intelligent agent may also change. Therefore, before the release of a candidate version or after a production issue occurs, it is usually necessary to use the execution trajectory generated in the production environment for offline replay and performance evaluation.

[0003] However, the relevant solutions have at least the following problems: only input and output and a few tracking fields are saved, and the complete execution context is not saved, so the original running conditions cannot be rebuilt after the production version is updated; external dependencies such as tool call results, retrieval results, and business data are constantly changing, so even if the input is the same, the replay results will differ due to external drift, making the comparison benchmark unreliable; only comparing the final output cannot locate the node where the difference first occurred, nor can it distinguish between upstream propagation effects and node self-effects; the evaluation results are disconnected from the release system, and manual judgment is required after degradation is discovered, resulting in delays in blocking and rollback.

[0004] Therefore, there is an urgent need for a method that can directly drive release control with attribution results. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an agent release control method based on playback attribution to solve the technical problems of root cause inability to locate and the disconnect between evaluation and release in related technologies.

[0006] In a first aspect, embodiments of the present invention provide an agent deployment control method based on playback attribution, the method comprising: The execution trajectory of the agent in the production environment is collected, and the execution context corresponding to the execution trajectory is determined. The execution trajectory includes the nodes traversed by the agent during execution, the dependencies between nodes, and the input and output information of the nodes. The execution context is used to characterize the component combination state and data state that the execution trajectory depends on during execution. A multi-component version fingerprint is generated based on the execution context, and the multi-component version fingerprint is used to uniquely identify the combination of components contained in the execution context; The external dependencies in the execution trajectory are solidified, and the external dependencies are external states, external call results, or non-deterministic operating conditions that affect the execution result of the execution trajectory. Based on the execution trajectory, the execution context, the multi-component version fingerprint, and the solidified external dependencies, a sample list and integrity verification information are generated to construct a reproducible replay sample package. In an isolated environment, the first release version and the second release version are executed using the same reproducible replay sample package to obtain the first node trajectory and the second node trajectory. Align the first node trajectory and the second node trajectory to obtain node alignment units; The node-level degradation contribution of the second release version relative to the first release version is calculated based on the node alignment unit, and component attribution is performed based on the node-level degradation contribution to obtain the component attribution result; Release control instructions are generated based on the component attribution results, and these instructions are used to control the release status of the second release version.

[0007] In conjunction with the first aspect, in one implementation, the execution context includes at least one of the following: tenant information, application information, agent information, release version information, prompt word version fingerprint, workflow version fingerprint, model version fingerprint, knowledge base index version fingerprint, tool version fingerprint, permission policy version fingerprint, and data watermark. Wherein, the tenant information is used to identify the tenant domain to which the execution trajectory belongs; the application information is used to identify the application to which the agent belongs; the agent information is used to identify the agent instance corresponding to the execution trajectory; the release version information is used to identify the release version of the agent corresponding to the execution trajectory; the prompt word version fingerprint is used to identify the version of the prompt word used by the agent; the workflow version fingerprint is used to identify the version of the workflow used by the agent; the model version fingerprint is used to identify the version of the model used by the agent; the knowledge base index version fingerprint is used to identify the version of the knowledge base index used by the agent; the tool version fingerprint is used to identify the version of the tool used by the agent; the permission policy version fingerprint is used to identify the version of the permission policy used by the agent; and the data watermark is used to identify the business data status when the execution trajectory is executed.

[0008] In conjunction with the first aspect, in one implementation, generating a multi-component version fingerprint based on the execution context includes: For each component in the execution context, a corresponding component fingerprint is generated based on the component's identification information, configuration information, and version information; The component fingerprints of all components are concatenated in a predetermined order, and their hash values ​​are calculated to obtain the multi-component version fingerprint.

[0009] In conjunction with the first aspect, in one implementation, the external dependency includes at least one of tool invocation results, retrieval recall results, business data, time information, and random number seeds, and the solidification of the external dependency includes: For the tool invocation result, store the invocation request to the target tool and the invocation response returned by the target tool when the execution trajectory is executed; For the retrieval and recall results, the retrieval query conditions, recalled document content, and sorting results during the execution of the execution trajectory are fixed; For the business data, create a read-only data snapshot of the business data at the execution time of the execution trajectory, or record a consistency identifier of the business data. The consistency identifier is used to obtain business data consistent with the execution time of the execution trajectory from the data source when driving the execution of the first release version or the second release version. For the time information, the logical clock during the execution of the execution trajectory is recorded. The logical clock is used to restore the timing relationship during the execution of the execution trajectory when driving the first release version or the second release version to execute. For the random number seed, the random number seed used when the execution trajectory is executed is recorded. The random number seed is used to make the random operation generate the same random sequence when driving the execution of the first release version and the second release version.

[0010] In conjunction with the first aspect, in one implementation, the step of aligning the first node trajectory and the second node trajectory to obtain the node alignment unit includes: The nodes in the first node trajectory and the nodes in the second node trajectory are matched using node identifiers, which are used to uniquely identify a node in the agent's workflow. When a node cannot be matched by the node identifier, a semantic signature is generated based on the node's attribute information, and the unmatched node is matched based on the semantic signature. The attribute information includes at least one of the node's position information in the workflow, the node's input and output feature information, and information about the components referenced by the node. Node alignment units are generated based on the matching results, and the alignment confidence of each matching node pair is determined. The node alignment units include matching node pairs, newly added nodes, and deleted nodes.

[0011] In conjunction with the first aspect or its corresponding implementation, in one implementation, the step of calculating the node-level degradation contribution of the second release version relative to the first release version based on the node alignment unit includes: Calculate the end-to-end overall degradation of the second release version relative to the first release version, where the end-to-end overall degradation is the positive part of the difference between the end-to-end score of the first release version and the end-to-end score of the second release version; The local quality difference of each matching node pair in the node alignment unit is obtained, and the local quality difference is used to characterize the degree of change in the output quality of the second release version node relative to the first release version node in the matching node pair; Obtain the propagation weight of each node in the matching node pair. The propagation weight is used to characterize the degree of influence of the output difference of the node on the propagation to downstream nodes. The contribution value of each node in the matched node pair is calculated based on the local quality difference, the propagation weight, and the alignment confidence of the node alignment unit, wherein the alignment confidence is a confidence level determined during the node alignment process; For the newly added node in the node alignment unit, the contribution value of the newly added node is determined according to the degree of influence of the newly added node on the overall output of the second release version; For the deleted node in the node alignment unit, the contribution value of the deleted node is determined according to the degree of contribution of the deleted node to the overall output of the first release version; The contribution values ​​of the matched node pairs, the newly added nodes, and the deleted nodes are normalized so that the sum of the contribution values ​​of all nodes after normalization is equal to the end-to-end overall degradation amount. The contribution value of each node after normalization is the node-level degradation contribution.

[0012] In conjunction with the first aspect or its corresponding implementation, in one implementation, the step of performing component attribution based on the node-level degradation contribution to obtain component attribution results includes: Determine the component version fingerprint referenced by each node in the node alignment unit; When the node references a component version fingerprint, the node-level degradation contribution of the node is allocated to the component corresponding to the component version fingerprint; When the node references multiple component version fingerprints, the allocation weight is determined based on the counterfactual replacement result of each component, and the node-level degradation contribution of the node is allocated to the component corresponding to the component version fingerprint according to the allocation weight. The degradation contributions assigned to the same component are aggregated to obtain the degradation contribution for each component; Attribution confidence is calculated based on at least one of the following: sample reproducibility level, alignment confidence of the node alignment unit, playback result variance, evaluator consistency, and proportion of unfixed external dependencies. The attribution confidence characterizes the reliability of the component attribution results. The sample reproducibility level is determined based on the completeness of information contained in the reproducible playback sample package. The playback result variance is calculated based on the variance obtained from multiple playback results driven by the reproducible playback sample package in the first and second release versions, respectively. The evaluator consistency is the degree of consistency among multiple evaluations by the evaluator used to score the outputs of the first and second release versions. The proportion of unfixed external dependencies is the percentage of external dependencies not processed by the fixing operation. The output includes the attribution confidence and the degradation contribution of each component.

[0013] In conjunction with the first aspect or its corresponding implementation, in one implementation, generating release control instructions based on the component attribution results includes: When a security-related component degenerates in the component attribution results and the attribution confidence exceeds a preset confidence threshold, a blocking instruction is generated. When the sum of the degradation contributions of all components exceeds the preset overall degradation threshold and the attribution confidence exceeds the preset confidence threshold, a release blocking instruction or a grayscale reduction instruction is generated. The grayscale reduction instruction is used to reduce the proportion of traffic to users in the second release version. When the sum of the degradation contributions of all components exceeds the preset overall degradation threshold and the attribution confidence does not exceed the preset confidence threshold, a pause release and replay instruction is generated, or a manual review instruction is generated. When the sum of the degradation contributions of all components does not exceed the preset overall degradation threshold and there are no security-related components degrading, a release permission instruction is generated.

[0014] In conjunction with the first aspect, in one implementation, the reproducible replay sample package includes the sample list and the integrity verification information. The sample list includes a sample identifier, the execution trajectory, a set of component version fingerprints of the execution context, a set of solidified external dependencies, and version association information. The integrity verification information is used to verify the integrity of the reproducible replay sample package, and the version association information includes the identifier of the agent release version corresponding to the execution trajectory and the collection time of the execution trajectory.

[0015] In conjunction with the first aspect, in one embodiment, the method further includes: When the release status of the second release version is release blocked or grayscale ratio reduced, the reproducible replay sample package, the component attribution result, the release version information of the second release version, and the attribution component identifier corresponding to the second release version are associated and stored in the regression sample set.

[0016] This invention reconstructs the complete runtime conditions at production time by binding execution trajectories with execution contexts and generating multi-component version fingerprints. By solidifying external dependencies in the execution trajectories, it eliminates the interference of external environmental changes on replay comparisons between versions, enabling two versions to share the same reproducible replay sample package for homogeneous pairing execution in an isolated environment, ensuring the consistency of the comparison benchmark. By aligning nodes in the dual-version node trajectories and calculating node-level degradation contributions, it can accurately locate the first node of difference and distinguish between upstream propagation effects and node self-effects. Then, it aggregates node contributions according to the referenced component version fingerprints, realizing component-level degradation attribution. Finally, it automatically generates release control instructions based on the component attribution results, shortening the response time from problem discovery to blocking, degradation, or rollback.

[0017] Secondly, embodiments of the present invention provide an agent deployment control device based on playback attribution, the device comprising: The acquisition module is used to acquire the execution trajectory of the agent in the production environment and determine the execution context corresponding to the execution trajectory. The execution trajectory includes the nodes passed by the agent during execution, the dependencies between nodes, and the input and output information of the nodes. The execution context is used to characterize the component combination state and data state that the execution trajectory depends on during execution. A fingerprint generation module is used to generate a multi-component version fingerprint based on the execution context, wherein the multi-component version fingerprint is used to uniquely identify the combination of components contained in the execution context; The solidification module is used to solidify the external dependencies in the execution trajectory. The external dependencies are external states, external call results, or non-deterministic operating conditions that affect the execution result of the execution trajectory. The sample construction module is used to generate a sample list and integrity verification information based on the execution trajectory, the execution context, the multi-component version fingerprint and the solidified external dependencies, and to construct a reproducible replay sample package. The replay module is used to drive the execution of the first release version and the second release version respectively in an isolated environment using the same reproducible replay sample package to obtain the first node trajectory and the second node trajectory. An alignment module is used to align the first node trajectory and the second node trajectory to obtain a node alignment unit; The attribution module is used to calculate the node-level degradation contribution of the second release version relative to the first release version based on the node alignment unit, and to perform component attribution based on the node-level degradation contribution to obtain the component attribution result. The release control module is used to generate release control instructions based on the component attribution results, and the release control instructions are used to control the release status of the second release version.

[0018] Thirdly, embodiments of the present invention provide a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform a playback attribution-based agent release control method according to the first aspect or any corresponding embodiment described above.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions, which are used to cause a computer to execute a playback-attribution-based agent deployment control method according to the first aspect or any corresponding embodiment described above. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating an agent deployment control method based on playback attribution according to an embodiment of the present invention. Figure 2 This is a structural block diagram of an intelligent agent release control device based on playback attribution according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In related technologies, production data feedback schemes for intelligent agent systems typically collect user input, intelligent agent output, and some call information, and re-execute the same input using the current operating environment, or directly compare the final output of the production version with that of the candidate version to determine whether the candidate version has experienced performance degradation. However, these schemes have at least the following problems: they only save input and output and a few tracking fields, without saving the complete execution context, so the original operating conditions cannot be reconstructed after the production version is updated; external dependencies such as tool call results, retrieval results, and business data are constantly changing, so even if the input is the same, the replay results will differ due to external drift, making the comparison benchmark unreliable; comparing only the final output cannot locate the initial node of the difference, nor can it distinguish between upstream propagation effects and node self-effects; the evaluation results are disconnected from the release system, and manual judgment is required after degradation is detected, resulting in delays in blocking and rollback.

[0024] In view of this, embodiments of the present invention provide an agent deployment control method based on playback attribution to solve the technical problems of root cause inability to locate and the disconnect between evaluation and deployment in related technologies. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] This embodiment provides an agent deployment control method based on playback attribution. Figure 1 This is a flowchart of an agent deployment control method based on playback attribution according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Collect the execution trajectory of the agent in the production environment and determine the execution context corresponding to the execution trajectory. The execution trajectory includes the nodes passed by the agent during execution, the dependencies between nodes, and the input and output information of the nodes. The execution context is used to characterize the component combination state and data state that the execution trajectory depends on.

[0026] In this embodiment, the execution trajectory of an agent in a production environment is collected, and the execution context corresponding to the execution trajectory is determined. The execution trajectory is a sequence of nodes traversed by the agent during the execution of a production task. A node is an execution unit in the agent's workflow. These nodes are arranged in execution order and interconnected through dependencies. Each node corresponds to a specific processing step, which can be a model invocation step, a tool invocation step, a retrieval step, or a policy determination step. During execution, nodes receive input data and generate output data, which are recorded in the execution trajectory as input / output information. Dependencies are the association methods between nodes, used to characterize whether the execution of one node depends on the output or completion status of another node. Dependencies define the execution order and data flow between nodes.

[0027] A single production execution by an intelligent agent relies on the coordination of multiple components, including prompts, workflows, models, knowledge base indexes, tools, and permission policies. The versions and configurations of these components collectively determine the agent's behavior and output at that moment. The component composition state records the version information of each component during the execution trajectory, used to reconstruct the agent's configuration environment at the time of execution. The data state is used to obtain business data consistent with that execution moment during subsequent playback.

[0028] In the specific implementation, execution events of the agent in the production environment are captured asynchronously. The collected execution events are deduplicated and sorted, parent-child and causal relationships are filled in, missing intervals in the trajectory are identified and marked, and standardized execution trajectories are generated. Each node in the execution trajectory is assigned a node identifier upon generation to uniquely identify the node in the agent's workflow. The node type, start time, end time, execution status, and summaries of the node's input and output data are also recorded. For streaming output nodes, the generation time of the first output unit and the streaming output completion time, as well as a summary of the aggregated content of the streaming output, are also recorded. For tool call nodes and retrieval nodes, a normalized request summary, external dependency identifier, call sequence number, response summary, and exception information are also recorded. Dependencies between nodes are implemented by recording the correspondence between parent and child node identifiers. Each node records the parent node identifier it depends on, thus forming a complete node dependency topology.

[0029] In some embodiments of the present invention, the execution context includes at least one of the following: tenant information, application information, agent information, release version information, prompt word version fingerprint, workflow version fingerprint, model version fingerprint, knowledge base index version fingerprint, tool version fingerprint, permission policy version fingerprint, and data watermark. Among them, tenant information is used to identify the tenant domain to which the execution trajectory belongs; application information is used to identify the application to which the agent belongs; agent information is used to identify the agent instance corresponding to the execution trajectory; release version information is used to identify the release version of the agent corresponding to the execution trajectory; prompt word version fingerprint is used to identify the version of the prompt word used by the agent; workflow version fingerprint is used to identify the version of the workflow used by the agent; model version fingerprint is used to identify the version of the model used by the agent; knowledge base index version fingerprint is used to identify the version of the knowledge base index used by the agent; tool version fingerprint is used to identify the version of the tool used by the agent; permission policy version fingerprint is used to identify the version of the permission policy used by the agent; and data watermark is used to identify the business data status during the execution of the execution trajectory.

[0030] In this embodiment, tenant information, application information, agent information, and release version information are used to uniquely locate the execution trajectory and isolate the tenant. The prompt word version fingerprint, workflow version fingerprint, model version fingerprint, knowledge base index version fingerprint, tool version fingerprint, and permission policy version fingerprint are used to identify the specific versions of each component used by the agent in this execution. The data watermark is used to indicate the business data status during the execution of the trajectory, ensuring that business data consistent with the execution time can be obtained during subsequent replays.

[0031] Tenant information identifies the tenant domain to which the execution trajectory belongs. In a multi-tenant agent platform, data and configurations of different tenants are isolated from each other, and tenant information ensures that the attribution of the execution trajectory is clear. Application information identifies the application to which the agent belongs; an application can contain multiple agent instances, and application information is used to associate the execution trajectory with a specific application; agent information can be an agent instance identifier or an application identifier, used to uniquely determine which agent generated the execution trajectory; release version information records the agent version number corresponding to when the execution trajectory was generated, used to associate it with the actual release version in the production environment.

[0032] **Prompt words:** These are the instruction templates or system prompts used by the agent during execution. The prompt word version fingerprint is generated based on the prompt word content and its configuration parameters, uniquely identifying the prompt word version. **Workflow:** This is the node topology and flow logic followed by the agent during execution. The workflow version fingerprint is generated based on the workflow topology definition and its node configuration, uniquely identifying the workflow version. **Model:** This is the basic language model or specialized model invoked by the agent during execution. The model version fingerprint is generated based on the model identifier, model parameters, and inference configuration, uniquely identifying the model version. **Knowledge base index:** This is the vector index or full-text index that the agent relies on when performing retrieval enhancement generation. The knowledge base index version fingerprint is generated based on the index identifier, index snapshot version, and retrieval configuration, uniquely identifying the knowledge base index version. **Tools:** These are the external functional interfaces that the agent can invoke during execution. The tool version fingerprint is generated based on the tool identifier, tool interface definition, and tool configuration, uniquely identifying the tool version. **Permission policy:** This is the set of rules that the agent uses when determining permissions. The permission policy version fingerprint is generated based on the policy identifier, policy rule set, and policy scope, uniquely identifying the permission policy version. Data watermarks can be timestamps, transaction identifiers, or index commit numbers, used to record the point in time when business data is consistent during the execution of the execution trajectory.

[0033] In the specific implementation, various information in the execution context are obtained by querying the configuration registry, artifact repository, and release system. Once the execution trajectory is collected, the component version information valid at the corresponding time point is read from the dual-temporal configuration repository based on the release token in the execution trajectory. Tenant information, application information, agent information, and release version information serve as the basic attribution information for the execution trajectory and are extracted from the request context of the production environment during execution trajectory collection. After determining the component version, the component version fingerprint is calculated by analyzing the component's stability identifier, semantic version, normalized configuration hash, content hash, and construct digest. Then, the component fingerprints of all components are concatenated in a predetermined order, and their hash values ​​are calculated to obtain the multi-component version fingerprint.

[0034] In an intelligent customer service scenario, an execution process of the intelligent agent involves four nodes: intent recognition, permission determination, knowledge retrieval, and answer generation. The data acquisition module captures the execution events of these four nodes, recording that the input to the intent recognition node is the user's query for refund rules, and the output is the intent classification result "after-sales refund"; the input to the permission determination node is the user's identity information and the intent classification result, and the output is the permission determination result "allow"; the input to the knowledge retrieval node is the refund rule query statement, and the output is the retrieved document identifier and document fragment; and the input to the answer generation node is the retrieved document fragment and generated prompt words, and the output is the final reply content. The dependencies between the four nodes are as follows: the intent recognition node and the permission determination node have no pre-dependencies; the knowledge retrieval node depends on the output of the permission determination node; and the answer generation node depends on the output of the knowledge retrieval node. The nodes, their dependencies, and the node input / output information are used as the execution trajectory. The tenant information corresponding to the execution trajectory is a specific enterprise tenant identifier, the application information is an intelligent customer service application identifier, the intelligent agent information is a refund-specific intelligent agent instance, and the release version information is production version R100. The prompt word version fingerprint corresponds to prompt word P20, the workflow version fingerprint corresponds to workflow W31, the model version fingerprint corresponds to model M7, the knowledge base index version fingerprint corresponds to index I55, the tool version fingerprint corresponds to tool T12, the permission policy version fingerprint corresponds to policy A9, and the data water level identifier records the consistency snapshot number of the business database at the time of execution.

[0035] By including tenant information, application information, agent information, and release version information in the execution context, the attribution and version source of the execution trajectory are fully recorded, facilitating subsequent tenant isolation and version tracing of samples. By including version fingerprints of each component in the execution context, the complete component combination state upon which the execution trajectory depends is uniquely represented, providing accurate component version information for subsequent reproducible replay and version comparison. By including data watermarks in the execution context, the point in time when business data is consistent during the execution of the trajectory is recorded, providing a basis for obtaining business data consistent with the execution time during subsequent replay.

[0036] Step S102: Generate a multi-component version fingerprint based on the execution context. The multi-component version fingerprint is used to uniquely identify the combination of components contained in the execution context.

[0037] In this embodiment of the application, a multi-component version fingerprint is generated based on the execution context. The multi-component version fingerprint is used to uniquely represent the combination of components contained in the execution context.

[0038] A multi-component version fingerprint is a comprehensive representation of the version information of all components in the execution context. It is used to uniquely identify the complete component combination corresponding to an execution in subsequent replay and comparison processes. The multi-component version fingerprint consolidates the scattered version information of multiple components into a unified identifier, making the execution context easily referenced and verified.

[0039] In some embodiments of the present invention, step S102 specifically includes the following sub-steps: For each component in the execution context, a corresponding component fingerprint is generated based on the component's identification information, configuration information, and version information; The component fingerprints of all components are concatenated in a predetermined order, and their hash values ​​are calculated to obtain the multi-component version fingerprint.

[0040] Among them, the component fingerprint is the version identifier of a single component in the execution context, generated based on the component's identification information, configuration information, and version information. The component fingerprint is used to uniquely represent the state of the component at a specific time. The multi-component version fingerprint is a comprehensive identifier generated by combining the component fingerprints of all components in the execution context in a predetermined order, used to uniquely represent all component combinations contained in the execution context.

[0041] In the specific implementation, firstly, for each component in the execution context, a corresponding component fingerprint is generated based on the component's identification information, configuration information, and version information. The component's identification information is used to uniquely identify the component's identity, the component's configuration information records the specific parameter settings of the component in this execution, and the component's version information records the component's version number or revision number. As described in the previous steps, for the prompt word component, a prompt word version fingerprint is generated based on the prompt word content and its configuration parameters; for the workflow component, a workflow version fingerprint is generated based on the workflow topology definition and its node configuration; for the model component, a model version fingerprint is generated based on the model identifier, model parameters, and inference configuration; for the knowledge base index component, a knowledge base index version fingerprint is generated based on the index identifier, index snapshot version, and retrieval configuration; for the tool component, a tool version fingerprint is generated based on the tool identifier, tool interface definition, and tool configuration; for the permission policy component, a permission policy version fingerprint is generated based on the policy identifier, policy rule set, and policy scope. The generation of each component fingerprint uses normalized encoding and hash calculation to ensure that components in the same state generate the same fingerprint value. Then, the component fingerprints of all components are concatenated into a complete string in a predetermined order. A hash value is calculated on this concatenated string to obtain the multi-component version fingerprint. The concatenation order can be a fixed order of tenant information, application information, agent information, release version information, prompt word version fingerprint, workflow version fingerprint, model version fingerprint, knowledge base index version fingerprint, tool version fingerprint, permission policy version fingerprint, and data watermark. The hash calculation uses a cryptographic hash function, such as the SHA-256 algorithm, ensuring that the probability of different component combinations producing different fingerprint values ​​is extremely low.

[0042] Continuing with the aforementioned intelligent customer service scenario example, the execution context includes prompt word version P20, workflow version W31, model version M7, knowledge base index version I55, tool version T12, and permission policy version A9. Component fingerprints are generated for each of these six components. These fingerprints are then concatenated in a predetermined order: prompt word version fingerprint, workflow version fingerprint, model version fingerprint, knowledge base index version fingerprint, tool version fingerprint, and permission policy version fingerprint. The concatenated result is then hashed using SHA-256 to obtain a fixed-length hash string. This string serves as the multi-component version fingerprint for the current execution context. When any component version in the execution context changes, the multi-component version fingerprint changes accordingly, uniquely identifying the new component combination.

[0043] By generating multi-component version fingerprints based on the execution context, all component combinations contained in the execution context are uniquely represented by a unified identifier. When constructing a reproducible replay sample package, only this fingerprint needs to be carried to completely record the component combination state at the execution time. At the same time, during the sample package verification and version comparison process, the multi-component version fingerprints can be compared to quickly determine whether the component combinations depended on by the two executions are consistent, providing a reliable version comparison basis for subsequent dual-version differential replay.

[0044] Step S103: Solidify the external dependencies in the execution trajectory. External dependencies are external states, external call results, or non-deterministic operating conditions that affect the execution result of the execution trajectory.

[0045] In this embodiment, the external dependencies in the execution trajectory are solidified. The external dependencies are external states, external call results, or non-deterministic operating conditions that affect the execution result of the execution trajectory.

[0046] The execution trajectory relies on various external factors to complete node operations during execution. These factors include the data content stored in external data sources, the call results returned by external service interfaces, and time information and random sequences generated during execution. If these factors are retrieved from external sources during replay, the replay results will differ from the original execution results because the external data may have been updated and the return results of external services may have changed. Therefore, it is necessary to record and save the external states, external call results, and non-deterministic operating conditions that affect the execution trajectory's outcome in a static form, so that subsequent replays can obtain the same external dependency return results as at the original execution moment.

[0047] The external state refers to the data content read from external data sources during the execution process. This can include document content retrieved from the knowledge base, business records in the business database, key-value data in the cache system, and document identifiers, document fragments, and ranking scores in the retrieval results. The external service call result is the return content received after the execution trajectory calls external service interfaces during execution. This can include response data returned by tool interfaces, results returned by application programming interface calls, and inference results from external model services. Nondeterministic operating conditions are factors that introduce randomness or time-dependent logic into the execution trajectory during execution. These can include system time during execution, temporal relationships between nodes, the initial seed value of the random number generator, and sampling parameters and random seeds during model inference. Staticization involves recording and storing these factors in a persistent form, so that this external dependency no longer depends on the external environment during subsequent replays, but can be directly reproduced using the stored static data.

[0048] In the specific implementation, during the execution of the execution trajectory, access events of each node to external dependencies are asynchronously captured. For external state-type dependencies, the external data content read during the execution of the execution trajectory is recorded, including the complete document content and ranking score in the knowledge base retrieval results, and field values ​​of business data. For external call results, the content of the call request sent to the target service and the content of the call response returned by the target service are recorded. For non-deterministic runtime condition-type dependencies, the logical clock value of each node during the execution of the execution trajectory is recorded. The logical clock value is used to restore the temporal relationship and relative order during the execution of the execution trajectory during replay. The random number seed value used during the execution of the execution trajectory is recorded. This seed value is used to ensure that random operations generate the same random sequence as the original execution during replay. The system timestamp or transaction identifier during the execution of the execution trajectory is recorded. This identifier is used to obtain business data consistent with the execution time of the execution trajectory from the data source during replay.

[0049] Each external dependency record is associated with a corresponding node in the execution trajectory, establishing a mapping relationship between the node and the call identifier. External dependencies are categorized by dependency type during recording. For external state-type dependencies, the read data content and reading conditions are saved, with the reading conditions used to confirm the data acquisition method during replay. For service call-type dependencies, the call request and call response are saved, with the call request used to identify the corresponding call event during replay, and the call response used to directly return data to the calling node during replay. For non-deterministic runtime condition-type dependencies, the logical clock value and random number seed value used for each node during execution are recorded. After recording, external dependencies are stored as a static data set in a solidified form. This static data set is bound to the execution trajectory, ensuring consistent dependency return results can be obtained during replay without accessing the external environment. For dependencies referencing external state-type dependencies, a data level identifier at the execution time of the execution trajectory is recorded during solidification. This data level identifier is used to obtain business data consistent with the execution time of the execution trajectory from the data source during replay.

[0050] For example, in a smart customer service scenario, the agent's knowledge retrieval node sends a query to the knowledge base and retrieves the search results, which include document identifiers and document fragments. The solidification operation completely records and stores the query conditions, retrieved document content, and sorting results for this retrieval. During subsequent playback, the knowledge retrieval node no longer initiates a real-time search to the knowledge base; instead, it directly reads the search results from the solidified external dependencies and returns them. Even if the knowledge base index has been updated, the playback results remain consistent with the original execution. Similarly, for tool calls, the solidification operation records the tool call request and the response data returned by the tool. During playback, the stored response data is used directly, avoiding interference from changes in the tool interface state on the playback results. Non-deterministic runtime condition dependencies will be explained in detail in the examples corresponding to the subsequent sub-steps.

[0051] By solidifying the external dependencies in the execution trajectory, the external environment state on which the execution trajectory depends is completely preserved. Subsequent playback no longer requires real-time access to external data sources or external services, eliminating the interference caused by changes in the external environment on the playback results, and ensuring the reproducibility of the playback process and the reliability of the comparison benchmark.

[0052] In some embodiments of the present invention, the external dependency includes at least one of the following: tool call result, retrieval recall result, business data, time information, and random number seed. Step S103 may specifically include the following sub-steps: For tool call results, store the execution trajectory of the call request to the target tool and the call response returned by the target tool during execution; For the retrieval and recall results, the retrieval query conditions, recalled document content, and sorting results during the execution of the fixed execution trajectory are defined. For business data, create a read-only data snapshot of the business data at the execution time of the execution trajectory, or record a consistency identifier for the business data. The consistency identifier is used to obtain business data consistent with the execution time of the execution trajectory from the data source when driving the execution of the first or second release version. For time information, the logical clock is recorded during the execution of the execution trajectory. The logical clock is used to restore the timing relationship of the execution trajectory when driving the first or second release version. For the random number seed, the random number seed used during the execution of the record execution trajectory is used to ensure that random operations produce the same random sequence when driving the execution of the first and second release versions.

[0053] In the embodiments of this application, different types of external dependencies are processed using different solidification methods when solidifying external dependencies in the execution trajectory.

[0054] For tool call results, the system stores the call requests to the target tool and the call responses returned by the target tool during the execution of the execution trajectory. Tool call results are the returned data received by the agent after calling an external tool interface during execution, such as weather information returned by a weather query tool or query results returned by a database query tool. Once the tool call results are recorded, during subsequent replays, when a node initiates the same tool call again, the stored call response is directly returned without actually calling the external tool interface. In the specific implementation, the tool call node in the execution trajectory generates a call event during execution. This event records the target tool's identifier, call request parameters, and call response content. After capturing this call event, the call request and call response are associated and stored in the external dependency set. Each tool call result is mapped to the corresponding call event in the execution trajectory through a call identifier, which can contain a combination of node identifier, dependency identifier, and call sequence number. In the replay environment, when a node performs a tool call operation, the corresponding tool call result is searched based on the call identifier. If a match is found, the call response is directly returned to the node, ensuring that the node receives the same call return result as the original execution during replay.

[0055] For retrieval recall results, the retrieval query conditions, recalled document content, and ranking results are fixed during the execution of the execution trajectory. The retrieval recall results are a list of relevant documents retrieved from the knowledge base or retrieval engine by the agent during knowledge retrieval operations. Each document contains a document identifier, document fragment content, and a relevance score. The retrieval recall results depend on the version of the knowledge base index and retrieval configuration parameters during the original execution. As the knowledge base content is updated, the same query conditions may return different document sets at different times. In the specific implementation, the retrieval node in the execution trajectory generates a retrieval event during execution. This event records the retrieval query conditions, the list of recalled document identifiers, the content fragments of each document, and the ranking results. After capturing this retrieval event, the above information is completely recorded and solidified into the external dependency set, while also recording the node identifier corresponding to the retrieval node. In the replay environment, when a node performs a retrieval operation, the replay engine no longer initiates a real-time retrieval request to the knowledge base. Instead, it directly reads the recalled document content and ranking results corresponding to the query conditions from the solidified retrieval recall results and returns them to the node. This ensures that the retrieval results during replay are consistent with the original execution time, eliminating the impact of knowledge base index updates on the replay results.

[0056] Business data refers to data records from business systems referenced by the intelligent agent during execution, such as order information, user data, and product inventory. Business data may change after the execution trajectory is completed. To ensure that subsequent replays obtain business data consistent with the execution trajectory's execution time, snapshots or consistency identifiers are needed to fix the business data state. A read-only data snapshot is a read-only copy of the business data created at the execution trajectory's execution time. This copy records a consistent view of all business data at that moment. During replay, the replay engine directly reads business data from the read-only data snapshot without accessing the real-time business database in the production environment. A consistency identifier is a consistency marker for the business data at the execution trajectory's execution time. It can be a timestamp, a transaction identifier, or a log sequence number. This identifier is used to retrieve business data consistent with the execution trajectory's execution time from the data source during replay. In specific implementations, for data sources that support querying by time point, the system timestamp or transaction identifier at the time of execution trajectory execution is recorded. During replay, the data source returns the data view at the corresponding time based on this identifier. For data sources that do not support querying by time point, the creation of a read-only data snapshot is triggered during the execution trajectory's execution. The snapshot is stored in a separate storage location, and a snapshot identifier is recorded. Read-only data snapshots and consistency identifiers can be used interchangeably. When using read-only data snapshots, the snapshot data is read directly during playback. When using consistency identifiers, the identifier is used to query the data from the data source at the corresponding time point during playback.

[0057] Time information refers to the time-related external dependencies during the execution of the execution trajectory, including the start time, end time, execution duration, and temporal relationships between nodes. During replay, if the actual system clock at the time of replay is used directly, the behavior of nodes dependent on time logic may deviate due to the difference between the replay time and the original execution time. For example, decision branches based on time conditions may take different paths. A logical clock is a time stamp independent of the actual system clock, used to reconstruct the temporal relationships and relative order of the execution trajectory during replay. In the specific implementation, the logical clock value of each node is recorded during execution trajectory acquisition. The logical clock value can be an incrementing sequence number or a logical timestamp, and the logical clock value of the same node remains consistent across different execution versions. The logical clock is stored in an external dependency set in association with node events. In the replay environment, the logical clock value is injected during node execution. The node reads this logical clock value as the time reference, rather than reading the actual system clock of the replay environment. By recording the logical clock and injecting it during playback, the first and second release versions obtain the same timing reference during execution. The timing-related node behaviors remain consistent in the dual-version playback and will not differ due to different playback times.

[0058] The random number seed is the initial value of the random number generator; the same random number seed produces the same random number sequence. During agent execution, random operations may be involved, such as the randomness of model sampling, data shuffling, and random sampling. Different execution times produce different random sequences, potentially leading to different outputs for the same input. During replay, if the random number seed is not recorded, the two replay versions will each generate different random sequences, and the output difference may stem from the randomness itself rather than version changes. In the specific implementation, the initial seed value of the random number generator is captured and recorded in the external dependency set during execution trajectory acquisition. For model inference services that support deterministic sampling, the sampling parameters and random seed during model inference are recorded. For random operation nodes in the workflow, the random seed used by that node is recorded in the execution trajectory. In the replay environment, the first and second release versions initialize the random number generator with the same random number seed value when starting replay, ensuring that random operations produce the same random sequence in both versions. When multiple paired playbacks are required, each playback can use the same seed value to obtain a consistent random sequence, or different seed values ​​can be used to perform multiple sets of playbacks to assess the range of influence of randomness on the results.

[0059] For example, in a smart customer service scenario, when the agent's knowledge retrieval node executes, it retrieves documents D8v3 and D12v2 from the knowledge base. The solidification operation records the retrieval query conditions, the complete content and ranking score of document D8v3, and the complete content and ranking score of document D12v2 to the external dependency set. When the agent's work order tool invocation node executes, it invokes the work order creation tool. The solidification operation records the work order parameters in the invocation request and the work order number returned by the tool. Order data at the execution time of the business system's execution trajectory is saved using a read-only data snapshot method, and the snapshot identifier is recorded in the external dependency set. The execution order of nodes is recorded using logical clock values, with the logical clock values ​​for each node being 1, 2, 3, and 4, respectively. The initial seed value of the random number generator, 12345, is recorded in the external dependency set. During subsequent replay, the retrieval node directly reads documents D8v3 and D12v2 from the solidified retrieval results, without querying the real-time knowledge base; the tool call node directly returns the stored work order number, without actually calling the work order interface; business data is read from the read-only data snapshot; each node restores the execution order according to the logical clock value; random operations use the seed value 12345 to generate the same random sequence as the original execution.

[0060] By using corresponding solidification methods to handle different types of external dependencies, the external data and service call results that the execution trajectory depends on are completely preserved. In subsequent replays, the two versions share the same solidified external dependencies, eliminating the interference of external environment changes on replay comparisons and ensuring the reproducibility of replay results and the fairness of the two-version comparison.

[0061] Step S104: Generate a sample list and integrity verification information based on the execution trajectory, execution context, multi-component version fingerprint, and solidified external dependencies, and construct a reproducible replay sample package.

[0062] In this embodiment, a sample list and integrity verification information are generated based on the execution trajectory, execution context, multi-component version fingerprint, and solidified external dependencies to construct a reproducible replay sample package.

[0063] The execution trajectory records the nodes traversed by the agent during execution, the dependencies between nodes, and the input and output information of each node. It serves as the original input source driving the version execution during replay. The execution context records the component composition state and data state that the execution trajectory depends on, providing the basis for reconstructing the component environment during replay. The multi-component version fingerprint uniquely identifies all component compositions contained in the execution context, serving as an identifier for subsequent verification of component composition consistency. The solidified external dependencies record the external data and external service call results referenced during the execution trajectory, forming the data foundation for isolating external environment drift during replay.

[0064] The above four pieces of information need to be integrated into a verifiable and traceable data object. This data object uses the sample list as an index and integrity verification information as a tamper-proof guarantee to form a reproducible and replayable sample package.

[0065] The sample manifest serves as an index and directory for the reproducible replayable sample package. It records the package's version information, sample identifier, and a list of references to all objects contained within the package, including references to execution traces, a set of component version fingerprints for the execution context, references to the solidified set of external dependencies, and version association information. The sample manifest is the core metadata of the reproducible replayable sample package, enabling rapid retrieval, referencing, and verification of its content. Integrity verification information, used to verify the integrity of the sample manifest, includes an integrity hash value and a digital signature. The integrity hash value is a hash calculated from the sample manifest content, and the digital signature is a signature obtained by signing the integrity hash value. Together, the integrity verification information and the sample manifest constitute the core structure of the reproducible replayable sample package.

[0066] In the specific implementation, the construction of the reproducible replay sample package is performed as follows: First, a sample manifest file is created as the index and directory of the sample package. The sample manifest file records the version information of the sample package, the sample identifier, and a list of references to all objects contained in the sample package. The execution trajectory is stored as an independent object in the content-addressed storage, and the sample manifest records a reference to this object. This object contains a complete sequence of nodes, dependencies between nodes, and input / output information for each node. The execution context is stored as an independent object in the content-addressed storage, and the sample manifest records a reference to this object. This object contains tenant information, application information, agent information, release version information, data watermark, and version fingerprint information for each component. The version fingerprint of multiple components is directly recorded in the sample manifest file as a metadata field of the sample manifest and is associated with the execution context object. The solidified external dependency set is stored as an independent object in the content-addressed storage, and the sample manifest records a reference to this object. This object contains a list of tool call results, a list of retrieval recall results, a reference to business data snapshots or a consistency identifier, a logical clock value, and a random number seed value.

[0067] For large objects in the sample package, such as search results containing numerous document fragments or tool call results containing complete request and response content, content-addressed storage is used. Content-addressed storage determines the object's storage address based on the hash value of its content, and multiple objects with the same content share the same storage copy. The sample package stores the object's storage address reference, the object's content hash value, and the retrieval strategy, not the object itself. When constructing the sample package, a hash value is calculated for each object to be stored, and the object is stored in the content-addressed object library based on the hash value. The object type, storage address, and content hash value are recorded in the sample manifest. For sensitive data fields, encryption is performed using the tenant key before storage, and the encrypted data is stored in the content-addressed object library. Encryption metadata, including encryption algorithm identifiers and key reference information, is recorded in the sample manifest, enabling the playback environment to decrypt and read the sample content when it has the appropriate permissions.

[0068] When constructing reproducible replayable sample packages, external dependencies can be handled in three ways: fully frozen, reference snapshot, and deterministic simulation. The fully frozen method stores the complete data of the external dependency directly in content-addressed storage, recording a reference to the object in the sample manifest. The reference snapshot method only stores a snapshot reference identifier of the external data in the external data source; this identifier is used to read data from the external data source during playback. The deterministic simulation method generates deterministic simulated data based on the characteristics of the external dependency, using the simulated data to replace the real external dependency during playback.

[0069] After all objects are stored, the integrity hash value of the sample manifest file is calculated, and a digital signature algorithm is used to sign the integrity hash value to generate an integrity signature. The integrity hash value and integrity signature together constitute the integrity verification information. The sample manifest and integrity verification information are packaged to form a reproducible replayable sample package. Before the reproducible replayable sample package is written to the sample repository, integrity verification is performed, and only after the verification passes can it be added to the repository.

[0070] Continuing with the aforementioned intelligent customer service scenario, the execution context includes tenant information, application information, agent information, release version information R100, prompt word version fingerprint P20, workflow version fingerprint W31, model version fingerprint M7, knowledge base index version fingerprint I55, tool version fingerprint T12, permission policy version fingerprint A9, and data watermark. The solidified external dependencies include retrieval results, tool call results, logical clock, and random number seed. A sample list is created, identified as SP-9001. The execution trajectory object, execution context object, and solidified external dependency set are stored in the content addressing object library, with their respective storage addresses and content hash values ​​recorded in the sample list. Multi-component version fingerprints are directly recorded in the metadata fields of the sample list. After SHA-256 hash calculation and digital signature, the integrity hash value and integrity signature are used as integrity verification information. The sample list and integrity verification information are packaged to obtain the reproducible replayable sample package SP-9001, which is written to the sample repository for subsequent replay use.

[0071] By constructing a reproducible replay sample package, the execution trajectory, execution context, multi-component version fingerprints, and solidified external dependencies are integrated into an independent, complete, and self-contained data set. This sample package no longer depends on the real-time state of the production environment and can be used to drive version execution in any isolated environment. It provides a complete, consistent, and verifiable input foundation for subsequent same-source pairing replays, ensuring the reproducibility of the replay process and the fairness of the dual-version comparison.

[0072] Step S105: In an isolated environment, the first release version and the second release version are executed using the same reproducible replay sample package to obtain the first node trajectory and the second node trajectory.

[0073] In this embodiment of the application, in an isolated environment, the same reproducible replay sample package is used to drive the execution of the first release version and the second release version respectively, so as to obtain the first node trajectory and the second node trajectory.

[0074] To fairly compare the performance differences between the two releases, both versions need to be executed separately under the same input conditions and environmental constraints, and the execution process should be recorded as node trajectories. The first and second releases share the same reproducible replay sample package to ensure that both versions receive exactly the same inputs, component version fingerprints, and fixed external dependencies.

[0075] The isolated environment is an independent execution space isolated from the production environment, used to host the execution process of the release version. Resource quotas, network configurations, storage access, and permission policies in the isolated environment are isolated from the production environment; any operations during replay will not affect the normal operation of the production environment. The first release version is the agent version already deployed and running in the production environment, serving as the benchmark for version comparison. The second release version is the candidate agent version to be evaluated, serving as the target object for version comparison. Driven execution is the process of taking a reproducible replay sample package as input and guiding the release version to execute along the original execution trajectory path according to the execution context recorded in the sample package and the fixed external dependencies. The first node trajectory is a complete sequence recording all node execution information during the execution process of the first release version. The second node trajectory is a complete sequence recording all node execution information during the execution process of the second release version.

[0076] In the implementation, two execution containers with equivalent resource quotas are first created. These containers have the same CPU quota, GPU quota, memory size, timeout time, and concurrency limits. The same solidified external dependency proxy is injected into both containers. This proxy intercepts node access requests to external dependencies during replay and returns the stored solidified external dependency data. The same logical clock and random number seed are injected into both containers, ensuring consistent timing references and the same random sequences for both versions during replay. The same identity and permission configurations and the same evaluator configurations are injected into both containers, ensuring consistent execution permissions and subsequent scoring conditions for both versions. After the isolation environment is built, the reproducible replay sample package is loaded into each of the two containers.

[0077] The execution trajectory is read from the reproducible replay sample package as the driving input, and each node is driven to execute sequentially according to the execution order and dependencies recorded in the execution trajectory. When driving each node, the component version fingerprint of the corresponding node in the execution context is read, and the corresponding component version is loaded based on the component version fingerprint. When a node initiates an external dependency access request during execution, the external dependency proxy intercepts the request, searches for the corresponding record in the solidified external dependency set based on the request's matching key, and directly returns the response data from the record if a match is found; otherwise, the event is marked and the unexplained difference is recorded. When a node involves time-related operations, the node reads the injected logical clock value as the time basis, rather than reading the actual system clock of the execution container. When a node involves random operations, the node initializes the random number generator using the injected random number seed, ensuring that random operations produce the same random sequence. The execution processes of the two containers are independent and do not interfere with each other, each generating a complete node execution record.

[0078] During the execution of the first release version, input information, output information, execution status, start time, end time, and exception information for each node are collected and organized into the first node trajectory according to the execution order. During the execution of the second release version, execution information for each node is collected in the same way and organized into the second node trajectory according to the execution order. Both node trajectories include node identifier, node type, dependencies between nodes, input summary, output summary, and execution status, maintaining the same data structure as the original execution trajectory.

[0079] Continuing with the aforementioned intelligent customer service scenario, two isolated containers with equivalent resource quotas are first constructed. The first release version R100 and the second release version R101 are deployed in the two containers respectively. The reproducible replay sample package SP-9001 is simultaneously loaded into both containers, sharing the same solidified external dependencies, the same logical clock value, and the same random number seed value. After executing version R100, the complete node trajectory of version R100 is obtained, including all execution information for the intent recognition node, permission determination node, knowledge retrieval node, and answer generation node. After executing version R101, the complete node trajectory of version R101 is obtained. The overall score for version R100 is 0.94, and the overall score for version R101 is 0.71. By comparing the execution results of the two versions under the same isolated environment and the same input conditions, the performance change of version R101 relative to version R100 can be accurately evaluated.

[0080] By using the same reproducible replay sample package to drive the execution of the first and second release versions in an isolated environment, with both versions executing under identical input conditions, external dependency states, time references, and random sequences, the interference of execution environment differences on version comparison is eliminated. This ensures that the difference between the two node trajectories originates solely from changes in the release version itself, providing a fair and reliable trajectory data foundation for subsequent node alignment and degradation attribution.

[0081] Step S106: Align the first node trajectory and the second node trajectory to obtain the node alignment unit.

[0082] In this embodiment of the invention, since the workflow topology of the first release version and the second release version may be different, for example, nodes have been added, deleted, split or merged in the workflow, the number of nodes, the order of nodes and the dependencies between nodes in the two trajectories may not be completely consistent. Therefore, node alignment is performed on the first node trajectory and the second node trajectory.

[0083] The first node trajectory and the second node trajectory are the node execution records generated by the first and second release versions respectively after being driven by the same reproducible replay sample package. Node alignment is the process of establishing node correspondences between the two node trajectories. It is used to determine which nodes in the first node trajectory correspond to the same functional step as which nodes in the second node trajectory, and which nodes are newly added and which nodes are deleted. The node alignment unit is the data structure produced by the node alignment operation, used to record the complete result of the node correspondence between the two node trajectories.

[0084] In some embodiments of the present invention, step S106 specifically includes the following sub-steps: The node identifier is used to match the nodes in the first node trajectory and the nodes in the second node trajectory. The node identifier is used to uniquely identify a node in the agent's workflow. When a node cannot be matched by its identifier, a semantic signature is generated based on the node's attribute information, and the unmatched node is matched based on the semantic signature. The attribute information includes at least one of the following: the node's position information in the workflow, the node's input and output feature information, and the information of the components referenced by the node. Node alignment units are generated based on the matching results, and the alignment confidence of each matching node pair is determined. The node alignment units include matching node pairs, newly added nodes, and deleted nodes.

[0085] In this context, a matching node pair consists of a node from a first node trajectory and a node from a second node trajectory, with the two nodes in the pair considered to correspond to the same functional step. Attribute information is a set of information describing the characteristics of a node, including at least one of the following: the node's position in the workflow, the node's input / output characteristics, and information about the components referenced by the node. Location information records the relative position of a node in the workflow topology, including a list of predecessor and successor nodes. This location information helps identify the functional identity of a node by leveraging its context. Input / output feature information records the pattern structure of the node's input and output data, including the field names, types, and number of fields in the input data, as well as the field names, types, and number of fields in the output data. This input / output feature information helps identify the functional identity of a node by leveraging its data flow characteristics. Information about components referenced by a node records the version information of components referenced by the node during execution, including prompt word components, model components, tool components, or knowledge base index components. This information helps identify the functional identity of a node by leveraging its dependent components. Semantic signature is a feature summary generated according to predetermined rules based on the above attribute information, used to characterize the functional semantics of a node. Alignment confidence is a numerical indicator characterizing the reliability of the correspondence between two nodes in a matching node pair. A higher confidence score indicates a higher degree of trustworthiness in the two nodes corresponding to the same functional step.

[0086] In the specific implementation, node identifier-based matching is the highest priority matching method because node identifiers are the most direct and reliable basis for determining node correspondences. Node identifiers are extracted from all nodes in the first node trajectory to construct a first node identifier set; node identifiers are extracted from all nodes in the second node trajectory to construct a second node identifier set. The intersection of the two sets is performed. When a node identifier in the first node trajectory is identical to a node identifier in the second node trajectory, and this node identifier is unique in both trajectories, the corresponding two nodes constitute a matching node pair, and the alignment confidence of this matching node pair is high confidence.

[0087] Then, when a match cannot be made using the node identifier, a semantic signature is generated based on the node's attribute information, and the unmatched node is matched based on the semantic signature.

[0088] In the specific implementation, unmatched nodes refer to nodes that are not matched in the first node trajectory and the second node trajectory after the node identifier-based matching step. For each unmatched node in the first node trajectory, its location information, input / output feature information, and information about the components referenced by the node are extracted, concatenated according to predetermined rules, and a hash value is calculated to generate the semantic signature of the node. For each unmatched node in the second node trajectory, a semantic signature is generated in the same way. Then, the semantic signatures of the unmatched nodes in the first node trajectory and the semantic signatures of the unmatched nodes in the second node trajectory are compared. When the semantic signatures of the two nodes are the same or the semantic similarity exceeds a preset threshold, the corresponding two nodes constitute a matching node pair. The alignment confidence of the matching node pair is determined based on the similarity of the semantic signatures; the higher the similarity, the higher the alignment confidence. After the semantic signature matching is completed, nodes that are still unmatched in the first node trajectory are marked as deleted nodes, and nodes that are still unmatched in the second node trajectory are marked as added nodes. That is to say, added nodes are nodes that exist in the second node trajectory but cannot find a corresponding node in the first node trajectory, representing newly introduced functional steps in the candidate release version. A deleted node is a node that exists in the first node trajectory but cannot be found in the second node trajectory, representing a feature step that is removed in the candidate release version.

[0089] Finally, node alignment units are generated based on the matching results, and the alignment confidence of each matching node pair is determined.

[0090] The matching results include all matched node pairs and the alignment confidence of each matched node pair, as well as all newly added nodes and all deleted nodes. The node alignment unit is a data unit generated based on the matching results, used to record the complete result of the node correspondence between two node trajectories. The node alignment unit includes a list of matched node pairs, a list of newly added nodes, and a list of deleted nodes. Each item in the list of matched node pairs records the node identifier in the first node trajectory, the node identifier in the second node trajectory, the matching method, and the matching confidence. The matching method includes two methods: node identifier-based matching and semantic signature-based matching. The list of newly added nodes records the node identifier, node type, location information, and input / output information of each newly added node in the second node trajectory. The list of deleted nodes records the node identifier, node type, location information, and input / output information of each deleted node in the first node trajectory. The alignment confidence of each matched node pair is recorded as a numerical value in the matched node pair; a higher alignment confidence indicates a higher degree of credibility that the two nodes correspond to the same functional step.

[0091] For example, in a certain intelligent customer service scenario, the first node trajectory includes an intent recognition node, a permission determination node, a knowledge retrieval node, and an answer generation node. The second node trajectory includes an intent recognition node, a query rewriting node, a permission determination node, a knowledge retrieval node, and an answer generation node. After matching using node identifiers, the intent recognition node, permission determination node, knowledge retrieval node, and answer generation node each form a matching node pair with a high alignment confidence level. The query rewriting node does not have a corresponding node identifier and remains an unmatched node after matching based on node identifiers. It is marked as a new node and recorded in the new node list of the node alignment unit.

[0092] It should be noted that in the actual node alignment process, if a correspondence cannot be established between certain nodes in the first node trajectory and the second node trajectory after all levels of alignment strategies, including exact matching, association matching, structural matching, and semantic matching, the node will be marked as an incomparable node. Exact matching and semantic matching refer to the matching methods described in the preceding steps; association matching refers to determining the correspondence based on the derivation relationship of nodes in version evolution; and structural matching refers to determining the correspondence based on the positional similarity of nodes in the workflow topology. Since incomparable nodes do not have a comparable semantic or functional correspondence between two release versions, and local quality differences cannot be defined, in the node-level degradation contribution calculation of this embodiment, incomparable nodes do not participate in the allocation and normalization process of contribution values. The end-to-end overall degradation amount is explained only by the contribution values ​​of matched node pairs, added nodes, and deleted nodes; if there is an unexplained degradation portion caused by incomparable nodes, it can be considered as an uncertainty factor in the subsequent attribution confidence calculation.

[0093] Through the aforementioned sub-steps, the node alignment operation is refined into two progressive stages: identifier-based matching and semantic-based matching. This ensures that accurate node correspondences can still be established even when the workflow topology undergoes minor changes. The matched node pairs recorded in the node alignment unit provide the basis for subsequent node-level difference comparisons. Newly added and deleted nodes are explicitly identified and included in the node alignment unit, ensuring that subsequent degradation contribution calculations cover all node types and do not overlook the impact of newly added or deleted nodes due to workflow evolution. The alignment confidence is determined and recorded during the matching process, providing a reliable basis for subsequent attribution confidence calculations.

[0094] Step S107: Calculate the node-level degradation contribution of the second release version relative to the first release version based on the node alignment unit, and perform component attribution based on the node-level degradation contribution to obtain the component attribution result.

[0095] In this embodiment of the invention, after obtaining the node alignment unit, it is necessary to quantify the distribution of the degree of performance degradation of the second release version relative to the first release version at the node level, and then attribute the degradation contribution at the node level to specific components, thereby determining which components are the main causes of the overall degradation.

[0096] Node-level degradation contribution is the contribution of each node to the overall degradation, characterizing the extent to which changes in the node's output led to a decrease in the overall performance of the second release compared to the first release. Component attribution is the process of aggregating the degradation contributions of each node according to the components referenced by that node, used to determine the degree of contribution of each component to the overall degradation. The component attribution result is the final output attribution conclusion, including the contribution of each component to the overall degradation and the reliability of the attribution result.

[0097] In the specific implementation, the calculation of node-level degradation contribution is based on the node correspondence established in the node alignment unit. The output difference of each corresponding node in the first node trajectory and the second node trajectory is compared, and combined with the node's position and influence in the agent's workflow, the contribution value of that node to the overall degradation amount is calculated. Component attribution is based on the component version fingerprint referenced by each node during execution, assigning the node-level degradation contribution of that node to the corresponding component. The degradation contributions of all nodes within the same component are then summed to obtain the degradation contribution of that component. Finally, the component attribution result is output, which includes at least the degradation contribution and attribution confidence of each component.

[0098] For example, in a certain intelligent customer service scenario, after calculating the node-level degradation contribution, the contribution value of the knowledge retrieval node is 0.19, the contribution value of the answer generation node is 0.02, and the sum of the contribution values ​​of other nodes is 0.02, with an overall degradation amount of 0.23. During component attribution, the component version fingerprint referenced by the knowledge retrieval node is knowledge base index version I56, and the component version fingerprint referenced by the answer generation node is model version M7 and prompt word version P20. Therefore, the degradation contribution of the knowledge base index component is 0.19, and the combined degradation contribution of the model and prompt word components is 0.02. The attribution confidence is 0.92. The final output component attribution result includes the degradation contribution of the knowledge base index component (0.19), the attribution confidence of 0.92, and the degradation contributions of the remaining components.

[0099] By calculating the degradation contribution at the node level and aggregating the contribution value to the component level, the root cause of degradation is precisely located to the specific component. The component attribution results provide a quantifiable and traceable attribution basis for subsequent release control decisions.

[0100] In some embodiments of the present invention, step S107, "calculating the node-level degradation contribution of the second release version relative to the first release version based on the node alignment unit," specifically includes the following sub-steps: Calculate the end-to-end overall degradation of the second release version relative to the first release version, where the end-to-end overall degradation is the positive part of the difference between the end-to-end score of the first release version and the end-to-end score of the second release version; The local quality difference of each matching node pair in the node alignment unit is obtained, and the local quality difference is used to characterize the degree of change in the output quality of the second release version node relative to the first release version node in the matching node pair; Obtain the propagation weight of each node in the matching node pair. The propagation weight is used to characterize the degree of influence of the output difference of the node on the propagation to downstream nodes. The contribution value of each node in the matched node pair is calculated based on the local quality difference, the propagation weight, and the alignment confidence of the node alignment unit, wherein the alignment confidence is a confidence level determined during the node alignment process; For the newly added node in the node alignment unit, the contribution value of the newly added node is determined according to the degree of influence of the newly added node on the overall output of the second release version; For the deleted node in the node alignment unit, the contribution value of the deleted node is determined according to the degree of contribution of the deleted node to the overall output of the first release version; The contribution values ​​of the matched node pairs, the newly added nodes, and the deleted nodes are normalized so that the sum of the contribution values ​​of all nodes after normalization is equal to the end-to-end overall degradation amount. The contribution value of each node after normalization is the node-level degradation contribution.

[0101] In the specific implementation, an evaluator is first used to score the final outputs of the first node trajectory and the second node trajectory, respectively. The evaluator scores the output content based on preset evaluation rules and scoring templates, obtaining the end-to-end score for the first release version and the end-to-end score for the second release version. The score for the first release version is subtracted from the score for the second release version; if the difference is positive, then this difference is the overall end-to-end degradation.

[0102] Then, the local quality difference of each matching node pair in the node alignment unit is obtained. The local quality difference is used to characterize the degree of change in the output quality of the second release version node relative to the first release version node in the matching node pair. Specifically, based on the output information of the two nodes in the matching node pair, the node output scores of the first release version node and the second release version node are calculated according to the preset node-level evaluation rules. The local quality difference is obtained by subtracting the score of the first release version node from the score of the second release version node.

[0103] Then, the propagation weight of each node in the matched node pair is obtained, and the contribution value of each node in the matched node pair is calculated based on the local quality difference, propagation weight, and alignment confidence of the node alignment unit. The contribution value of each node in the matched node pair is obtained based on the comprehensive calculation of the node's local quality difference, propagation weight, and alignment confidence. The local quality difference reflects the degree of change in the node's own output, the propagation weight reflects the scope of the change of the node propagating downstream, and the alignment confidence reflects the reliability of the node correspondence. The three together determine the magnitude of the node's contribution to the overall degradation. Specifically, based on the inter-node dependencies recorded in the execution trajectory, the number of downstream nodes of each node and the length of the dependency path from the node to the terminal are calculated, and the propagation weight value of the node is determined based on the above information; the contribution values ​​are calculated for the first node trajectory node and the second node trajectory node in the matched node pair respectively. For each node in the matched node pair, the absolute value of the local quality difference is multiplied by the propagation weight, and then multiplied by the alignment confidence to obtain the initial value of the node's contribution value. The initial direction of the contribution value is consistent with the direction of the local quality difference. That is, when the local quality difference is negative, the contribution value is positive, indicating that the node causes degradation. When the local quality difference is positive, the contribution value is negative, indicating that the node brings improvement.

[0104] The addition of a new node may have a positive, negative, or neutral impact on the overall output of the second release. The greater the impact of the new node on the overall output of the second release, the greater its contribution. In the specific implementation, the contribution of the new node is determined by counterfactual reasoning: in the second release, the new node is disabled or its output is replaced with a preset default value and then re-executed. The change in the execution result is compared, and this change represents the degree of impact of the new node on the overall output.

[0105] After a node is removed, its original contribution to the overall output of the first release disappears in the second release. This lost contribution is the node's contribution to the overall degradation. In the specific implementation, the contribution value of the removed node is determined through counterfactual reasoning. After disabling the removed node in the first release and re-executing, the change in the execution result is compared. This change represents the degree of contribution of the removed node to the overall output of the first release.

[0106] Finally, normalization is used to scale the contribution values ​​of each node proportionally, so that the sum of the scaled contribution values ​​of each node equals the end-to-end overall degradation amount. The normalized contribution value of each node is the node-level degradation contribution, which quantifies the actual contribution share of each node to the overall degradation amount. The sum of the node-level degradation contributions of all nodes equals the end-to-end overall degradation amount.

[0107] For example, in a certain intelligent customer service scenario, the end-to-end score of the first release version R100 is 0.94, and the end-to-end score of the second release version R101 is 0.71, with an overall degradation of 0.23. The matching node pairs include four pairings: intent recognition, permission determination, knowledge retrieval, and answer generation. The knowledge retrieval node has a local quality difference of -0.25, a propagation weight of 0.8, an alignment confidence of 0.95, and a contribution value of 0.19. The answer generation node has a local quality difference of -0.05, a propagation weight of 0.5, an alignment confidence of 0.95, and a contribution value of 0.02. The query rewrite node is a newly added node, and its impact on the overall output of the second release version is 0.02, with a contribution value of 0.02. The sum of all node contribution values ​​is 0.23, which equals the end-to-end overall degradation. After normalization, the contribution value of each node is the node-level degradation contribution.

[0108] The above calculation method is only one optional implementation of node-level degradation contribution. In other implementations, ablation analysis, causal graph propagation, approximate Shapley values, or other marginal contribution calculation methods can also be used to determine node degradation contribution.

[0109] Through the above sub-steps, the overall end-to-end degradation amount is accurately allocated to each node. Matching node pairs, newly added nodes, and deleted nodes are all included in the contribution calculation system. The node-level degradation contribution fully covers all sources of difference, providing an accurate node-level data foundation for subsequently aggregating node contributions to the component level.

[0110] In some embodiments of the present invention, step S107, "performing component attribution based on node-level degradation contribution and obtaining component attribution results," specifically includes the following sub-steps: Determine the component version fingerprint referenced by each node in the node alignment unit; When a node references a component version fingerprint, the node's node-level degradation contribution is allocated to the component corresponding to the component version fingerprint; When a node references multiple component version fingerprints, the allocation weight is determined based on the counterfactual replacement result of each component, and the node-level degradation contribution of the node is allocated to the component corresponding to the component version fingerprint according to the allocation weight. The degradation contributions assigned to the same component are aggregated to obtain the degradation contribution for each component; Attribution confidence is calculated based on at least one of the following: sample reproducibility level, alignment confidence of the node alignment unit, playback result variance, evaluator consistency, and proportion of unfixed external dependencies. The attribution confidence characterizes the reliability of the component attribution results. The sample reproducibility level is determined based on the completeness of information contained in the reproducible playback sample package. The playback result variance is calculated based on the variance obtained from multiple playback results driven by the reproducible playback sample package in the first and second release versions, respectively. The evaluator consistency is the degree of consistency among multiple evaluations by the evaluator used to score the outputs of the first and second release versions. The proportion of unfixed external dependencies is the percentage of external dependencies not processed by the fixing operation. The output includes the attribution confidence and the degradation contribution of each component.

[0111] In this embodiment of the application, component attribution is performed based on node-level degradation contribution to obtain component attribution results.

[0112] Node-level degradation contribution quantifies the share of each node's contribution to the overall degradation, but a node itself is merely an execution unit; the real root cause of degradation lies in the changes in the components referenced by the node. Component attribution is the process of aggregating the degradation contributions of a node according to the version fingerprints of the components referenced by that node. Through this aggregation, the size of each component's contribution to the overall degradation can be determined, thereby pinpointing the root cause of the degradation.

[0113] The component attribution result is a conclusive data output after summarizing the degradation contribution and attribution reliability of each component. The degradation contribution of each component is the sum of the node-level degradation contributions of all nodes belonging to that component, used to quantify the extent to which that component caused the overall degradation. Attribution confidence is a numerical indicator characterizing the reliability of the component attribution result; a higher attribution confidence indicates a higher degree of credibility in the component attribution conclusion.

[0114] In the specific implementation, component attribution is performed as follows: First, the component version fingerprint referenced by each node in the node alignment unit is determined. In the execution context, each node references a specific component version during execution, including prompt word version, workflow version, model version, knowledge base index version, tool version, and permission policy version, etc. The component version fingerprint of each node is determined when the execution context is generated, and the mapping relationship between nodes and component version fingerprints is recorded in the node trajectory. Then, all nodes in the node alignment unit are traversed. For each node, the component version fingerprint it references is extracted from its execution record, establishing the correspondence between nodes and component version fingerprints.

[0115] When a node references a component version fingerprint, contribution allocation is performed according to the following rules: When a node references a component version fingerprint, the node-level degradation contribution of that node is allocated to the component corresponding to that component version fingerprint. When a node references multiple component version fingerprints, the allocation weight is determined based on the counterfactual replacement result of each component, and the node-level degradation contribution of that node is allocated to the components corresponding to each component version fingerprint according to the allocation weight. The counterfactual replacement result is determined by replacing each changed component in the candidate version with the corresponding component version of the first release version and then re-executing, comparing the change in execution result after each replacement. The contribution ratio of each changed component to the local quality difference of the node is used as the allocation weight, and the node-level degradation contribution of that node is allocated according to this weight. Then, the degradation contributions of all nodes allocated to the same component version fingerprint are aggregated, that is, all node-level degradation contributions allocated to that component are added together to obtain the degradation contribution of each component.

[0116] Attribution confidence is calculated as follows: Attribution confidence is calculated based on at least one of the following: sample reproducibility level, alignment confidence of node alignment units, playback result variance, evaluator consistency, and percentage of unfixed external dependencies. Attribution confidence characterizes the reliability of component attribution results; a higher attribution confidence indicates a more credible component attribution conclusion. Sample reproducibility level is determined by the completeness of information contained in the reproducible playback sample package; the more complete the information, the higher the sample reproducibility level, and the more credible the attribution result obtained based on that sample. Specifically, reproducibility levels can be divided into four levels: L0 (Input Only): The sample package contains only user input information and does not contain component fingerprints or fixed external dependency data; L1 (Complete Component Fingerprint): The sample package contains the complete component version fingerprint of the execution context but does not contain fixed external dependency data; L2 (Critical Dependency Frozen): The sample package contains component fingerprints and fixed data of some critical external dependencies; L3 (All External Dependencies and Data Watermark Frozen): The sample package contains complete component fingerprints, fixed data of all external dependencies, and data watermark indicators. In subsequent release blocking, L2 / L3 samples will be used preferentially, and risk aggregation will be weighted according to level.

[0117] Alignment confidence is the confidence level determined during node alignment, characterizing the reliability of the two nodes in a matching node pair corresponding to the same functional step. A higher alignment confidence level indicates higher attribution confidence. Replay result variance is calculated based on the variance of replay results from multiple executions of the first and second release versions using reproducible replay sample packages. A smaller variance indicates more stable replay results and higher attribution confidence. Evaluator consistency is the degree of consistency among multiple evaluations of the evaluators used to score the outputs of the first and second release versions. Higher consistency indicates more reliable scoring results and higher attribution confidence. Unfixed external dependency ratio is the proportion of external dependencies not processed by fixed operations. A higher unfixed external dependency ratio indicates a greater likelihood of being affected by changes in the external environment during replay, and lower attribution confidence. In the specific implementation, each of the above indicators is assigned a value and weighted according to preset weighting coefficients to obtain the attribution confidence level.

[0118] Finally, the output includes the component attribution results, which contain the attribution confidence score and the degradation contribution of each component. The data structure of the component attribution results includes an attribution confidence score field and a degradation contribution field for each component. The degradation contribution field for each component contains the component version fingerprint and the degradation contribution value of that component.

[0119] Continuing with the aforementioned intelligent customer service scenario, after calculating the node-level degradation contribution, the knowledge retrieval node's node-level degradation contribution is 0.19, the answer generation node's node-level degradation contribution is 0.02, and the total node-level degradation contribution of the remaining nodes is 0.02. The knowledge base index version fingerprint referenced by the knowledge retrieval node is I56, and the model version fingerprint referenced by the answer generation node is M7 and the prompt word version fingerprint is P20. During component attribution, the knowledge retrieval node only references one component version fingerprint, I56, and its 0.19 contribution is directly assigned to the knowledge base index component. The answer generation node references two components: model version fingerprint M7 and prompt word version fingerprint P20. After replacing the candidate versions M7 and P20 with the first released versions of the model (M7old) and prompt word (P20old) respectively, and re-executing the test, the contribution ratio of M7 is determined to be 60% and that of P20 to be 40% based on the score recovery after the two replacements. According to this weighting, 0.02 of the answer generation node's contribution is allocated to the model component and the prompt word component, with the model component receiving 0.012 and the prompt word component receiving 0.008. The final degradation contribution of the knowledge base index component is 0.19, the model component is 0.012, the prompt word component is 0.008, and the remaining components have no degradation contribution. The sample reproducibility level is L3, the alignment confidence is 0.95, the replay result variance is 0.01, the evaluator consistency is 0.98, the proportion of unfixed external dependencies is 0.02, and the calculated attribution confidence is 0.92. After weighted calculation based on the preset weight configuration, the attribution confidence level is 0.92. The final output component attribution results include the attribution confidence level of 0.92 and the degradation contribution of all components.

[0120] Through component attribution, the degradation contribution of nodes is aggregated at the component level, the root cause of degradation is precisely located to the specific component version, and the attribution confidence score quantifies the reliability of the attribution results. The output component attribution results provide a quantifiable, traceable, and confidence-assessed decision basis for release control decisions.

[0121] Step S108: Generate release control instructions based on component attribution results. Release control instructions are used to control the release status of the second release version.

[0122] Release control instructions refer to control commands issued to the release platform, instructing the platform to perform specified release control operations on the second release version. A release control instruction includes a target release version identifier, a control action, and a reason code. The target release version identifier indicates the release version to which the control action is applied. The control action indicates the specific type of operation performed by the release platform. The reason code records the basis for generating the instruction, for subsequent auditing and traceability. Release status refers to the stage of the second release version in the release process, including the pending evaluation status, replay status, release-ready status, review-required status, and blocked status, as well as the monitoring status, hold status, downgraded status, and rollback status after entering the production environment.

[0123] In the implementation, release control instructions are generated according to preset release control rules based on component attribution results. These release control rules are configured in the system as conditional statements; the input to each rule is the component attribution result, and the output is the release control action. When a component attribution result meets the conditions of a rule, the corresponding control action is executed, generating the corresponding release control instruction. Different scenarios correspond to different control actions, and the decision-making basis for the control actions is the degradation contribution and attribution confidence of each component in the component attribution results.

[0124] In some embodiments of the present invention, step S108 specifically includes the following sub-steps: When a security-related component degenerates in the component attribution results and the attribution confidence exceeds a preset confidence threshold, a release blocking instruction is generated. When the sum of the degradation contributions of all components exceeds the preset overall degradation threshold and the attribution confidence exceeds the preset confidence threshold, a release blocking instruction or a grayscale reduction instruction is generated. The grayscale reduction instruction is used to reduce the proportion of traffic to users in the second release version. When the sum of the degradation contributions of all components exceeds the preset overall degradation threshold and the attribution confidence does not exceed the preset confidence threshold, a pause release and replay instruction is generated, or a manual review instruction is generated. When the sum of the degradation contributions of all components does not exceed the preset overall degradation threshold and there are no security-related components degrading, a release permission instruction is generated.

[0125] In this embodiment of the invention, the release control rules described above cover at least four typical scenarios: blocking scenarios when security components degrade, blocking or downgrading scenarios when overall degradation exceeds a threshold, scenarios where version issues require intermediate measures, and scenarios where release is permitted without significant degradation. Security component degradation refers to the degradation of components related to security compliance in the component attribution results, such as unauthorized access, sensitive information leakage, and policy violations, where indicators deteriorate in the second release version compared to the first release version. Specifically, for example, permission policy components may lead to unauthorized access, and content security components may lead to sensitive information leakage. Security component degradation is high-risk; even if the degradation is small, release should be immediately blocked to prevent security risks from entering the production environment.

[0126] When the sum of the degradation contributions of all components exceeds a preset overall degradation threshold and the attribution confidence exceeds a preset confidence threshold, a release blocking instruction or a grayscale reduction instruction is generated. The sum of the degradation contributions of all components equals the end-to-end overall degradation amount, reflecting the overall performance decline of the second release version relative to the first release version. When the overall degradation amount exceeds the preset overall degradation threshold, it indicates that the overall performance of the second release version is significantly worse than that of the first release version, and the release should be blocked or the user-facing traffic ratio of the second release version should be reduced to control the impact of the risk. The grayscale reduction instruction is used to reduce the user-facing traffic ratio of the second release version, reducing the user traffic of the second release version below the preset ratio and continuing to observe, rather than completely stopping the release. It is suitable for situations where the degradation level is within a critical range or where further data collection is required.

[0127] When the sum of the degradation contributions of all components exceeds a preset overall degradation threshold, but the attribution confidence does not exceed a preset confidence threshold, a pause release and replay append instruction is generated, or a manual review instruction is generated. An overall degradation exceeding the threshold indicates that the version does indeed have a problem that needs to be addressed, but the confidence is insufficient to support a blocking decision; in this case, an intermediate measure is required. The pause release and replay append instruction can improve the confidence through more data, while the manual review instruction can compensate for insufficient attribution confidence by leveraging human expertise. Either of these options can be selected, and the specific method used can be determined based on the preset operation and maintenance strategy.

[0128] When the sum of the degradation contributions of all components does not exceed the preset overall degradation threshold and there are no security-related component degradations, a release permission instruction is generated. This indicates that the second release version does not have significant degradation compared to the first release version and has no security risks, allowing the release process to proceed normally.

[0129] Continuing the previous example, the component attribution results show that the degradation contribution of the knowledge base index component is 0.19, the attribution confidence is 0.92, the sum of the degradation contributions of all components is 0.23, the preset overall degradation threshold is 0.08, and the preset confidence threshold is 0.85. Since the attribution confidence of 0.92 exceeds the threshold of 0.85, and the overall degradation of 0.23 exceeds the overall degradation threshold of 0.08, it is determined that the second release version has significant degradation, and a release blocking instruction is generated for the second release version. The release blocking instruction includes the version identifier R101 of the second release version, the control action of blocking release, the reason code of overall degradation exceeding the threshold, and a reference to the component attribution results. After receiving the release blocking instruction, the release platform sets the release status of the second release version to blocked status and stops the release process for that version.

[0130] By automatically generating release control instructions based on component attribution results, the evaluation and release processes are seamlessly integrated. The response time from detecting degradation to executing blocking, downgrading, or rollback is significantly shortened, avoiding time delays and human errors caused by manual judgment and operation. Four decision branches cover all four scenarios: security-related degradation blocking, overall degradation with sufficient confidence blocking or downgrading, intermediate state handling of overall degradation but insufficient confidence, and allowing release without significant degradation. There are no indeterminate intervals. Release control instructions include the target release version identifier, control actions, and cause codes, making release control operations traceable and auditable, providing a complete decision-making basis for subsequent release process optimization and issue review.

[0131] In some embodiments of the present invention, the above method further includes the following steps: When the release status of the second release version is blocked or the grayscale ratio is reduced, the reproducible replay sample package, component attribution results, release version information of the second release version, and the attribution component identifier corresponding to the second release version will be associated and stored in the regression sample set.

[0132] In this embodiment of the invention, a sample set is also collected. The regression sample set is a collection of samples that trigger degradation and their associated information. Samples in the regression sample set are used as test cases for replay during the evaluation of subsequent release versions to continuously verify whether similar degradation issues recur. When the release status of the second release version is blocked or the grayscale ratio is reduced, it indicates that the second release version has significant degradation. The reproducible replay sample package is a valid sample capable of reproducing this degradation issue and has value for long-term storage and reuse. The attribution component identifier is the identification information of the main component triggering the degradation, used to mark the degradation component associated with the sample, facilitating subsequent retrieval and filtering of regression samples by component dimension.

[0133] In the specific implementation, when the release status of the second release version is set to release blocking or reduced grayscale ratio, the regression sample accumulation process is automatically triggered. Reproducible replay sample packages, component attribution results, and release version information of the second release version are obtained from the current replay task. The attribution component identifier corresponding to the second release version is determined, based on the component with the largest degradation contribution in the component attribution results. This information is then associated and stored in the regression sample set. Before storage, the regression samples are deduplicated based on input fingerprints, trajectory fingerprints, and root cause fingerprints. Input fingerprints are hash values ​​generated based on the sample's input information, used to identify samples with the same input. Trajectory fingerprints are hash values ​​generated based on the sample's execution trajectory, used to identify samples with the same execution path. Root cause fingerprints are hash values ​​generated based on the attribution component identifier and degradation type, used to identify samples with the same root cause. When a new sample has the same input fingerprint, trajectory fingerprint, and root cause fingerprint as an already stored sample, the new sample is considered a duplicate sample and is not stored repeatedly. Only the version association information and trigger count of the existing sample are updated to avoid a large number of duplicate samples in the regression sample set. Each regression sample retains complete version association information during storage, including source release version identifier, candidate release version identifier, attribution component identifier, fix version identifier, and evaluation history, thus preserving the complete traceability chain of the regression sample.

[0134] For example, in a certain intelligent customer service scenario, the component attribution results of the reproducible replay sample package SP-9001 indicate that knowledge base index version I56 is the main component causing degradation, with an attribution confidence of 0.92, and the release status of the second release version R101 is release blocked. The reproducible replay sample package SP-9001, the component attribution results, the release version information of the second release version R101, and the attribution component identifier I56 are associated and stored in the regression sample set. After the subsequent index snapshot I57 is updated and verified, SP-9001 is set as a mandatory regression sample for subsequent versions. The replay of this sample is automatically executed during each new candidate version evaluation, ensuring that index-related degradation issues are not reintroduced.

[0135] By storing the samples that trigger degradation and their associated information into the regression sample set, valuable degradation samples are continuously accumulated as regression evaluation assets. Version association information makes the source and evolution history of each regression sample traceable. Deduplication avoids the redundant expansion of the regression sample set. The regression samples are automatically executed in subsequent version evaluations to ensure that similar degradation issues are continuously verified, forming a complete closed loop from problem discovery to problem reproduction and verification.

[0136] Through the above steps, this invention integrates execution trajectory collection, context solidification, external dependency freezing, sample package construction and replay, node alignment attribution, and release control into a complete technical solution. The collection of execution trajectories and the construction of reproducible replay sample packages are executed asynchronously. Execution events in the production environment are written to a local buffer or message channel. The collection service deduplicates and sorts the execution events, and completes the parent-child and causal relationships. The context resolver obtains the component versions valid at the time of production execution and determines the execution context. The dependency freezer records the results of tool calls and retrieval recalls or requests snapshot services to solidify business data. The sample builder generates a sample list and integrity signature based on the execution trajectory, execution context, multi-component version fingerprints, and solidified external dependencies, completing the construction of the reproducible replay sample package. Whether the reproducible replay sample package enters the replay queue is determined based on user negative feedback, low ratings, anomalies, representativeness, rare paths, and sampling strategies. The sample packets entering the replay queue drive the execution of the first and second release versions in the isolation environment, obtain the first node trajectory and the second node trajectory, and generate release control instructions after node alignment, node-level degradation contribution calculation and component attribution, so as to control the release status of the second release version.

[0137] The technical solution of this invention has at least the following beneficial effects: by binding the execution trajectory with the execution context and generating multi-component version fingerprints, the complete operating conditions at the production moment can be reconstructed; by solidifying the external dependencies in the execution trajectory, the interference of external environment changes on the replay comparison between versions is eliminated, enabling the two versions to share the same reproducible replay sample package for homogeneous pairing execution in an isolated environment, ensuring the consistency of the comparison benchmark; by aligning the nodes of the two versions and calculating the node-level degradation contribution, the difference in the first node can be accurately located and the upstream propagation effect and the node's own effect can be distinguished, and then the node contribution is aggregated according to the referenced component version fingerprint, realizing component-level degradation attribution; finally, the release control command is automatically generated based on the component attribution result, shortening the response time from problem discovery to blocking, degradation or rollback.

[0138] This embodiment also provides a playback-attribution-based agent deployment control device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0139] This embodiment provides an agent deployment control device based on playback attribution, such as... Figure 2 As shown, it includes: The acquisition module 201 is used to acquire the execution trajectory of the agent in the production environment and determine the execution context corresponding to the execution trajectory. The execution trajectory includes the nodes passed by the agent during execution, the dependencies between nodes, and the input and output information of the nodes. The execution context is used to characterize the component combination state and data state that the execution trajectory depends on during execution. The fingerprint generation module 202 is used to generate a multi-component version fingerprint based on the execution context, wherein the multi-component version fingerprint is used to uniquely identify the combination of components contained in the execution context; The solidification module 203 is used to solidify the external dependencies in the execution trajectory, wherein the external dependencies are external states, external call results, or non-deterministic operating conditions that affect the execution result of the execution trajectory. The sample construction module 204 is used to generate a sample list and integrity verification information based on the execution trajectory, the execution context, the multi-component version fingerprint and the solidified external dependencies, and to construct a reproducible replay sample package. The replay module 205 is used to drive the execution of the first release version and the second release version respectively in an isolated environment using the same reproducible replay sample package to obtain the first node trajectory and the second node trajectory. Alignment module 206 is used to align the first node trajectory and the second node trajectory to obtain a node alignment unit; The attribution module 207 is used to calculate the node-level degradation contribution of the second release version relative to the first release version according to the node alignment unit, and to perform component attribution according to the node-level degradation contribution to obtain the component attribution result. The release control module 208 is used to generate release control instructions based on the component attribution results, and the release control instructions are used to control the release status of the second release version.

[0140] In this embodiment, an intelligent agent release control device based on playback attribution is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0141] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0142] This invention also provides a computer device having the above-described features. Figure 2 The diagram shows an agent release control device based on playback attribution.

[0143] The following is a detailed reference. Figure 3The diagram illustrates a structural schematic suitable for implementing a computer device according to embodiments of the present invention. The computer device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes based on a program stored in a read-only memory (i.e., ROM 302) or a program loaded from memory 308 into random access memory (i.e., RAM 303). The RAM 303 also stores various programs and data required for the operation of the computer device. The processor 301, ROM 302, and RAM 303 are interconnected via a bus 304. Input / output (i.e., I / O interface 305) is also connected to the bus 304.

[0144] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows the computer device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Computer equipment with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0145] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a memory 308, or installed from a ROM 302. When the computer program is executed by the processor 301, it performs the functions defined in a playback attribution-based agent deployment control method according to embodiments of the present invention.

[0146] Figure 3 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0147] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, it implements the playback attribution-based agent deployment control method shown in the above embodiments.

[0148] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0149] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for agent deployment control based on playback attribution, characterized in that, The method includes: The execution trajectory of the agent in the production environment is collected, and the execution context corresponding to the execution trajectory is determined. The execution trajectory includes the nodes traversed by the agent during execution, the dependencies between nodes, and the input and output information of the nodes. The execution context is used to characterize the component combination state and data state that the execution trajectory depends on during execution. A multi-component version fingerprint is generated based on the execution context, and the multi-component version fingerprint is used to uniquely identify the combination of components contained in the execution context; The external dependencies in the execution trajectory are solidified, and the external dependencies are external states, external call results, or non-deterministic operating conditions that affect the execution result of the execution trajectory. Based on the execution trajectory, the execution context, the multi-component version fingerprint, and the solidified external dependencies, a sample list and integrity verification information are generated to construct a reproducible replay sample package. In an isolated environment, the first release version and the second release version are executed using the same reproducible replay sample package to obtain the first node trajectory and the second node trajectory. Align the first node trajectory and the second node trajectory to obtain node alignment units; The node-level degradation contribution of the second release version relative to the first release version is calculated based on the node alignment unit, and component attribution is performed based on the node-level degradation contribution to obtain the component attribution result; Release control instructions are generated based on the component attribution results, and these instructions are used to control the release status of the second release version.

2. The method according to claim 1, characterized in that, The execution context includes at least one of the following: tenant information, application information, agent information, release version information, prompt word version fingerprint, workflow version fingerprint, model version fingerprint, knowledge base index version fingerprint, tool version fingerprint, permission policy version fingerprint, and data watermark. Wherein, the tenant information is used to identify the tenant domain to which the execution trajectory belongs; the application information is used to identify the application to which the agent belongs; the agent information is used to identify the agent instance corresponding to the execution trajectory; the release version information is used to identify the release version of the agent corresponding to the execution trajectory; the prompt word version fingerprint is used to identify the version of the prompt word used by the agent; the workflow version fingerprint is used to identify the version of the workflow used by the agent; the model version fingerprint is used to identify the version of the model used by the agent; the knowledge base index version fingerprint is used to identify the version of the knowledge base index used by the agent; the tool version fingerprint is used to identify the version of the tool used by the agent; the permission policy version fingerprint is used to identify the version of the permission policy used by the agent; and the data watermark is used to identify the business data status when the execution trajectory is executed.

3. The method according to claim 1, characterized in that, The step of generating a multi-component version fingerprint based on the execution context includes: For each component in the execution context, a corresponding component fingerprint is generated based on the component's identification information, configuration information, and version information; The component fingerprints of all components are concatenated in a predetermined order, and their hash values ​​are calculated to obtain the multi-component version fingerprint.

4. The method according to claim 1, characterized in that, The external dependencies include at least one of the following: tool call results, retrieval and recall results, business data, time information, and random number seeds. The solidification of the external dependencies includes: For the tool invocation result, store the invocation request to the target tool and the invocation response returned by the target tool when the execution trajectory is executed; For the retrieval and recall results, the retrieval query conditions, recalled document content, and sorting results during the execution of the execution trajectory are fixed; For the business data, create a read-only data snapshot of the business data at the execution time of the execution trajectory, or record a consistency identifier of the business data. The consistency identifier is used to obtain business data consistent with the execution time of the execution trajectory from the data source when driving the execution of the first release version or the second release version. For the time information, the logical clock during the execution of the execution trajectory is recorded. The logical clock is used to restore the timing relationship during the execution of the execution trajectory when driving the first release version or the second release version to execute. For the random number seed, the random number seed used when the execution trajectory is executed is recorded. The random number seed is used to make the random operation generate the same random sequence when driving the execution of the first release version and the second release version.

5. The method according to claim 1, characterized in that, The step of aligning the first node trajectory and the second node trajectory to obtain a node alignment unit includes: The nodes in the first node trajectory and the nodes in the second node trajectory are matched using node identifiers, which are used to uniquely identify a node in the agent's workflow. When a node cannot be matched by the node identifier, a semantic signature is generated based on the node's attribute information, and the unmatched node is matched based on the semantic signature. The attribute information includes at least one of the node's position information in the workflow, the node's input and output feature information, and information about the components referenced by the node. Node alignment units are generated based on the matching results, and the alignment confidence of each matching node pair is determined. The node alignment units include matching node pairs, newly added nodes, and deleted nodes.

6. The method according to claim 5, characterized in that, The step of calculating the node-level degradation contribution of the second release version relative to the first release version based on the node alignment unit includes: Calculate the end-to-end overall degradation of the second release version relative to the first release version, where the end-to-end overall degradation is the positive part of the difference between the end-to-end score of the first release version and the end-to-end score of the second release version; The local quality difference of each matching node pair in the node alignment unit is obtained, and the local quality difference is used to characterize the degree of change in the output quality of the second release version node relative to the first release version node in the matching node pair; Obtain the propagation weight of each node in the matching node pair. The propagation weight is used to characterize the degree of influence of the output difference of the node on the propagation to downstream nodes. The contribution value of each node in the matched node pair is calculated based on the local quality difference, the propagation weight, and the alignment confidence of the node alignment unit, wherein the alignment confidence is a confidence level determined during the node alignment process; For the newly added node in the node alignment unit, the contribution value of the newly added node is determined according to the degree of influence of the newly added node on the overall output of the second release version; For the deleted node in the node alignment unit, the contribution value of the deleted node is determined according to the degree of contribution of the deleted node to the overall output of the first release version; The contribution values ​​of the matched node pairs, the newly added nodes, and the deleted nodes are normalized so that the sum of the contribution values ​​of all nodes after normalization is equal to the end-to-end overall degradation amount. The contribution value of each node after normalization is the node-level degradation contribution.

7. The method according to claim 6, characterized in that, The step of attributing components based on the node-level degradation contribution to obtain component attribution results includes: Determine the component version fingerprint referenced by each node in the node alignment unit; When the node references a component version fingerprint, the node-level degradation contribution of the node is allocated to the component corresponding to the component version fingerprint; When the node references multiple component version fingerprints, the allocation weight is determined based on the counterfactual replacement result of each component, and the node-level degradation contribution of the node is allocated to the component corresponding to the component version fingerprint according to the allocation weight. The degradation contributions assigned to the same component are aggregated to obtain the degradation contribution for each component; Attribution confidence is calculated based on at least one of the following: sample reproducibility level, alignment confidence of the node alignment unit, playback result variance, evaluator consistency, and proportion of unfixed external dependencies. The attribution confidence characterizes the reliability of the component attribution results. The sample reproducibility level is determined based on the completeness of information contained in the reproducible playback sample package. The playback result variance is calculated based on the variance obtained from multiple playback results driven by the reproducible playback sample package in the first and second release versions, respectively. The evaluator consistency is the degree of consistency among multiple evaluations by the evaluator used to score the outputs of the first and second release versions. The proportion of unfixed external dependencies is the percentage of external dependencies not processed by the fixing operation. The output includes the attribution confidence and the degradation contribution of each component.

8. The method according to claim 7, characterized in that, The step of generating release control instructions based on the component attribution results includes: When a security-related component degenerates in the component attribution results and the attribution confidence exceeds a preset confidence threshold, a blocking instruction is generated. When the sum of the degradation contributions of all components exceeds the preset overall degradation threshold and the attribution confidence exceeds the preset confidence threshold, a release blocking instruction or a grayscale reduction instruction is generated. The grayscale reduction instruction is used to reduce the proportion of traffic to users in the second release version. When the sum of the degradation contributions of all components exceeds the preset overall degradation threshold and the attribution confidence does not exceed the preset confidence threshold, a pause release and replay instruction is generated, or a manual review instruction is generated. When the sum of the degradation contributions of all components does not exceed the preset overall degradation threshold and there are no security-related components degrading, a release permission instruction is generated.

9. The method according to claim 1, characterized in that, The reproducible replay sample package includes the sample list and the integrity verification information. The sample list includes a sample identifier, the execution trajectory, the component version fingerprint set of the execution context, the solidified external dependency set, and version association information. The integrity verification information is used to verify the integrity of the reproducible replay sample package. The version association information includes the identifier of the agent release version corresponding to the execution trajectory and the collection time of the execution trajectory.

10. The method according to claim 1, characterized in that, The method further includes: When the release status of the second release version is release blocked or grayscale ratio reduced, the reproducible replay sample package, the component attribution result, the release version information of the second release version, and the attribution component identifier corresponding to the second release version are associated and stored in the regression sample set.

Images