Service agent self-evolution method

By recording and evaluating the service trajectory of service agents in an interactive simulation environment and optimizing the parameters of service agents using a reward model, the stability and adaptability problems of service agents in complex scenarios in existing technologies are solved, and continuous iterative optimization and stable behavioral control of service agents are realized.

CN123070170APending Publication Date: 2026-11-10RAJAX NETWORK &TECHNOLOGY (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611572554.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-10
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing service-oriented intelligent agents struggle to stably control detailed behaviors in complex service scenarios. Their training methods are limited by manually labeled data and supervised training, making it difficult to adapt to changes in online tasks in a timely manner. This results in limited adaptability of the models to complex service decisions and unseen scenarios.

Method used

By controlling a service agent to perform service tasks multiple times in an interactive simulation environment, recording service trajectories, and using a reward model to generate result rewards and process rewards, trajectory comparisons and model parameter updates are performed to optimize the behavior of the service agent.

Benefits of technology

It achieves stable control and continuous iterative optimization of the behavior of service agents in complex service scenarios, reduces interference during training, and improves the model's adaptability to different task conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123070170A_ABST
    Figure CN123070170A_ABST
Patent Text Reader

Abstract

This specification provides a self-evolution method for service agents. The method includes: acquiring service tasks and corresponding task evaluation information; controlling the service agent to execute multiple times in an interactive simulation environment based on the service tasks, obtaining a trajectory group consisting of multiple service trajectories; inputting each service trajectory and task evaluation information into a pre-trained reward model to obtain the result reward and process reward of the service trajectory, wherein the result reward is related to the user's goal and acceptance criteria, and the process reward is related to key node evaluation items; aggregating the result reward and process reward of the service trajectory to obtain the trajectory reward of the service trajectory; performing a relative comparison of the trajectory rewards of each service trajectory within the trajectory group to obtain the relative advantage of each service trajectory; and updating the model parameters of the service agent based on the relative advantage of each service trajectory.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and more particularly to a method for self-evolution of service agents. Background Technology

[0002] With the development of large language models and intelligent agent technology, service-oriented intelligent agents based on large language models are now able to understand user requests, generate responses, and invoke external tools to complete information queries or business operations. The processing quality in service scenarios depends not only on the accuracy of the response but also on factors such as the timing of clarification, the order of information confirmation, the method of tool invocation, response boundaries, solution switching, and state transitions in multi-round interactions. These factors typically manifest as continuously changing decision preferences, which are difficult to fully describe using a limited number of fixed rules.

[0003] Service-oriented intelligent agents typically optimize their capabilities by adding prompts and business rules to constrain model behavior, or by adopting an iterative model path of "sample collection → manual annotation → supervised training → version release." However, as service scenarios and constraints increase, continuously adding prompts and rules can easily lead to increased input content, conflicting rules, and distracted model attention, making it difficult to stably control detailed behavior in different scenarios. Supervised training, on the other hand, relies on pre-collected and manually annotated samples, and the model's capabilities are easily limited by the coverage, timeliness, and quality of the training data.

[0004] When online task types, user expression methods, or business status change, existing training data may not be able to cover the new service situations in a timely manner, and the model's adaptability to complex service decisions and unseen scenarios is limited. Therefore, there is still room for improvement in the training methods of related service-oriented intelligent agents in terms of learning complex service behaviors and continuously iterating and optimizing model capabilities. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a self-evolution method for service agents is proposed, comprising: Obtain service tasks and corresponding task evaluation information. The service tasks include initial service context and callable tool information. The task evaluation information is used to describe user goals, acceptance conditions, and key node evaluation items. The control service agent executes the service task multiple times in the interactive simulation environment to obtain a trajectory group consisting of multiple service trajectories. The service trajectory includes multiple trajectory nodes and execution results. The trajectory nodes are used to record any one of the trajectory events, such as interactive actions, tool calls, tool returns, and service status changes. Each service trajectory and the task evaluation information are input into a pre-trained reward model to obtain the result reward and process reward of the service trajectory. The result reward is related to the user goal and the acceptance criteria, and the process reward is related to the key node evaluation items. The service trajectory reward is obtained by aggregating the result reward and process reward of the service trajectory. The trajectory rewards of each service trajectory within the trajectory group are compared relative to each other to obtain the relative advantages of each service trajectory. The model parameters of the service agent are updated based on the relative advantages of each service trajectory.

[0006] According to a second aspect of the embodiments of this specification, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0007] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect.

[0008] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0009] As can be seen from the above embodiments, this specification controls the service agent to execute the same service task multiple times in an interactive simulation environment, so that each service trajectory in the same trajectory group has the same task input conditions. This can provide comparable data for comparing different service behaviors and avoid the trajectory generation process directly affecting the actual service environment.

[0010] By recording interactive actions, tool calls, tool returns, service status changes, and execution results in the service trajectory, and combining these with task evaluation information to generate result rewards and process rewards by the reward model, the execution results and process of the service task can be reflected in the reward information used for model optimization. Aggregating result rewards and process rewards into trajectory rewards and comparing them relatively within the same trajectory group allows the relative advantages to better reflect the behavioral differences between different service trajectories, reducing the interference of different task conditions on the model optimization signal.

[0011] By updating the model parameters of the service agent based on its relative advantage, the service agent can adjust its parameters according to the service trajectory and corresponding evaluation results generated in the interactive simulation environment, thereby supporting the learning of complex service behaviors and the iterative optimization of model capabilities.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the self-evolutionary lifecycle of a service agent provided in an exemplary embodiment.

[0014] Figure 2 This is a schematic diagram illustrating the composition of a quality of service label provided in an exemplary embodiment.

[0015] Figure 3 This is a schematic diagram of a reward model training process provided in an exemplary embodiment.

[0016] Figure 4 This is a schematic diagram of the input-output relationship of a reward model training provided in an exemplary embodiment.

[0017] Figure 5 This is a schematic diagram of the interactive simulation environment structure provided in an exemplary embodiment.

[0018] Figure 6 This is a flowchart illustrating a service agent self-evolution method provided in an exemplary embodiment.

[0019] Figure 7 This is a schematic diagram of the input and output data flow of the self-evolution process of a service agent provided in an exemplary embodiment.

[0020] Figure 8 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0022] It should be noted that the user interaction information, order or business status information, tool call information, user feedback information, and service trajectories formed by the above information involved in the embodiments of this specification can all be obtained and used with the authorization of the user or relevant entity. The collection, storage, use, processing, transmission, provision, and disclosure of related data shall comply with applicable laws, regulations, and relevant standards, and corresponding authorization, authorization withdrawal, or rejection entry points may be provided to users or relevant entities. The data processing scope of the embodiments of this specification can be configured according to specific service scenarios, authorization scope, and compliance requirements.

[0023] I. Application Scenarios of Service Agents

[0024] One or more embodiments of this specification can be applied to online customer service, intelligent voice customer service, or other scenarios where intelligent agents perform continuous service tasks. In such scenarios, the service agent can identify user requests based on user input, business status, and historical interaction context, supplement information through multiple rounds of interaction, and invoke appropriate tools based on the current service status to obtain tool return information, update the service status, and provide processing results. Therefore, a single operation of the service agent typically corresponds to a continuous process including multiple rounds of interaction, tool invocation, status changes, and a final processing result.

[0025] With improvements in the understanding, generation, and tool invocation capabilities of general-purpose models, service agents are generally able to handle common user requests. However, completing customer service tasks does not equate to consistently providing services that meet user goals and service standards. The quality of a service task may depend on multiple consecutive fine-grained decisions, such as whether to appease or clarify first, ask one necessary question at a time or provide multiple branches simultaneously, when to execute an action directly, when to request confirmation, how to faithfully restate the information returned by the tool, and how to provide alternative paths after the user rejects the first solution.

[0026] The appropriateness of the aforementioned service behaviors varies depending on user goals, business status, and prior interactions, and is usually difficult to exhaustively enumerate using a few independent "must-do" or "prohibit-do" rules. Even a combination of actions that seem reasonable at a local level may cause the overall service direction to deviate from the user's needs. Therefore, this type of service capability depends not only on whether the model possesses relevant knowledge, but also on the response strategies and decision boundaries formed by the model during continuous service provision.

[0027] One related approach is to continuously add behavioral rules to the prompts. This method can constrain a small number of explicit behaviors, but as the service scenarios and business definitions increase, the number of rules and the length of the input also increase accordingly. When different rules are triggered in the same context, unclear priorities or rule conflicts may also occur. Longer prompts occupy the model's context space and increase the attention burden, making it difficult to continuously and stably control the details of the service process.

[0028] Another related approach employs a sequential process of "sample collection, manual annotation, offline training, and model release." This method can improve the model using existing labeled samples, but training materials typically need to be created manually, one sample at a time, and the annotation workload increases with the business scenario and the number of samples. Furthermore, the multiple sequential steps make it difficult for newly emerging online issues to be reflected in subsequent model versions in a timely manner. The same or similar problems may reappear before the next model update.

[0029] As business scenarios expand, different problem types often require separate collection, labeling, and verification of samples, potentially increasing manual processing workload in tandem with the number of scenarios. Changes in service rules or business definitions may render some labeling conclusions in existing samples inapplicable, necessitating re-identification and revision of affected samples. Consequently, data preparation and quality verification can easily become limiting factors in the model iteration process.

[0030] Furthermore, supervised fine-tuning typically uses a pre-given target response or processing sequence as the learning objective, limiting the processing methods the model can learn. While alignment based on human preferences can establish a hierarchy among multiple candidate outcomes, its optimization scope is still influenced by the candidate set. For service tasks involving multiple rounds of interaction, tool calls, and state changes, training only around a single response or a pre-given processing sequence is insufficient to fully reflect the correlation between different decisions during continuous service processes.

[0031] II. Basic Concepts and Overall Process of Self-Evolution of Service Agents

[0032] To facilitate understanding of the subsequent embodiments, the relevant terms used in this specification are explained below. These terms are used to describe the meaning and interrelationships of various technical objects in the self-evolution process of the service agent, and do not constitute a limitation on their specific implementation forms.

[0033] 1. A Service Agent is a model runtime unit that processes a complete service task. A Service Agent can include a Planner and an Executor. The Planner receives the initial service context, user input, current service status, and information on available tools, identifies the current service stage and pending tasks, and determines the next interactive action, tool call, or service path. The Executor converts the processing content determined by the Planner into interactive action instructions or tool call instructions that can be executed by the interactive environment, receives tool return information, updates the service status, and generates user-facing responses. The Planner and Executor can be implemented using the same model and its different functional interfaces, or they can be implemented using independent models or program modules.

[0034] 2. A service trajectory refers to a structured record of the entire process generated by a service agent executing a service task. A service trajectory can include the input context of each round, planning information or decision state, tool calls and their parameters, tool return information, user-facing responses, service state changes, and task execution results. The information in the service trajectory can be organized into multiple trajectory nodes according to the time of generation or execution order. Each trajectory node records trajectory events such as interactive actions, tool calls, tool returns, or service state changes generated at the corresponding time.

[0035] 3. Service Quality Score: This score characterizes the overall service quality corresponding to a complete service trajectory. The service quality score can be determined jointly by result-oriented and process-oriented information. Result-oriented information characterizes the degree of conformity between the task execution result and the user's objectives or acceptance criteria, while process-oriented information characterizes the degree of conformity between the task execution process and service specifications or required procedures.

[0036] 4. A reward model is a model that predicts the evaluation result of a service trajectory based on its corresponding task evaluation information. In one implementation, the reward model can be a Domain Reward Model (DRM), which is pre-trained using sample service trajectories, sample task evaluation information, and service quality labels from a specific service domain. The task evaluation information can describe user goals, acceptance criteria, and key node evaluation items, enabling the reward model to process the corresponding service trajectory in conjunction with the evaluation requirements of the service task, rather than relying solely on the general linguistic features of the response text.

[0037] The reward model can be configured with outcome evaluation outputs and process evaluation outputs. Outcome evaluation outputs are used to generate outcome rewards, while process evaluation outputs are used to generate process rewards. Process evaluation outputs can also be associated with one or more key nodes in the service trajectory that conform to the key node type to represent the execution status of different service stages.

[0038] 5. Outcome rewards are used to characterize the degree of conformity between the task execution results of the service trajectory and the user's goals and acceptance criteria. Outcome rewards can include outcome acceptance rewards and goal consistency rewards. Outcome acceptance rewards reflect whether the task execution results have met the corresponding acceptance requirements, while goal consistency rewards reflect whether the actual service direction reflected in the service trajectory is consistent with the user's goals. Outcome rewards thus indicate whether the task has been completed and whether the task processing direction still revolves around the user's goals.

[0039] 6. Process rewards are used to characterize the degree of conformity between the service process and the evaluation items of key nodes in the service trajectory. For example, process rewards can reflect whether the service agent performs necessary verification before responding, whether it requests confirmation in the prescribed order, whether it asks for necessary information at once, whether it faithfully restates the information returned by the tool, or whether it generates commitments beyond the scope of existing information. For service trajectories with multiple key nodes, the reward model can output the node process rewards associated with the corresponding key nodes separately and aggregate them according to the category to which the key nodes belong.

[0040] Node process rewards are aggregated according to the category of the key node. See also Figure 1 , Figure 1 The lowercase letters a to o in the code are process markers used to indicate the relative execution order of processing relationships within the complete lifecycle. When the same process marker corresponds to multiple parallel connections, these multiple parallel connections collectively represent the same processing relationship. Figure 1 In the diagram, solid boxes represent models or operating environments, while dashed boxes represent input, intermediate, or output data. Solid arrows indicate processing or data transfer relationships in the current lifecycle, while dashed arrows indicate the flow of related data into the next lifecycle.

[0041] In the complete lifecycle covered by the embodiments of this specification, a reward model can be trained using sample service trajectories, sample task evaluation information, and service quality labels, corresponding to process marker a. Failure modes are identified from low-quality service trajectories generated during online service, generating repeatable service tasks and corresponding task evaluation information, corresponding to process marker b. The service task is provided to the current service agent, corresponding to process marker c. The service task is loaded into the interactive simulation environment, corresponding to process marker d. The current service agent executes the service task multiple times in the interactive simulation environment configured with simulated users, simulated tools, business state machines, and service constraints, and receives feedback information from the interactive simulation environment, corresponding to process marker e, thereby forming a trajectory group composed of multiple service trajectories, corresponding to process marker f.

[0042] Each service trajectory in the trajectory group and its corresponding task evaluation information are input into a pre-trained reward model (process label g). The reward model outputs the result reward and process reward for each service trajectory (process label h). The result reward and process reward are aggregated, and the relative advantages of each service trajectory within the trajectory group are compared within the group (process label i). The model parameters of the service agent are updated according to the relative advantages of each service trajectory (process label j) to form a candidate model version (process label k).

[0043] The candidate model version is validated, corresponding to process marker l; the validated candidate model version is deployed as the online service version, corresponding to process marker m; the online service version generates new online service trajectories during operation, corresponding to process marker n; the generated data can be used to supplement service tasks and update training and evaluation data, corresponding to process marker o, thus entering the next round of reward model training and service agent update process.

[0044] III. Service Capability Type Identification and Training Route Determination In one embodiment of this specification, before training the service agent, the type of the capability to be optimized can be identified based on the target certainty, evaluation method, and degree of environmental interaction dependence, and a training route can be determined accordingly. The capability to be optimized can be extracted from online low-quality service trajectories and associated with relevant service contexts, trajectory nodes, user goals, and improvement requirements.

[0045] (1) Deterministic capabilities When the capability to be optimized has a clear input-output correspondence, and the model output can be determined to meet the requirements through standard examples or deterministic rules, it can be identified as a deterministic capability, also known as a disciplined capability. For example, generating responses according to a fixed format, extracting specified fields from information returned by tools, and performing confirmation operations under predetermined conditions can all be considered deterministic capabilities. For deterministic capabilities, supervised training samples consisting of model inputs and target outputs can be constructed, and the model parameters of the service agent can be updated using the difference between the model's predicted output and the target output.

[0046] Tool usage capabilities can include tool selection, parameter construction, failure retries, and rollback handling. When the tool's applicability conditions, parameter rules, and exception handling rules can be predetermined, tool call alignment methods can be used for training, and the success of the tool call, whether the tool selection meets the call conditions, and the validity of the call parameters can be used as automatically determined supervisory signals. When the tool selection, retry, or rollback path depends on multiple rounds of interaction content, user goals, and service status, and there is no unique processing action, the corresponding tool usage capabilities can also be classified as strategy-based capabilities.

[0047] (2) Strategic capabilities When the capability to be optimized does not have a unique target output, different service paths may all meet the basic requirements, and the behavioral effect needs to be evaluated in conjunction with the complete service trajectory, it can be identified as a strategic capability. For example, determining the execution order among clarifying requests, appeasing users, and invoking tools, or choosing an alternative service path after a user rejects the current solution, can all be considered strategic capabilities. For strategic capabilities, service tasks and task evaluation information can be generated, the service agent can be controlled to execute the same service task multiple times in an interactive simulation environment, and the relative advantages of each service trajectory can be determined using the result reward and process reward output by the reward model to update the model parameters of the service agent.

[0048] Capability type identification is performed before the reinforcement learning process begins. Supervised optimization and reinforcement learning correspond to different training paths, and supervised optimization is not a training step in the reinforcement learning process. For failure modes that contain both deterministic and policy-based capabilities, they can be broken down into multiple capability units to be optimized and fed into the corresponding training paths; alternatively, an initial model version can be formed through supervised optimization first, and then this initial model version can be used as the current service agent to perform policy-based capability training.

[0049] After the service agent acquires the corresponding service capabilities, capability compression can be implemented. Specifically, a validated large model can be used as the teacher model, and models with smaller parameter sizes can be used as student models. The service trajectories, interaction actions, or tool call results generated by the teacher model are used to perform policy distillation on the student models. When the service quality of the student models meets the preset quality requirements, the student models can be identified as candidate model versions to reduce inference computation and response latency.

[0050] Therefore, different types of capabilities to be optimized can be trained using training routes that match their evaluation methods.

[0051] IV. Training on Service Quality Evaluation System and Reward Model 4.1 Composition of Service Quality Labels In one embodiment of this specification, a corresponding quality of service (QoS) label can be generated for a complete sample service trajectory. Please refer to [link / reference]. Figure 2 Service quality labels can include result-dimensional supervision labels, process-dimensional supervision labels, and service quality scores. Among them, result-dimensional supervision labels are used to describe the quality of sample service trajectories in terms of task execution results and service direction, process-dimensional supervision labels are used to describe the quality of sample service trajectories in terms of service specifications and coverage of required service items, and service quality scores are used to form trajectory-level comprehensive supervision labels for complete sample service trajectories.

[0052] A sample service trajectory can include multiple trajectory nodes and execution results. Each trajectory node can record interactive actions, tool calls, tool returns, or service status changes. When generating service quality tags, the evaluation can be combined with the sample task evaluation information corresponding to the sample service trajectory. The sample task evaluation information can include user objectives, acceptance criteria, and key node evaluation items, providing a reference for evaluating the sample service trajectory.

[0053] The outcome dimension supervision label can be determined based on outcome evaluation information and goal consistency information.

[0054] (1) Results evaluation information The result evaluation information is used to characterize the degree to which the execution result of the sample service trajectory meets the acceptance criteria. The result evaluation information can be determined based on information such as the service status at the end of the service task, tool return information, order or work order processing status, whether the specified operation was completed, whether it was transferred to manual processing, and whether duplicate service requests occurred. The execution result of the sample service trajectory can be matched with the acceptance criteria, and the corresponding evaluation result can be determined based on the matching result. For example, the evaluation result can take a value from zero to one, where a larger value indicates a higher degree of satisfaction with the acceptance criteria; when the acceptance criteria are directly determinable, the evaluation result can also take a discrete value of zero or one; however, it is not limited to this.

[0055] (2) Target consistency information Goal consistency information is used to characterize the degree of consistency between the actual service direction of a sample service trajectory and the user's goal. Semantic comparison can be performed between the user's goal and the interactive actions, tool calls, service status changes, and execution results in the sample service trajectory to determine whether the service direction formed by each trajectory event points to the problem the user needs to solve. The evaluation result corresponding to goal consistency information can take a value from zero to one, where a larger value indicates a higher degree of consistency between the actual service direction and the user's goal; however, it is not limited to this. For example, if each local response in the sample service trajectory conforms to the expression specification, but the executed tool calls and the given solutions do not point to the user's goal, a lower goal consistency value can be determined.

[0056] Results evaluation information and goal consistency information serve different evaluation purposes. Results evaluation information focuses on whether the task execution results meet the acceptance criteria, while goal consistency information focuses on whether the service process and execution results revolve around the user's goals. Therefore, it is possible to identify sample service trajectories where the execution results meet some acceptance criteria but the overall service direction deviates. The results dimension supervision label can be a structured label including results evaluation sub-items and goal consistency sub-items, or it can be a single numerical label formed based on the preset aggregation relationship between results evaluation information and goal consistency information. Specific settings can be configured according to the actual application scenario.

[0057] Process-dimensional monitoring labels can be determined based on service specification compliance assessment information and service coverage assessment information.

[0058] (3) Service standard compliance assessment information Service specification compliance assessment information is used to characterize whether trajectory events in sample service trajectories conform to preset service specifications. Preset service specifications may include evaluation requirements such as verifying information before responding, requesting user confirmation before performing specific operations, responding according to information returned by the tool, avoiding making promises without sufficient evidence, and asking necessary questions according to predetermined interaction methods. Rule-based evaluation models, language evaluation models, or a combination of both can be used to evaluate relevant trajectory nodes.

[0059] (4) Service coverage evaluation information Service coverage evaluation information is used to characterize whether the required service items corresponding to a service task are executed in the sample service trajectory. The types of key nodes that should be included when executing a service task can be determined based on the key node evaluation items, and these key node types are matched with trajectory nodes in the sample service trajectory. Service coverage evaluation information can include matched key node types, unmatched key node types, and the coverage status corresponding to each key node type.

[0060] Service specification compliance assessment information and service coverage assessment information are used to evaluate the execution quality and completeness of the service process, respectively. Sample service trajectories may cover all required service items, but some trajectory events may not conform to service specifications; conversely, all executed trajectory events may conform to service specifications, but necessary service items may be missing. Therefore, process-dimensional monitoring labels can simultaneously record service specification compliance assessment results and service coverage assessment results to distinguish between improper execution of trajectory events and missing required trajectory events.

[0061] For example, the process dimension supervision labels can adopt structured labels and include node process supervision labels that are respectively associated with multiple key sample nodes in the sample service trajectory. The node process supervision labels, as a refinement of the process dimension supervision labels, are used to describe the process quality of the corresponding key sample nodes at the node level.

[0062] A sample key node can be a trajectory node in the sample service trajectory that matches the key node type indicated by the key node evaluation item. Service specification compliance evaluation information and service coverage evaluation information can each include multiple evaluation details. Each evaluation detail can record the corresponding key node type, evaluation conditions, evaluation results, and the matched sample key nodes. Based on the correspondence between the evaluation details and sample key nodes, the node process supervision labels associated with the corresponding sample key nodes in the process dimension supervision labels can be determined.

[0063] Based on the event attributes of the trajectory events recorded by the key nodes in the samples, these key nodes can be categorized into service path types, tool call types, or interaction action types. Service path type key nodes can correspond to service step selection, service step sequence, service status transition, or alternative path selection; tool call type key nodes can correspond to tool selection, call conditions, call parameters, tool return processing, failure retries, or rollback processing; and interaction action type key nodes can correspond to request clarification, user confirmation, information response, result restatement, or solution explanation.

[0064] For key nodes in service path samples, the corresponding node process supervision labels can be determined based on whether the node is located within a service path appropriate to the user's goal, whether the corresponding service task has been completed, and whether the state transition between preceding and following nodes meets the evaluation requirements. For key nodes in tool call samples, the corresponding node process supervision labels can be determined based on whether the tool selection is appropriate, whether the call parameters conform to parameter rules, whether the tool's returned information is correctly processed, and whether the retries or rollbacks after failures meet the evaluation requirements. For key nodes in interactive action samples, the corresponding node process supervision labels can be determined based on whether the interactive action conforms to service specifications, whether necessary clarifications or confirmations have been completed, and whether the response content is consistent with the information already obtained.

[0065] The process supervision labels for multiple nodes in the process dimension can be categorized and aggregated according to the category to which each key node belongs, resulting in classification evaluation results for service path, tool call, and interaction action classes. These evaluation results are then aggregated to determine the label values ​​for the process dimension supervision labels. For key node types indicated by the key node evaluation items but not matched in the sample service trajectory, a corresponding key node missing penalty can be recorded in the process dimension supervision labels. This penalty is deducted when determining the label values ​​for the process dimension supervision labels, preventing the generation of process supervision labels for nodes that do not correspond to actual trajectory nodes.

[0066] In one implementation, the evaluation results corresponding to the result evaluation information, target consistency information, service specification compliance evaluation information, and service coverage evaluation information can be respectively denoted as: , , and The service quality score is determined using the following method:

[0067] in, Indicates service quality score. Indicates the compliance threshold. , and These represent the weights corresponding to the result evaluation information, service specification compliance evaluation information, and service coverage evaluation information, respectively, and satisfy the following conditions:

[0068] When the sample service trajectory does not violate any veto compliance constraints, the compliance threshold can be set to one; when the sample service trajectory violates any veto compliance constraint, the compliance threshold can be set to zero, resulting in a service quality score of zero. The evaluation results corresponding to the goal consistency information can be used as a multiplicative gating factor to adjust the contribution of result evaluation, service specification compliance evaluation, and service coverage evaluation to the service quality score. When the actual service direction deviates from the user's goal, even if other evaluation results are high, the service quality score will decrease accordingly.

[0069] The aggregation method described above is only one implementation. In other embodiments, weighted aggregation or gated aggregation can be performed based on the label values ​​corresponding to the result-dimensional and process-dimensional supervision labels to obtain the service quality score. Before aggregation, each evaluation result can be normalized to the same value range to reduce the dimensional differences caused by different evaluation methods. This embodiment does not impose any restrictions on the aggregation method.

[0070] 4.2 Reward Model Training Please see Figure 3 as well as Figure 4 In one embodiment of this specification, the model training device may execute steps S301 to S304 to supervise the training of the reward model to be trained using sample service trajectories, sample task evaluation information, and service quality labels. The reward model to be trained may be a service domain evaluation model built based on a pre-trained language model, or it may be a sequence evaluation model capable of processing structured trajectory data and text data.

[0071] In S301, the sample service trajectory, the corresponding sample task evaluation information, and the service quality label are obtained.

[0072] A sample service trajectory can be a complete trajectory formed by a service agent when executing historical or simulated service tasks, including multiple trajectory nodes and execution results. Each trajectory node can record interactive actions, tool calls, tool returns, or service status changes, as well as the order of occurrence, time information, and relationships between preceding and following nodes.

[0073] The sample task evaluation information may include user objectives, acceptance criteria, and key node evaluation items. User objectives describe the problem that the corresponding service task needs to solve; acceptance criteria describe the state or conditions that the execution result should meet; key node evaluation items indicate the types of key nodes that the sample service trajectory should include when executing the service task and the corresponding evaluation requirements.

[0074] Service quality labels can include outcome-level monitoring labels, process-level monitoring labels, and a service quality score. Outcome-level monitoring labels can be determined based on the outcome evaluation information and target consistency information of the sample service trajectory; process-level monitoring labels can be determined based on the service specification compliance evaluation information and service coverage evaluation information of the sample service trajectory; and the service quality score can be obtained by aggregating the outcome-level monitoring labels and process-level monitoring labels.

[0075] Process-dimensional supervision labels can include node process supervision labels associated with multiple key nodes in the sample service trajectory. Node process supervision labels can be determined based on the correspondence between the evaluation details in service specification compliance assessment information and service coverage assessment information and the corresponding key nodes in the samples. As a refinement of the process-dimensional supervision labels, node process supervision labels are used to supervise the reward model at the node level.

[0076] Based on trajectory and task identifiers, sample service trajectories, sample task evaluation information, and service quality labels can be associated. Before training, the values ​​of each label can be normalized to ensure that the result-dimensional supervision labels, process-dimensional supervision labels, node process supervision labels, and service quality scores are in the same or comparable value range. Multiple training samples that have undergone association and normalization can form a reward model training dataset, which can be divided into training data subsets and validation data subsets according to a preset ratio.

[0077] In S302, the sample service trajectory and sample task evaluation information are input into the reward model to be trained to obtain the prediction result reward and the prediction process reward. The prediction result reward and the prediction process reward are then aggregated to obtain the prediction trajectory reward.

[0078] In one implementation, interactive actions, tool calls, tool returns, and service state changes can be serialized according to the order in which each trajectory node occurs in the sample service trajectory. The serialized sample service trajectory and sample task evaluation information are then input into the reward model to be trained. The reward model to be trained can extract the node representation of each trajectory node and form the trajectory representation of the sample service trajectory based on the node representation and the contextual relationships between nodes.

[0079] The reward model to be trained can include a shared feature extraction part, a result reward output part, and a process reward output part. The shared feature extraction part is used to extract the correlation features between the sample service trajectory and the sample task evaluation information; the result reward output part is used to output the predicted result reward based on the trajectory representation; and the process reward output part is used to output the predicted process reward based on the trajectory representation and the representation of each node. Different output parts can share some model parameters or have independent output parameters.

[0080] The prediction result reward can include a prediction result acceptance reward and a prediction target consistency reward. The prediction result acceptance reward characterizes the degree to which the execution result of the sample service trajectory meets the acceptance criteria, while the prediction target consistency reward characterizes the degree to which the actual service direction of the sample service trajectory is consistent with the user's goal. Both the prediction result acceptance reward and the prediction target consistency reward can take continuous values ​​from zero to one.

[0081] The prediction process reward can include the prediction node process reward associated with each sample key node. The reward model to be trained can match the corresponding key node type with the trajectory nodes in the sample service trajectory according to the key node type indicated by the key node evaluation item, and output the prediction node process reward for the matched sample key nodes. The reward model to be trained can also aggregate multiple prediction node process rewards to obtain the prediction process reward corresponding to the complete sample service trajectory.

[0082] In one implementation, the basic trajectory reward can be determined based on the acceptance reward for the prediction result and the reward for the prediction process. The basic trajectory reward can be determined using a weighted sum, a weighted average, or other pre-defined aggregation method. For example, the basic trajectory reward can be calculated as follows:

[0083] in, Indicates the base trajectory reward. This indicates a reward for the acceptance of the prediction results. This indicates a reward for the prediction process. and These represent the corresponding aggregate weights.

[0084] The consistency reward of the predicted target can be used as a multiplicative gating factor to adjust the gating of the base trajectory reward, thus obtaining the predicted trajectory reward. For example, it can be calculated as follows:

[0085] in, Indicates the reward for predicting the trajectory. This represents the reward for consistency with the prediction target. A higher reward for consistency with the prediction target indicates a greater retention of the base trajectory reward within the predicted trajectory reward; conversely, a lower reward for consistency with the prediction target indicates a greater suppression of the base trajectory reward. Therefore, this can reduce the predicted trajectory reward for sample service trajectories where the actual service direction deviates from the user's goal but the local execution result or service process evaluation is high.

[0086] In one implementation, compliance checks can also be performed on the sample service trajectory based on constraint information related to service compliance. When at least one trajectory node in the sample service trajectory records a trajectory event that violates a veto-compliant constraint, the compliance threshold can be set to zero, and the predicted trajectory reward can be set to zero. The corresponding predicted trajectory reward can be expressed as:

[0087] in, This represents the compliance threshold. The compliance threshold can be one when the sample service trajectory does not violate any veto compliance constraints; it can be zero when the sample service trajectory violates any veto compliance constraint. Compliance detection can be performed by the compliance identification part in the reward model to be trained, or by a rule verification module independent of the reward model to be trained.

[0088] In S303, the joint training loss is determined based on the differences between the prediction result reward and the result dimension supervision label, the prediction process reward and the process dimension supervision label, and the prediction trajectory reward and the service quality score.

[0089] The predicted result reward and the result dimension supervision label can be input into a preset result loss function to determine the result dimension training loss. When the result dimension supervision label includes a result evaluation sub-item and a target consistency sub-item, the difference between the predicted result acceptance reward and the result evaluation sub-item, and the difference between the predicted target consistency reward and the target consistency sub-item can be calculated separately. Then, the losses corresponding to the two differences are weighted and aggregated to obtain the result dimension training loss.

[0090] The training loss for the process dimension can be determined based on the difference between the predicted process reward and the process dimension supervision label. When the process dimension supervision label uses continuous numerical labels, the corresponding difference can be calculated using squared error loss, absolute error loss, or Hubble loss; when the process dimension supervision label includes discrete evaluation results, the corresponding difference can be calculated using cross-entropy loss.

[0091] In one implementation, when the process dimension supervision label includes multiple node process supervision labels, the node process supervision labels can be used to perform node-level supervision on the reward model to be trained. Based on the event attributes represented by the key nodes of each sample, the key nodes of each sample can be divided into service path classes, tool call classes, or interactive action classes.

[0092] Key nodes in service path samples can correspond to service step selection, service step sequence, service state transition, or alternative path selection. The training loss for service path nodes can be determined based on the difference between the predicted node process reward and the node process supervision label corresponding to each key node in the service path samples.

[0093] Key nodes in tool invocation samples can correspond to tool selection, invocation conditions, invocation parameters, tool return handling, and failure retry or rollback handling. The training loss for tool invocation nodes can be determined based on the difference between the predicted node process reward and the node process supervision label corresponding to each key node in the tool invocation sample.

[0094] Key nodes for interactive action samples can correspond to clarification of requests, user confirmation, information response, result restatement, or solution explanation. The training loss for interactive action nodes can be determined based on the difference between the predicted node process reward and the node process supervision label corresponding to each key node.

[0095] For any category, the node loss between the predicted node process reward and the node process supervision label for each key node of that category can be calculated separately. The node losses within the same category can then be summed, averaged, or weighted to obtain the node training loss for that category. If no key nodes of a certain category exist in a training batch, the node training loss for that category can be omitted from that training batch.

[0096] The training loss of service path nodes, tool call nodes, and interactive action nodes can be weighted and aggregated, and the aggregated result can be used as the training loss of the process dimension.

[0097] When the label values ​​of the process dimension supervision labels are directly obtained by aggregating the process supervision labels of multiple nodes, the process dimension training loss can be determined using the node training loss corresponding to each category. For key node types indicated by the key node evaluation terms but not matched in the sample service trajectory, the process reward can be predicted by the key node missing penalty constraint in the process dimension supervision labels, instead of constructing a predicted node process reward for non-existent sample key nodes.

[0098] The trajectory-level regression loss can be calculated based on the difference between the predicted trajectory reward and the service quality score. In one implementation, the difference between the predicted trajectory reward and the service quality score can be squared to obtain the trajectory-level regression loss. When the service quality labels may contain evaluation noise, Hubel loss or absolute error loss can also be used to reduce the impact of a small number of outlier labels on model parameter updates.

[0099] The training loss can be weighted and aggregated from the result dimension training loss, the process dimension training loss, and the trajectory-level regression loss to obtain the joint training loss. The weights corresponding to each training loss can be set according to the number of samples with different labels in the training data, the range of values, and the training stability. The result dimension training loss is used to constrain the prediction result reward, the process dimension training loss is used to constrain the prediction process reward and the prediction node process reward, and the trajectory-level regression loss is used to constrain the aggregated result of the prediction result reward and the prediction process reward.

[0100] In S304, the model parameters of the reward model to be trained are updated according to the joint training loss to obtain the reward model.

[0101] Specifically, the joint training loss can be backpropagated, and the model parameters of the shared feature extraction part, the result reward output part, and the process reward output part can be updated according to the gradient of the joint training loss relative to the model parameters of the reward model to be trained, so that the predicted result reward, the predicted process reward, and the predicted trajectory reward output by the reward model to be trained are close to the corresponding supervision labels.

[0102] Batch training can be used, where multiple sample service trajectories are input into each training batch, the corresponding joint training loss is calculated, and one or more model parameter updates are performed. In one implementation, all model parameters of the reward model to be trained can be updated. In another implementation, some pre-trained model parameters can be fixed, and only the result reward output, the process reward output, or the parameter tuning layer set in the reward model to be trained can be updated, thereby reducing the computational resources required for training.

[0103] During training, the training loss in the result dimension, the training loss in the process dimension, and the trajectory-level regression loss can be calculated using a subset of validation data. The model training status can then be determined based on the joint training loss corresponding to the subset of validation data. When the joint training loss reaches a preset loss condition, or when the change in validation results over multiple consecutive training rounds is less than a preset change threshold, model parameter updates can be stopped, and the current model can be designated as the completed reward model.

[0104] Through joint training with trajectory-level regression supervision, outcome-dimensional supervision, and process-dimensional supervision, the reward model can learn the correspondence between complete sample service trajectories and service quality scores, as well as the correspondence between key nodes of different categories of samples and node process supervision labels. The trained reward model can be used to process newly generated service trajectories by the service agent and output outcome rewards and process rewards.

[0105] 4.3 Reward Model Calibration Based on Real User Feedback In one implementation, the service quality score in the service quality label can be calibrated using real user feedback corresponding to the sample service trajectory, and the calibrated service quality score can be used to update the reward model. This calibration process is optional and can be performed after the reward model has completed its initial training, or after a preset number of real user feedback samples have been accumulated.

[0106] The service quality score generated based on outcome evaluation information, goal consistency information, service specification compliance evaluation information, and service coverage evaluation information may deviate from users' evaluation of the actual service effect. For example, the sample service trajectory may have resolved the user's problem, but due to low scores on some process evaluation items, the service quality score is lower than the user's actual evaluation; or, the sample service trajectory may formally meet the acceptance conditions and service specifications, but the user may subsequently repeatedly raise the same request, refuse the solution, or file a complaint, causing the service quality score to be higher than the actual service effect. Real user feedback can be used to identify and correct such deviations.

[0107] User feedback can be linked to corresponding sample service trajectories based on trajectory identifiers, service session identifiers, order identifiers, or work order identifiers. User feedback can include satisfaction ratings, evaluation texts, operational results of accepting or rejecting handling solutions, as well as feedback information such as repeated service requests, complaints, cancellations, or transfers to manual processing that occur within a preset time range.

[0108] Since user feedback may be affected by misoperation, abnormal states, or irrelevant factors, it's possible to determine whether genuine user feedback meets preset credibility conditions. These preset credibility conditions may include: genuine user feedback originates from an authenticated service session; genuine user feedback can be uniquely associated with a sample service trajectory; genuine user feedback occurs within a preset time range after the end of the corresponding service trajectory; genuine user feedback does not involve duplicate submissions, abnormal frequency, or missing data; and the service item indicated by the genuine user feedback is consistent with the service item processed by the sample service trajectory. For evaluation text, the evaluation object, evaluation direction, and evaluation confidence level can also be identified. When the evaluation object is clearly defined and the evaluation confidence level reaches a preset threshold, the evaluation text is determined to be credible feedback.

[0109] When genuine user feedback meets preset credibility conditions, feedback evaluation values ​​can be determined based on the genuine user feedback. For structured feedback such as satisfaction levels and acceptance or rejection of handling plans, it can be converted to the same value range as the service quality score according to a preset mapping relationship. For feedback such as evaluation text, repeated service requests, or complaints, feedback evaluation values ​​can be determined using feedback evaluation models or preset rules. When the same sample service trajectory corresponds to multiple genuine user feedbacks, weights can be set according to feedback type, credibility, and time of occurrence, and multiple feedback evaluation results can be aggregated.

[0110] The deviation between the feedback evaluation value and the service quality score can be determined. Preset calibration conditions can include the deviation reaching a preset deviation threshold, or the feedback evaluation value and the service quality score indicating opposite evaluation directions. If the deviation does not meet the preset calibration conditions, the original service quality score can be retained, and calibration can be skipped.

[0111] If the deviation meets the preset calibration conditions, the service quality score can be corrected based on the feedback evaluation value to obtain a calibrated service quality score. For user feedback with high credibility, the feedback evaluation value can replace the original service quality score; alternatively, the feedback evaluation value and the original service quality score can be weighted and aggregated based on the credibility of the user feedback. A single calibration range can also be set to ensure that the change in the calibrated service quality score relative to the original service quality score does not exceed a preset range, thereby reducing the excessive impact of individual user feedback.

[0112] After obtaining the calibration service quality score, the sample service trajectory and the corresponding sample task evaluation information can be input into the reward model. The calibration service quality score is used as the trajectory-level supervision label. The model parameters of the reward model are updated according to the difference between the predicted trajectory reward output by the reward model and the calibration service quality score.

[0113] By screening reliable user feedback that has a clear correspondence with sample service trajectories and using the feedback evaluation values ​​to calibrate the service quality score, the systematic deviation between the preset evaluation method and the actual service evaluation by users can be reduced. The updated reward model enables the trajectory rewards output for newly generated service trajectories to more closely reflect the actual service effect represented by reliable user feedback, thereby providing an evaluation basis for the relative comparison of subsequent service trajectories and the updating of model parameters of the service agent.

[0114] V. Service Tasks and Environment Construction for Self-Evolution of Service Agents 5.1 Service task generation based on online failure mode In one embodiment of this specification, an online service trajectory generated during the online service process that meets a preset low-quality condition can be identified as a failed service trajectory. The preset low-quality condition may include a service quality score below a preset threshold, being in a low-quality range for multiple consecutive evaluation periods, or hitting a preset failure event.

[0115] Based on the user requests, service recipients, tool call chains, and execution results in the failed service trails, the service scenario and problem type to which each failed service trail belongs can be determined. The service scenario can represent the service link where the failure occurred, and the problem type can represent the reasons for failure, such as failure to achieve the result, deviation from the service direction, missing required service items, abnormal tool calls, or non-compliant interaction actions.

[0116] Clustering failed service trajectories according to service scenarios and problem types can yield multiple clusters of failed trajectories. Clustering can be performed using preset classification rules, or by extracting semantic features, trajectory node features, and service status features of the failed service trajectories, and then clustering based on the similarity between these features.

[0117] A cluster of failure trajectories whose number of failed service trajectories reaches a preset threshold, or whose frequency of occurrence within a preset time range reaches a preset frequency threshold, can be identified as a high-frequency failure trajectory cluster. For each high-frequency failure trajectory cluster, the user target, abnormal trajectory nodes, state changes, and execution results of each failed service trajectory can be compared, and the failure characteristics shared by multiple failed service trajectories can be identified as the corresponding failure mode.

[0118] For each failure mode, the initial service context, callable tool information, initial service status, user objectives, acceptance criteria, and key node evaluation items can be extracted from the corresponding failure service trajectory and system real-state data. Then, a service task and corresponding task evaluation information can be generated using a preset task template, a task generation model, or a combination of both. The system real-state data may include the order status, work order status, tool availability status, and other status data related to service execution at the time the failure service trajectory occurred.

[0119] Service tasks and task evaluation information are two related but functionally distinct types of data. Service tasks may include the initial service context and information about callable tools, used to drive the service agent to execute tasks in an interactive simulation environment. Task evaluation information may include user objectives, acceptance criteria, and key node evaluation items, used to support the reward model in evaluating the service trajectory formed by the service agent in executing the service task. Service tasks and task evaluation information can be linked by the same task identifier.

[0120] The service task does not provide a standard answer or a pre-determined service path. The service agent selects interaction actions or tool calls only based on the service task and information from the interactive simulation environment. Task evaluation information may not be provided to the service agent but is instead provided to the reward model after trajectory generation. Therefore, the same service task can be executed multiple times by the same model version or by different model versions to generate service trajectories with different behavioral paths.

[0121] The generated task set can possess the properties of repeatability, comparability across different scenarios, and incremental expansion. Repeatability means that the same service task can be executed multiple times and multiple service trajectories can be sampled; comparability across different scenarios means that different model versions can be executed on the same task set and the trajectory evaluation results can be compared; incremental expansion means that new failure modes that occur during online service can continuously generate new service tasks and task evaluation information and add them to the task set. Thus, recurring failure problems online can be transformed into training tasks that can be repeatedly executed and evaluated, providing a task foundation for the subsequent self-evolution of the service agent.

[0122] 5.2 Construction of the Interactive Simulation Environment Please see Figure 5 In one embodiment of this specification, the interactive simulation environment may include a simulated user module, a simulated tool module, and a business state machine constructed based on real historical trajectories and tool call specification information, for providing the service agent with a repeatable user interaction, tool call, and service state change environment.

[0123] The real-world historical trajectory can include user input, interactive actions, tool calls, tool returns, service status changes, and execution results generated during the historical service process. When building an interactive simulation environment, user expression patterns, user response patterns, and service status change patterns under different service scenarios can be extracted from the real-world historical trajectory.

[0124] The tool invocation specification information describes the invocation conditions, input parameter rules, return information rules, and exception handling rules of the tool. Invocation conditions may include the service status that allows the tool to be invoked and the prerequisite operations; input parameter rules may include required parameters, parameter types, and parameter value ranges; return information rules may include the data structure and status identifier for normal returns; and exception handling rules may include the corresponding exception results for parameter errors, tool unavailability, call timeouts, or missing return information.

[0125] The simulated user module generates user interaction information based on real historical trajectories to interact with the service agent. When executing service tasks, the simulated user module can generate user responses, supplementary requests, and acceptance or rejection of the proposed solutions, based on the initial service context, user goals, current service status, and interactive actions generated by the service agent. The simulated user module can be implemented using a rule-based model, a generative model, or a combination of both.

[0126] The simulation tool module generates tool return information corresponding to tool calls based on tool call specification information. When a tool call meets the callability conditions and the input parameters conform to the input parameter rules, the simulation tool module can generate tool return information that conforms to the return information rules, taking into account the current service status. When a tool call does not meet the corresponding conditions, the simulation tool module can generate abnormal results such as parameter errors, call failures, call timeouts, or missing return information according to the exception handling rules.

[0127] The business state machine records the current service state and updates it based on interactive actions generated by the service agent, tool calls, and tool return information generated by the simulation tool module. Service states can include pending clarification, pending query, pending confirmation, processing, processing successful, processing failed, or service terminated. The business state machine determines the next service state based on the current service state and preset state transition conditions, and provides the updated service state to the service agent.

[0128] When executing a service task, the simulated user module can be initialized using the initial service context, and the business state machine can be initialized using the initial service state. The service agent generates interactive actions or tool calls based on the service task; the simulated user module or simulated tool module generates corresponding return information; the business state machine updates the service state based on the interactive actions, tool calls, and tool return information generated in this round. The above process can be executed cyclically until the service state reaches the task completion state, task failure state, or task termination state.

[0129] The interactive simulation environment can record interactive actions, user interaction information, tool calls, tool return information, and service status changes in the order of trajectory events, forming a service trajectory that includes multiple trajectory nodes and execution results. By repeatedly initializing the interactive simulation environment, the same service task can be executed multiple times, providing trajectory data for subsequent trajectory evaluation and service agent model parameter updates.

[0130] 5.3 Configuration and Execution Verification of Service Constraints In one embodiment of this specification, the interactive simulation environment includes a constraint verification module. The constraint verification module is configured with service constraint information related to the service domain to which the service task belongs. The service constraint information may include service operation constraints, service compliance constraints, and task termination constraints. Specifically, service operation constraints limit the tool's calling permissions, calling conditions, parameter ranges, and allowed handling methods; service compliance constraints limit the specifications that the interactive content and handling behavior should meet; and task termination constraints describe the service status, execution result, or exit conditions that should be met when the service task ends.

[0131] Service constraint information can be represented using rule tables, conditional expressions, parameter thresholds, state transition conditions, or combinations thereof, and can be configured according to service domain, task type, or service scenario. For example, service operation constraints can limit specific tools to be invoked only when the service status meets preset conditions, or limit tool invocation parameters to be within the corresponding value range; service compliance constraints can limit the generation of commitment content lacking supporting information, or the execution of actions beyond the current service authority; task termination constraints can limit task termination when the order status reaches the target status, the user request has been processed, manual processing is required, or continued interaction no longer meets the processing conditions.

[0132] During the execution of service tasks, the service agent can generate instructions to be executed based on the initial service context, user interaction information, tool return information, and the current service status. These instructions can include interactive action instructions or tool invocation instructions. Interactive action instructions are used to generate responses, queries, or confirmations to be output to the simulated user; tool invocation instructions are used to instruct the interactive simulation environment to invoke the corresponding simulation tool, and can carry information such as the tool identifier, invocation parameters, and expected operation type.

[0133] Before executing any instruction, the interactive simulation environment sends it to the constraint verification module. The constraint verification module parses the instruction type, operation object, and instruction parameters corresponding to the instruction, and then invokes the service operation constraints and service compliance constraints that match the current service task for verification. Verification methods may include rule matching, threshold comparison, permission verification, parameter format verification, state precondition verification, or semantic compliance detection.

[0134] When an instruction to be executed satisfies the corresponding service constraints, the constraint verification module outputs execution permission information. The interactive simulation environment executes the instruction and acquires user interaction information, tool return information, or service status change information. When an instruction to be executed violates the corresponding service constraints, the constraint verification module refuses to execute the instruction and generates a rejection message. The rejection message may include a rejection identifier, the type of constraint violated, the reason for rejection, and a prompt allowing the instruction to be regenerated. The interactive simulation environment records the rejected instruction and the corresponding rejection message as trajectory events in the trajectory nodes of the service trajectory. Therefore, instructions that violate constraints will not cause the simulation tool to execute or the service status to change. Simultaneously, the rejection process can serve as traceable trajectory information for the reward model to evaluate the instruction generation process of the service agent.

[0135] The constraint verification module can also read the current service status after each round of interaction, tool return, or service status update, and check whether the current service status meets the task termination conditions based on the task termination constraints. When the task termination conditions are not met, the interactive simulation environment provides the updated service status to the subsequent execution process. When the task termination conditions are met, the constraint verification module generates a task termination signal, causing the interactive simulation environment to stop receiving subsequent pending instructions generated by the service agent for the current service task, and determines the execution result of the service trajectory based on the service status at the time of termination. The execution result can be used to indicate task completion, task failure, transfer to manual processing, exit due to constraints, or other preset termination status.

[0136] By performing constraint verification before the actual execution of the instruction to be executed, it is possible to prevent instructions that do not meet the operating conditions or compliance requirements from changing the service state in the interactive simulation environment; by recording rejected instructions and rejection information, the behavioral basis of the service agent under constrained scenarios can be preserved, providing trajectory data for subsequent process evaluation and parameter updates; by combining service state execution task termination detection, it is possible to reduce the situation where invalid interactions or tool calls are generated after the task reaches the end state, and to make the execution result of the service trajectory correspond to the service state at the time of termination.

[0137] VI. The Self-Evolution Process of Service Agents Please see Figure 6 as well as Figure 7 In one embodiment of this specification, a self-evolution method for service agents is provided. This method can be executed by a training server, computing cluster, or other electronic devices with model training capabilities. The service agent generates multiple service trajectories in an interactive simulation environment based on service tasks. A pre-trained reward model evaluates each service trajectory based on task evaluation information, and then updates the service agent's model parameters using the relative advantages between the trajectories, thereby forming a reinforcement learning closed loop from task execution and trajectory evaluation to parameter updating.

[0138] In S601, the service task and its corresponding task evaluation information are obtained. The service task includes the initial service context and the information of the callable tools. The task evaluation information is used to describe the user's goals, acceptance conditions and key node evaluation items.

[0139] Service tasks and task evaluation information are two related but different types of data. Service tasks provide the service agent with the input needed to perform tasks, while task evaluation information provides the basis for the reward model to evaluate the service trajectory. The two can be linked through task identifiers, scenario identifiers, or other related identifiers.

[0140] The initial service context describes the information that the service agent can obtain at the start of a service task. This can include user input, historical interactions, service object status, executed operations, and other information related to the current service request. The callable tool information limits the tools that the service agent can use to execute tasks. This can include tool identifiers, tool function descriptions, input parameter requirements, and invocation conditions.

[0141] User objectives describe the target state that the user expects to achieve through this service; acceptance criteria describe the evaluation conditions that the execution result or process of the service task should meet; key node evaluation items describe the types of trajectory nodes that should be paid attention to when executing the service task and their evaluation content. For example, key node evaluation items may indicate whether an information query was performed before replying to the user, whether the tool call parameters meet the requirements, and whether the information repeated to the user is consistent with the information returned by the tool.

[0142] In one implementation, the service task is provided to the service agent, but the task evaluation information is not used as direct input for the service agent to generate interactive actions; it is only provided to the reward model during the trajectory evaluation stage. This reduces the need for the service agent to directly generate service trajectories that superficially meet evaluation requirements based on acceptance criteria, allowing the task evaluation information to maintain a relatively independent evaluation role. In other implementations, the parts of the user goal that need to be perceived by the service agent can be written into the initial service context, while the corresponding user goal is retained in the task evaluation information for evaluation by the reward model.

[0143] In S602, the control service agent executes service tasks multiple times in the interactive simulation environment to obtain a trajectory group consisting of multiple service trajectories. Each service trajectory includes multiple trajectory nodes and execution results. The trajectory nodes are used to record any trajectory event among interactive actions, tool calls, tool returns, and service state changes.

[0144] At the start of each execution, the interactive simulation environment can be initialized according to the initial service context in the service task, and callable tool information can be provided to the service agent. The service agent generates interactive actions or tool calls based on the current context. The interactive simulation environment responds to interactive actions by generating user interaction information, or responds to tool calls by generating tool return information, and updates the corresponding service status. The updated user interaction information, tool return information, and service status can continue to serve as the basis for the service agent to generate subsequent trajectory events until the current service task ends.

[0145] Service agents can use a sampling strategy that includes randomness to generate interactive actions or tool calls, allowing the same service task to form different service paths during multiple executions. Multiple executions can use the same version of the service agent and the same initial service conditions, thus giving each service trajectory within a trajectory group a common task basis while preserving differences in action selection, tool call order, or interaction paths.

[0146] During execution, multiple trajectory nodes can be generated according to the order in which trajectory events occur. Each trajectory node can record a node identifier, trajectory event type, event content, service status before and after the event, and its association with preceding and following trajectory nodes. The execution result of the service trajectory is used to represent the result status formed when this task execution ends, such as task completed, task incomplete, transferred to manual processing, or exited due to preset conditions. Thus, a service trajectory can form a complete structured record from the initial service context to the execution result.

[0147] Multiple service trajectories generated for the same service task are grouped together according to the task identifier. Each service trajectory within a trajectory group can have the same task evaluation information, allowing subsequent evaluations and comparisons to be conducted based on a common evaluation criterion.

[0148] In S603, the service trajectories and task evaluation information are input into the pre-trained reward model to obtain the result reward and process reward of the service trajectory. The result reward is related to the user's goal and acceptance conditions, while the process reward is related to the evaluation items of key nodes.

[0149] The reward model can process each service trajectory within a trajectory group separately. The service trajectories input into the reward model can be serialized according to the order in which trajectory nodes occur, preserving the correspondence between interactive actions, tool calls, tool returns, service status changes, and execution results. Task evaluation information can be input into the reward model along with the serialized service trajectories, enabling the model to identify the task execution status reflected by the service trajectories under the constraints of user objectives, acceptance criteria, and key node evaluation items.

[0150] Outcome rewards are used to evaluate task execution results from the perspective of the complete service trajectory, and their evaluation criteria are related to user goals and acceptance conditions. Process rewards are used to evaluate the trajectory content related to key node evaluation items during the execution of service tasks. They can reflect whether the service agent implemented the corresponding trajectory events in the process of forming the execution results, and whether the content of the corresponding trajectory events meets the evaluation requirements.

[0151] In S604, the result reward and process reward of the service trajectory are aggregated to obtain the trajectory reward of the service trajectory.

[0152] The result reward and process reward can be numerically transformed, scaled, or outlier-handled separately, enabling them to be aggregated within a unified reward space. The aggregation method can employ preset aggregation rules or a method determined through training. The trajectory reward can be represented as a scalar evaluation value corresponding to a complete service trajectory, comprehensively reflecting the overall performance of that service trajectory in terms of both task execution results and process.

[0153] (1) Process reward aggregation based on key node classification In one implementation, the critical node evaluation item describes the types of critical nodes that the service trajectory should include when performing a service task and their evaluation requirements. Examples include whether necessary verification was completed before a response, whether the tool selection and parameters were correct, whether the confirmation order was compliant, whether the restatement faithfully reflects the tool's return, and whether an alternative path was provided after a solution was rejected. The type of critical node can be described by the event type, location, target object, tool identifier, relationship between preceding and following nodes, or service status change conditions of the corresponding trajectory event.

[0154] The reward model matches key node evaluation items with trajectory nodes in the service trajectory and outputs node process rewards for actual key nodes that match the key node type. Node process rewards characterize the degree to which the trajectory events recorded by the corresponding key node conform to the key node evaluation items.

[0155] Electronic devices can categorize key nodes into service path, tool invocation, or interaction action categories based on the event attributes they represent. Service path key nodes characterize task execution phases, processing order, or service path transitions, such as whether necessary verification is completed before a response and whether an alternative path is switched when the current path is unavailable. Tool invocation key nodes characterize tool selection, invocation parameters, invocation timing, and the usage of tool return information. Interaction action key nodes characterize clarification, confirmation, explanation, information restatement, and other user-facing interactions.

[0156] The process rewards for key nodes are categorized and aggregated according to their respective categories because different categories of key nodes correspond to different evaluation objects and evaluation scales. Service path key nodes focus on evaluating the order and correlation between multiple trajectory events, tool call key nodes focus on evaluating the accuracy of tool calls, and interaction action key nodes focus on evaluating the interaction content. If all node process rewards were directly mixed, categories with a large number of nodes might have a greater impact on the process evaluation, making it difficult for categories with fewer nodes but more critical roles to be reflected in the total process reward.

[0157] Within each category, electronic devices can aggregate the process rewards of nodes belonging to that category using summation, averaging, weighted averaging, or other preset methods to obtain process rewards for service path categories, tool call categories, and interaction action categories. The aggregation method and node weights used for each category can be configured according to the importance of the nodes or evaluation requirements.

[0158] The electronic device then aggregates the process rewards for each category according to their respective category weights to obtain the total process reward. This total process reward is then aggregated with the result reward to obtain the trajectory reward for the service trajectory. By first performing intra-category aggregation and then inter-category aggregation, the impact of the number of nodes in different categories on process evaluation can be controlled. This allows the trajectory reward to reflect the evaluation results of service paths, tool calls, and interactive actions separately, and facilitates the identification of the impact of different types of service behaviors on the trajectory reward.

[0159] (2) Trajectory reward determination based on target consistency gating In one implementation, the outcome reward includes an outcome acceptance reward and a goal consistency reward. The outcome acceptance reward characterizes the degree to which the task execution result of the service trajectory meets acceptance criteria, such as whether the task is completed, whether the target state is achieved, or whether the predetermined handling result is formed. The goal consistency reward characterizes the degree of consistency between the actual service direction of the service trajectory and the user's goal. The actual service direction can be determined comprehensively based on the interaction content, tool calls, state changes, and execution results in the service trajectory.

[0160] Electronic devices can determine the basic trajectory reward based on the result acceptance reward and process reward of the service trajectory. When using the aforementioned key node classification and aggregation method, the total process reward and the result acceptance reward can be weighted and aggregated to obtain the basic trajectory reward. The basic trajectory reward is used to comprehensively represent the evaluation results of the service trajectory in terms of task execution results and execution process, but does not consider whether the service trajectory continuously revolves around the user's goal.

[0161] Electronic devices can convert the target consistency reward to a preset gating range and use it as a multiplicative gating factor to adjust the basic trajectory reward. When the target consistency reward is high, the basic trajectory reward is retained at a higher proportion; when the target consistency reward is low, the basic trajectory reward is correspondingly attenuated; when the target consistency reward drops to a preset lower limit, the basic trajectory reward can be suppressed to near zero. Therefore, the degree to which the basic trajectory reward is retained in the trajectory reward is positively correlated with the target consistency reward.

[0162] The reason for using multiplicative gating is that goal consistency is a holistic constraint on the entire service trajectory. If goal consistency rewards are only added as ordinary bonuses, higher result acceptance rewards or process rewards may offset the negative evaluation caused by goal deviation, allowing service trajectories that deviate from the user's goal to still receive higher trajectory rewards. For example, if the user's goal is to expedite the completion of a service task, but the service agent primarily interprets compensation rules, even if the interaction conforms to the service specifications and some operations are correct, the actual service direction of this trajectory may still be inconsistent with the user's goal.

[0163] By using the target consistency reward as a multiplicative gating factor, the target deviation can have an overall attenuation effect on the basic trajectory reward, reducing the compensation for directional deviation by local process performance or surface execution results. This allows the trajectory reward to simultaneously reflect the consistency of task execution results, execution process, and service direction, and reduces the positive reinforcement of directional deviation trajectory in subsequent relative advantage calculations.

[0164] (3) Trajectory rewards are reset to zero based on veto compliance constraints. In one implementation, the electronic device can perform compliance checks on each trajectory node in the service trajectory based on service compliance-related constraint information. This service compliance-related constraint information includes veto compliance constraints, which describe violations that cannot be compensated for by other evaluation results, such as exceeding operational authority, generating unsubstantiated commitments, or disclosing restricted information.

[0165] Compliance checks can employ methods such as rule matching, parameter comparison, permission verification, or semantic recognition to perform node-by-node checks on interactive actions, tool calls, tool returns, and service status changes recorded by trajectory nodes. When any trajectory event recorded by a trajectory node violates a veto-compliant constraint, the electronic device will set the trajectory reward for the corresponding service trajectory to zero. This zeroing process can override the trajectory reward obtained by aggregating result rewards and process rewards.

[0166] This mechanism provides a holistic evaluation after the service trajectory is formed, which differs from the constraint verification implemented by the interactive simulation environment before instruction execution. By setting veto compliance constraints, it avoids the negative evaluation of severely non-compliant trajectories from being offset by high outcome or process rewards, thus reducing the positive reinforcement of non-compliant behavior by subsequent parameter updates. The electronic device can also record the hit trajectory nodes, constraint types, and detection results to support subsequent trajectory analysis.

[0167] (4) Reward conflict correction based on counterfactual inference In one implementation, the electronic device can identify reward-conflicting service trajectories from a group of trajectories. A reward-conflicting service trajectory refers to a service trajectory where the outcome reward meets preset result requirements, but the process reward does not meet preset process requirements. Preset result requirements and preset process requirements can be represented by fixed thresholds, ranking positions within the trajectory group, or deviations relative to a comparison benchmark within the group.

[0168] A higher outcome reward and a lower process reward may indicate that while the service task produced an execution result that met the acceptance criteria, there were trajectory events during execution that did not meet the evaluation criteria for key nodes. It could also indicate that the process evaluation failed to accurately reflect the actual contribution of a certain trajectory event to the task execution result, or that the execution result depended on a chance environmental response. Directly aggregating outcome and process rewards according to preset aggregation weights may not distinguish between these different situations. Therefore, counterfactual inference can be performed on service trajectories with reward conflicts to estimate the causal contribution of relevant trajectory events to the task execution result, and the aggregation weights of outcome and process rewards can be adjusted accordingly.

[0169] The electronic device can identify at least one trajectory node to be replaced from reward-conflicting service trajectories, determine the trajectory events recorded by the trajectory node to be replaced as the original trajectory events, and replace the original trajectory events with candidate trajectory events. The trajectory node to be replaced can be determined based on lower node process rewards, unmet key node evaluation items, or the degree of anomalousness of the trajectory event. Candidate trajectory events can be extracted from service trajectories with higher process rewards in the same trajectory group, or they can be generated through pre-set trajectory event templates or resampling by the service agent. Candidate trajectory events can have the same event type and input conditions as the original trajectory events to reduce the influence of irrelevant variables on counterfactual evaluation.

[0170] When it is necessary to estimate the causal contribution of a single original trajectory event, each counterfactual inference can replace only one original trajectory event. When replacing multiple original trajectory events in the same counterfactual inference, the difference in results between the reward conflict service trajectory and the counterfactual service trajectory can characterize the joint causal contribution of multiple original trajectory events relative to their corresponding candidate trajectory events.

[0171] After the replacement is completed, the electronic device retains the service task, initial service context, and callable tool information unchanged, and preserves the trajectory content before the trajectory node to be replaced. Starting from the replacement position, the subsequent interaction process is re-enacted in the interactive simulation environment. The interactive simulation environment generates new tool return information, user interaction information, and service status change information based on the candidate trajectory events until the task ends, thus obtaining the counterfactual service trajectory.

[0172] The electronic device inputs the counterfactual service trajectory and its corresponding task evaluation information into the reward model to obtain counterfactual outcome rewards and counterfactual process rewards. The electronic device can calculate the difference between the outcome reward and the counterfactual outcome reward of the reward conflict service trajectory, and use the difference between the process reward and the counterfactual process reward of the reward conflict service trajectory as auxiliary evaluation information. Specifically, the difference in process rewards can be used to determine whether trajectory event replacement has changed the evaluation result of the corresponding execution process, but it does not change the direction of weight adjustment, which prioritizes outcome rewards.

[0173] Since the counterfactual service trajectory and the reward conflict service trajectory use the same service task, initial service context, and trajectory content before the trajectory node to be replaced, the electronic device only actively replaces the original trajectory events recorded by the trajectory node to be replaced with candidate trajectory events. This replacement causes corresponding changes in subsequent interaction content, tool calls, service status, and execution results. Therefore, the difference between the result rewards of the two service trajectories can be used as an estimate of the causal contribution of the replaced original trajectory event relative to the corresponding candidate trajectory event; in the case of replacing multiple original trajectory events, it can be used as an estimate of the joint causal contribution of the multiple replaced original trajectory events relative to the corresponding candidate trajectory events.

[0174] In one implementation, the result difference can be the result reward of the reward conflict service trajectory minus the result reward of the counterfactual service trajectory. The electronic device can determine whether the original trajectory event has a positive or negative impact on the task execution result based on the direction of the result difference, and characterize the degree of this impact based on the absolute value of the result difference. When the result difference is positive, the original trajectory event can be considered to have a positive contribution to the execution result relative to the candidate trajectory event; when the result difference is negative, the original trajectory event can be considered to have a negative contribution to the execution result relative to the candidate trajectory event; the larger the absolute value of the result difference, the greater the degree of influence of the original trajectory event relative to the candidate trajectory event on the task execution result.

[0175] Electronic devices can adjust the aggregation weight of outcome rewards and correspondingly decrease the aggregation weight of process rewards based on the degree of causal contribution represented by the absolute value of the outcome difference. Specifically, the larger the absolute value of the outcome difference, the greater the increase in the aggregation weight of outcome rewards and the greater the decrease in the aggregation weight of process rewards; conversely, the smaller the absolute value of the outcome difference, the smaller the adjustment to either type of aggregation weight. When the influence of process evaluation on trajectory rewards needs to be sufficiently weakened, the aggregation weight of process rewards can also be adjusted to a preset lower limit.

[0176] The electronic device re-aggregates the outcome rewards and process rewards of the reward conflict service trajectories according to the adjusted aggregation weights, resulting in updated trajectory rewards. This counterfactual evaluation does not perform undirected weight optimization between outcome and process rewards; instead, for reward conflict service trajectories with good outcomes but low process evaluations, it uses the task execution result as the primary evaluation criterion, weakening the influence of process rewards. This reduces the excessive suppression of the overall trajectory evaluation caused by lower process rewards when the process evaluation is inconsistent with the actual task execution result.

[0177] In S605, the trajectory rewards of each service trajectory within the trajectory group are compared relative to each other to obtain the relative advantages of each service trajectory.

[0178] A benchmark for comparison within a trajectory group can be determined based on the trajectory rewards of each service trajectory. This benchmark can be determined using the average, median, or ranking position of trajectory rewards, or other statistical results. By comparing the trajectory reward of each service trajectory with the benchmark, the relative advantage of that service trajectory compared to other service trajectories under the same task can be obtained.

[0179] When the trajectory reward is higher than the intra-group benchmark, the relative advantage can be positive; when the trajectory reward is lower than the intra-group benchmark, the relative advantage can be negative; when the trajectory reward is close to the intra-group benchmark, the relative advantage can be close to zero. The relative advantage can also be centralized or scaled to reduce the impact of different reward ranges between different trajectory groups on parameter updates.

[0180] Since each service trajectory within a group originates from the same service task and corresponds to the same task evaluation information, the relative comparison within the group can focus the evaluation on the performance differences between different execution paths, and reduce the impact of differences in the difficulty of different service tasks and differences in reward value scales on the training signal.

[0181] In S606, the model parameters of the service agent are updated based on the relative advantages of each service trajectory.

[0182] During reinforcement learning updates, the generation probabilities of interactive actions and tool calls in each service trajectory can be obtained, and the relative advantages of each service trajectory can be used as weights for parameter updates. For service trajectories with positive relative advantages, the probability of the service agent generating the corresponding interactive action or tool call in the given context can be increased; for service trajectories with negative relative advantages, the probability of generating the corresponding interactive action or tool call can be decreased. Thus, the service agent can learn relatively optimal action selection methods from different execution paths generated by the same service task.

[0183] Model parameters can include those used by the service agent to generate interactive actions and tool calls, as well as trainable adaptive parameters set in the model. Parameter updates can employ policy gradient updates, constrained magnitude updates, or other reinforcement learning update methods based on relative advantage. Constraints can also be placed on the magnitude of parameter updates, the range of policy changes, or the degree of shift in the generated distribution to reduce the disturbance to the service agent's original capabilities caused by a single training round.

[0184] After updating the model parameters, the updated service agent can be used for the next round of service trajectory generation. The next round can continue executing existing service tasks or newly added service tasks, repeating steps S602 to S606, thus forming a multi-round parameter update process. The updated service agent can also serve as a candidate model version for subsequent model version validation.

[0185] By using service tasks and task evaluation information to drive task execution and support trajectory evaluation respectively, unnecessary coupling between task execution information and evaluation criteria can be reduced. By generating multiple service trajectories for the same service task and conducting intra-group comparisons, relative advantage signals can be formed from the differences between different execution paths. By simultaneously utilizing result rewards and process rewards to form trajectory rewards, model parameter updates can take into account both task execution results and task execution processes. By feeding back the relative advantage to the model parameter update process of the service agent, service trajectories generated in the interactive simulation environment can be converted into reusable reinforcement learning data, enabling iterative updates of the service agent.

[0186] VII. Controlled Release and Regression of Candidate Service Agents After updating the model parameters of the service agent, the updated service agent can be identified as a candidate service agent, and a candidate model version can be generated. The candidate model version undergoes verification on a fixed evaluation set, shadow evaluation, and small-scale gray-scale testing in sequence. Once the corresponding release conditions are met, it is determined as the online effective version. If the verification fails at any stage, the release can be stopped or the previous online version can be rolled back.

[0187] 7.1 Fixed evaluation set non-backoff verification Electronic devices can control candidate model versions and the current online version to execute the same fixed evaluation set. The fixed evaluation set can include typical service scenarios, historical failure scenarios, boundary scenarios, and high-risk scenarios, and is configured with corresponding task evaluation information.

[0188] Under the same interactive simulation environment, tool configuration, and evaluation rules, electronic devices compare the task completion, service process evaluation, tool invocation accuracy, constraint satisfaction, and response performance of two model versions across key dimensions. When a candidate model version's evaluation results in all key dimensions are no lower than the current online version, it is allowed to enter shadow evaluation; if any key dimension regresses, the release of that candidate model version is stopped, and the corresponding service trajectory is used for model analysis or retraining.

[0189] 7.2 Shadow Evaluation During the shadow evaluation phase, electronic devices can replicate online service requests and send the same service request to both the current online version and the candidate model version. The processing result of the current online version takes effect on the user, while the processing result generated by the candidate model version is not output to the user and does not change the online service status; it is only used for offline comparison.

[0190] For tool calls that may cause state changes, tool return information can be generated by isolating the tool environment, using read-only tool interfaces, or simulating tool modules. Electronic devices compare the service trajectories formed by two model versions based on the same task evaluation information to detect whether the candidate model version experiences performance regressions not covered by a fixed evaluation set under real request distribution. When the shadow evaluation results meet the preset release conditions, the candidate model version enters a low-traffic gray-scale phase.

[0191] 7.3 Low-volume grayscale During the low-traffic gray-scale phase, electronic devices use a traffic allocation module to route a preset proportion of real service requests to the candidate model version, while the remaining service requests are handled by the current online version. The processing results generated by the candidate model version can then be applied to the corresponding users.

[0192] Candidate model versions can use the same tool configurations, service constraints, and operating parameters as the current online version. Requests are allocated randomly or hierarchically according to service scenarios to ensure comparability between the candidate and control groups. Electronic devices compare the task completion, service quality evaluation, abnormal exits, and response performance of the candidate and control groups. When a candidate model version meets the conditions for canary release within a preset observation period, its traffic ratio can be increased, or it can be designated as the new online effective version.

[0193] 7.4 Exception Rollback and Version Tracking Electronic devices can set rollback conditions based on the rollback magnitude, duration, or number of outlier samples for key metrics. When the rollback conditions are met during the gray-scale release, the traffic allocation module stops allocating new service requests to the candidate model version and switches the corresponding requests to the previous online version. Electronic devices can also mark the candidate model version as a version to be retrained and save the service requests that triggered the rollback, the service trajectory, and the evaluation results for subsequent parameter updates and regression validation.

[0194] Model versions and configuration versions can be managed separately. The model version identifies the model parameters of the service agent, while the configuration version identifies the prompt template, callable tool information, service constraint information, evaluation rules, and runtime parameters. For each candidate model version, the base model version, training data snapshot, reward model version, fixed evaluation set version, configuration version, and regression conclusions at each stage can be recorded, thereby supporting online result backtracking, version comparison, and anomaly localization.

[0195] Through the above phased verification, the effective scope of candidate model versions can be gradually expanded; combined with anomaly rollback and version records, when a candidate model version experiences performance regression, it can be restored to the previous online version and retain the data required for subsequent retraining.

[0196] 8. Online feedback data flywheel accumulation and multi-path reflux After the service agent goes online, electronic devices can continuously acquire online service trajectories, real user feedback, service quality evaluation results, and corresponding model version information, and store the above data in association with task identifiers and version identifiers. By filtering, organizing, and versioning the online data, training assets and evaluation assets can be formed for subsequent reward model training, service task generation, and model version verification.

[0197] In terms of hard-to-bear example mining, electronic devices can filter service trajectories based on the prediction confidence of the service agent when generating trajectory events, the service quality score of the service trajectory, and the relative comparison results within the group. Service trajectories whose model confidence meets a preset confidence requirement but whose service quality score meets a preset low-quality condition, or whose trajectory reward differs from the within-group benchmark by a preset difference requirement, can be identified as hard-to-bear examples. These service trajectories reflect scenarios where the service agent has high confidence in its own generated results, but the actual evaluation is low or significantly different from other trajectories in the same group, and can be used to supplement high-discrimination samples in subsequent training and regression validation.

[0198] Regarding the maintenance of gold standard lineage, electronic devices can establish sample lineage records for evaluation samples. These records can include the sample's source service trajectory, source model version, judgment body, judgment criteria, service rule caliber version, and the effective date of the corresponding rules. The judgment body can include deterministic evaluation procedures, human evaluators, or pre-trained evaluation models. By preserving sample lineage, the basis for evaluation results can be determined when evaluation rules or model versions change, and re-evaluation and version rollback of evaluation samples can be supported.

[0199] Regarding course playback, electronic devices can organize service tasks in layers based on task completion difficulty, number of interaction rounds, tool call complexity, number of key nodes, or historical failure frequency. The electronic devices adjust the sampling ratio based on the service agent's performance on tasks at each difficulty level. For example, when the service agent's evaluation results on low-difficulty tasks stabilize, the sampling ratio for higher-difficulty tasks can be increased; when a specific basic capability regresses, the playback ratio for corresponding low-difficulty tasks can be increased. Thus, the task sampling difficulty can be adjusted according to the service agent's capability status.

[0200] Regarding drift sampling, electronic devices can determine the online task distribution and the existing evaluation set distribution based on characteristics such as task type, user goal, tool usage method, and initial service status, and calculate the degree of distribution deviation between the two. When the distribution deviation meets the preset triggering conditions, drift samples are extracted from the corresponding online service trajectory, and service tasks, task evaluation information, or fixed regression sets are supplemented based on the drift samples to reduce the deviation between the evaluation set coverage and the online task distribution.

[0201] The aforementioned data can form multiple feedback paths. Low-quality service trajectories and corresponding high-frequency failure modes are fed back to the service task generation process to form the next round of service tasks and task evaluation information. Real user feedback that meets preset credibility conditions is fed back to the reward model calibration process to correct the corresponding service quality labels and update the reward model. Difficult-to-bearing examples, evaluation samples with sample lineage records, hierarchical task sets, and drifting samples can be included in training assets and fixed regression sets for subsequent service agent training, candidate model version verification, and version non-rollback determination.

[0202] Thus, the service trajectories, evaluation results, and abnormal samples generated during the controlled release process can enter the data accumulation process and flow back to the reward model training, service task construction, and fixed evaluation set verification, respectively. This allows the online results of a model update to serve as input for the next round of model training and version verification, forming a data loop across rounds.

[0203] IX. Verification Instructions To verify the technical effectiveness of this solution, the service agent obtained through the aforementioned self-evolutionary closed-loop training can be used as a candidate version, while the original general-purpose large model interface used online can be used as a control version for evaluation using the same criteria. Both the candidate and control versions use the same batch of 1000 fixed inputs and the same scoring criteria to reduce the impact of differences in input samples and evaluation criteria on the evaluation results, as shown in Table 1. Table 1

[0204] The evaluation results show that the candidate version outperformed the control version in all the above indicators. Notably, the increase in the pass rate was greater than the increase in the overall average score. Since the pass rate requires all evaluation dimensions of a service trajectory to simultaneously reach the corresponding threshold, this result indicates that the improvement in the candidate version is not merely due to a slight increase in the average score of multiple dimensions, but also reflects a reduction in the number of weak links that failed to reach the threshold.

[0205] The strategy compliance dimension improved by 5.1 points. This dimension mainly corresponds to process behaviors such as confirmation order, information verification, and action boundaries. This improvement is consistent with the mechanism of evaluating and providing feedback on specific process defects through process rewards and key node credit allocation. The security compliance dimension improved by 2.3 points, indicating that while the candidate version improved task execution and process performance, the compliance evaluation results did not regress. These comparative results illustrate that the self-evolutionary closed loop formed by repeated service task simulation, result and process reward evaluation, trajectory-level credit allocation, and relative advantage updates can improve the overall pass rate and process behavior performance in vertical domain service tasks.

[0206] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0207] Figure 8 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 8 As shown, device 800 mainly consists of a communication interface 802, a user interface 804, a processor 806, and a data storage 808. These components are interconnected and communicate with each other through a system bus, network, or other connection mechanisms. Figure 8The bus 810 is used as an example for illustration. Communication interface 802 enables device 800 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, communication interface 802 may include an antenna and associated processing devices for wireless communication with a radio access network or access point. Furthermore, communication interface 802 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, communication interface 802 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. Communication interface 802 may also include multiple physical communication interfaces, such as Wi-Fi interfaces, Bluetooth interfaces, and wide-area wireless interfaces.

[0208] User interface 804 includes receiving user input and providing output to the user. Therefore, user interface 804 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 804 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 804 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 800 may support remote access from other devices via communication interface 802 or another physical interface (not shown). User interface 804 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 804 may also be configured as a display device for rendering or displaying text fragments.

[0209] The processor 806 may contain one or more general-purpose processors and / or special-purpose processors.

[0210] Data storage 808 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 806. Data storage 808 may include removable and non-removable components.

[0211] Processor 806 is capable of executing program instructions 818 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 808 to perform the various functions described herein. Data storage 808 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 800, enable device 800 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 818 by processor 806 may result in processor 806 using data 812.

[0212] For example, program instructions 818 may include an operating system 822 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 800 and one or more application programs 820 (e.g., a browser, social application, or game application). Similarly, data 812 may include operating system data 816 and application data 814. Operating system data 816 is primarily accessible to the operating system 822, while application data 814 is primarily accessible to one or more application programs 820. Application data 814 may reside in a file system visible or hidden from the user of device 800.

[0213] Application 820 can communicate with operating system 822 through one or more application programming interfaces (APIs). These APIs help application 820 read and / or write application data 814, transmit or receive information via communication interface 802, receive or display information on user interface 804, etc.

[0214] In some terminology, application 820 may be simply referred to as "app". Furthermore, application 820 can be downloaded to device 800 through one or more online app stores or app markets. However, applications can also be installed on device 800 in other ways, such as through a web browser or a physical interface on device 800 (e.g., a USB port).

[0215] In some embodiments, the service agent self-evolution device can be applied to, for example... Figure 8 The device shown is used to implement the technical solution described in this specification; please refer to the above description for details.

[0216] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0217] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0218] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0219] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0220] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0221] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0222] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0223] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0224] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0225] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A method for self-evolution of service agents, characterized in that, include: Obtain service tasks and corresponding task evaluation information. The service tasks include initial service context and callable tool information. The task evaluation information is used to describe user goals, acceptance conditions, and key node evaluation items. The control service agent executes the service task multiple times in the interactive simulation environment to obtain a trajectory group consisting of multiple service trajectories. The service trajectory includes multiple trajectory nodes and execution results. The trajectory nodes are used to record any one of the trajectory events, such as interactive actions, tool calls, tool returns, and service status changes. Each service trajectory and the task evaluation information are input into a pre-trained reward model to obtain the result reward and process reward of the service trajectory. The result reward is related to the user goal and the acceptance criteria, and the process reward is related to the key node evaluation items. The result reward and process reward of the service trajectory are aggregated to obtain the trajectory reward of the service trajectory; The trajectory rewards of each service trajectory within the trajectory group are compared relative to each other to obtain the relative advantages of each service trajectory. The model parameters of the service agent are updated based on the relative advantages of each service trajectory.

2. The method according to claim 1, characterized in that, The key node evaluation item is used to describe the key node types that the service trajectory should include when executing the service task; the process reward includes the node process reward corresponding to the key node that exists in the service trajectory and matches the key node type; The process of aggregating the result rewards and process rewards of each service trajectory to obtain the trajectory reward for that service trajectory includes: Based on the event attributes represented by each key node, each key node is divided into service path class, tool call class, or interactive action class. Based on the category to which each key node belongs, the node process rewards corresponding to each key node are classified and aggregated to obtain the category process rewards corresponding to each category. The category process rewards corresponding to each category are aggregated to obtain the total process reward. The total process reward and the result reward are then aggregated to obtain the trajectory reward of the service trajectory.

3. The method according to claim 1, characterized in that, The result reward includes a result acceptance reward and a goal consistency reward. The result acceptance reward is used to characterize the degree to which the task execution result of the service trajectory meets the acceptance conditions, and the goal consistency reward is used to characterize the degree to which the actual service direction of the service trajectory is consistent with the user's goal. The aggregation of result rewards and process rewards for each service trajectory yields the trajectory reward for each service trajectory, including: The basic trajectory reward is determined based on the result acceptance reward and process reward of the service trajectory. Using the target consistency reward of the service trajectory as a multiplication gating factor, the basic trajectory reward is gating and adjusted to obtain the trajectory reward of the service trajectory, wherein the degree of retention of the basic trajectory reward in the trajectory reward is positively correlated with the target consistency reward.

4. The method according to claim 1, characterized in that, Also includes: From the trajectory group, the service trajectory whose result reward meets the preset result requirement but whose process reward does not meet the preset process requirement is identified as a reward conflict service trajectory; At least one trajectory event in the reward conflict service trajectory is taken as the original trajectory event, and the original trajectory event is replaced with a candidate trajectory event. Based on the candidate trajectory event, the counterfactual service trajectory is re-deduced in the interactive simulation environment to obtain the counterfactual service trajectory. The counterfactual service trajectory and the task evaluation information are input into the reward model to obtain the counterfactual result reward; The difference between the result reward of the reward conflict service trajectory and the counterfactual result reward is determined as the causal contribution estimate of the original trajectory event relative to the candidate trajectory event. The aggregation weight of the result reward and the process reward is adjusted according to the causal contribution estimate. The larger the absolute value of the causal contribution estimate, the higher the aggregation weight of the result reward and the lower the aggregation weight of the process reward. The trajectory reward of the reward conflict service trajectory is updated according to the adjusted aggregation weight.

5. The method according to claim 1 or 3, characterized in that, The trajectory rewards for determining each service trajectory also include: The compliance of each service trajectory is checked based on the constraint information related to business compliance; the constraint information related to business compliance includes veto-compliance constraints; If at least one trajectory node in the service trajectory records a trajectory event that violates the veto compliance constraint, the trajectory reward for the service trajectory will be determined to be zero.

6. The method according to any one of claims 1 to 4, characterized in that, The reward model is pre-trained in the following manner: The system acquires sample service trajectories, corresponding sample task evaluation information, and service quality tags. The service quality tags include result-dimensional supervision tags, process-dimensional supervision tags, and a service quality score. The result-dimensional supervision tags are determined based on the result evaluation information and target consistency information of the sample service trajectory. The process-dimensional supervision tags are determined based on the service specification compliance evaluation information and service coverage evaluation information of the sample service trajectory. The service quality score is obtained by aggregating the result-dimensional supervision tags and the process-dimensional supervision tags. The sample service trajectory and the sample task evaluation information are input into the reward model to be trained to obtain the prediction result reward and the prediction process reward. The prediction result reward and the prediction process reward are then aggregated to obtain the prediction trajectory reward. The joint training loss is determined based on the differences between the prediction result reward and the result dimension supervision label, the differences between the prediction process reward and the process dimension supervision label, and the differences between the prediction trajectory reward and the service quality score. The model parameters of the reward model to be trained are updated according to the joint training loss to obtain the reward model.

7. The method according to claim 6, characterized in that, The process dimension supervision label includes node process supervision labels that are associated with multiple key nodes in the sample service trajectory. The node process supervision labels are determined based on the correspondence between the service specification compliance evaluation information and the evaluation details in the service coverage evaluation information and the corresponding key nodes in the sample. The prediction process reward includes prediction node process rewards that are associated with each key node in the sample. The step of determining the joint training loss based on the differences between the prediction result reward and the result dimension supervision label, the differences between the prediction process reward and the process dimension supervision label, and the differences between the prediction trajectory reward and the service quality score includes: Based on the event attributes represented by the key nodes of each sample, the key nodes of each sample are divided into service path class, tool call class or interactive action class. For each category, the node-level training loss corresponding to the category is determined based on the difference between the predicted node process reward and the node process supervision label corresponding to the key nodes of each sample belonging to the corresponding category. The joint training loss is determined based on the difference between the prediction result reward and the result dimension supervision label, the difference between the prediction trajectory reward and the service quality score, and the node-level training loss corresponding to each category.

8. The method according to claim 6, characterized in that, Also includes: Obtain real user feedback corresponding to the sample service trajectory; If the user's genuine feedback meets the preset credibility conditions, a feedback evaluation value is determined based on the user's genuine feedback. Determine the deviation between the feedback evaluation value and the service quality score in the service quality label; If the deviation meets the preset calibration conditions, the service quality score is corrected according to the feedback evaluation value to obtain a calibrated service quality score. The calibrated service quality score is then used as supervision information to update the model parameters of the reward model.

9. The method according to claim 1, characterized in that, The process of obtaining the service task and its corresponding evaluation information includes: Online service trajectories generated during the online service process that meet the preset low quality conditions are identified as failed service trajectories. The business scenario and problem type to which each failed service trajectory belongs are determined, and the failed service trajectories are clustered according to the business scenario and problem type to obtain multiple failed trajectory clusters; The cluster of failure trajectories whose number of failure service trajectories meets the preset quantity condition is identified as the high-frequency failure trajectory cluster, and the corresponding failure mode is determined according to the common failure characteristics of the failure service trajectories in each high-frequency failure trajectory cluster. For each failure mode, based on the corresponding failure service trajectory and the actual system status data, at least one of the preset task template and task generation model is used to generate the service task and the corresponding task evaluation information.

10. The method according to claim 1, characterized in that, The interactive simulation environment includes a simulated user module, a simulated tool module, and a business state machine constructed based on real historical trajectories and tool call specification information; The tool invocation specification information is used to describe the tool's invocation conditions, input parameter rules, return information rules, and exception handling rules; The simulated user module is used to generate user interaction information that interacts with the service agent based on the real historical trajectory. The simulated tool module is used to generate tool return information corresponding to the tool call according to the tool call specification information. The business state machine is used to update the service status according to the interaction action, the tool call and the tool return information.

11. The method according to claim 1 or 10, characterized in that, The interactive simulation environment includes a constraint verification module, which is configured with service constraint information related to the service domain to which the service task belongs. The service constraint information includes constraint information related to business operations, business compliance, and task termination. The constraint verification module is used to verify the instruction to be executed based on the constraint information related to business operation and business compliance in the service constraint information before the instruction to be executed generated by the service agent is executed in the interactive simulation environment. If the instruction to be executed violates the corresponding constraint information, the module refuses to execute the instruction to be executed and returns a rejection message. The module also records the instruction to be executed and the rejection message as trajectory events in the trajectory nodes of the corresponding service trajectory. The instruction to be executed includes interactive action instructions or tool call instructions. The constraint verification module is also used to detect the service status based on the constraint information related to task termination in the service constraint information, and terminate the execution of the service task by the service agent when the service status meets the task termination conditions, and determine the execution result of the corresponding service trajectory based on the service status at the time of termination.