Link tracking method, device, medium, equipment and program product of intelligent agent
By acquiring and storing link tracing data throughout the entire lifecycle of an intelligent agent, the problem of the inability to track the entire lifecycle in existing technologies is solved, enabling detailed recording and optimization support for each stage of the intelligent agent.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING VOLCANO ENGINE TECH CO LTD
- Filing Date
- 2025-08-22
- Publication Date
- 2026-06-09
Smart Images
Figure CN120670264B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and more specifically, to a method, apparatus, medium, device, and program product for link tracing of intelligent agents. Background Technology
[0002] For agent link tracing, existing technologies can only record user input and agent output, making it impossible to evaluate the agent's performance and effectiveness at each stage, severely impacting agent optimization and performance improvement. Furthermore, these technologies only support link tracing for agents after deployment, failing to provide observation of the agent's entire lifecycle. Summary of the Invention
[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.
[0004] Firstly, this disclosure provides a link tracing method for intelligent agents, including:
[0005] Acquire link tracing data of the agent throughout its entire lifecycle. The link tracing data includes data generated by the agent from receiving a request to outputting the response result corresponding to the request. The link tracing data includes first link tracing data when the agent is running online, second link tracing data when the agent is being debugged, third link tracing data when the agent's workflow is being debugged, and fourth link tracing data when the agent is being evaluated.
[0006] The link tracing data is stored based on the original identifier corresponding to the link tracing data; wherein, the original identifier includes at least one of the session identifier, agent identifier, user identifier, and tenant identifier corresponding to the link tracing data;
[0007] In response to a query request, the target link tracing data corresponding to the original identifier carried in the query request is determined from the stored link tracing data, and the target link tracing data is displayed.
[0008] Secondly, this disclosure provides a link tracking device for an intelligent agent, comprising:
[0009] The acquisition module is configured to acquire link tracing data of the agent throughout its entire lifecycle. The link tracing data includes data generated by the agent during the process from receiving a request to the agent outputting the response result corresponding to the request. The link tracing data includes first link tracing data when the agent is running online, second link tracing data when the agent is being debugged, third link tracing data when the agent's workflow is being debugged, and fourth link tracing data when the agent is being evaluated.
[0010] The storage module is configured to store the link tracing data based on the original identifier corresponding to the link tracing data; wherein the original identifier includes at least one of the session identifier, agent identifier, user identifier, and tenant identifier corresponding to the link tracing data;
[0011] The display module is configured to, in response to a query request, determine the target link tracing data corresponding to the original identifier carried in the query request from the stored link tracing data, and display the target link tracing data.
[0012] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.
[0013] Fourthly, this disclosure provides an electronic device, comprising:
[0014] A storage device on which computer programs are stored;
[0015] A processing device for executing the computer program in the storage device to implement the steps of the method described in the first aspect.
[0016] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.
[0017] Based on the above technical solution, by acquiring link tracing data of the agent throughout its entire lifecycle, including data generated by the agent from receiving a request to outputting the corresponding response, and then storing the link tracing data based on the original identifier corresponding to the link tracing data, the system can determine and display the target link tracing data corresponding to the original identifier carried in the query request from the stored link tracing data. This not only covers link tracing during agent debugging, workflow debugging, online use of the agent, and agent evaluation, but also supports link tracing throughout the entire lifecycle of the agent. It can comprehensively and meticulously record the link information of the agent at each key stage, thereby providing support for agent optimization.
[0018] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description
[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:
[0020] Figure 1 This is a flowchart illustrating a link tracing method for intelligent agents according to some embodiments.
[0021] Figure 2 This is a schematic diagram of a second call chain according to some embodiments.
[0022] Figure 3 This is a schematic diagram of target link tracing data corresponding to the target debugging record, as shown in some embodiments.
[0023] Figure 4 This is a schematic diagram illustrating the display of second link tracing data according to some embodiments.
[0024] Figure 5 This is a schematic diagram illustrating the display of fourth link tracing data, according to some embodiments.
[0025] Figure 6 This is a schematic diagram of the structure of a link tracing device for an intelligent agent, as shown in some embodiments.
[0026] Figure 7 This is a schematic diagram of the structure of an electronic device according to some embodiments. Detailed Implementation
[0027] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0028] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.
[0029] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.
[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.
[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".
[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
[0033] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0034] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0035] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.
[0036] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0037] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0038] Before providing a detailed description of the link tracing method for intelligent agents provided in the embodiments of this disclosure, the actual terms used in the embodiments of this disclosure will be explained.
[0039] Log: A text record of discrete events (such as error stacks, business operation logs, etc.) used to provide detailed context of specific events.
[0040] Metrics: Metrics are aggregated statistics of system status (such as request volume, request success rate), which exist in the form of periodically sampled numerical values and are used for macro trend analysis.
[0041] A trace represents the complete execution process of a request or transaction from start to finish, recording the complete call chain of the request or transaction. A call chain is a directed acyclic graph (DAG) composed of multiple spans. Each trace has a unique TraceID, used to identify the entire call chain. For example, the process from a client initiating a request to the server completing its processing constitutes a trace. A trace provides a global picture of what happens when a request is made to an application.
[0042] Span: A span is the smallest unit of distributed tracing, representing a single logical operation in a call chain. It can be a method call, a block call, or a database access, etc. As a named and timed continuous execution segment in the call chain, a span records detailed information that occurs during the execution of a unit, including the span name, start and end times, status, and span attributes.
[0043] Span Attributes: Span attributes are key-value pairs that contain metadata. You can use metadata to annotate a span, such as model name, model version, application name, application version, etc.
[0044] A workflow refers to a series of steps or processes by which an intelligent agent performs a task or achieves a goal. Workflows can be predefined or dynamically generated, and they describe how an intelligent agent responds to environmental changes, makes decisions, executes actions, and learns to improve.
[0045] A conversation is a series of consecutive dialogues between a user and an agent, with each dialogue corresponding to a Trace.
[0046] The link tracing method for intelligent agents provided in the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0047] Figure 1 This is a flowchart illustrating a link tracing method for intelligent agents according to some embodiments. For example... Figure 1 As shown, this disclosure provides a link tracing method for an intelligent agent, which can be executed by an intelligent agent link tracing device, which can be implemented in software and / or hardware. Figure 1 As shown, the method may include the following steps.
[0048] In step 110, the link tracing data of the agent throughout its entire lifecycle is obtained. The link tracing data includes the data generated by the agent from receiving a request to outputting the response result corresponding to the request. The link tracing data includes the first link tracing data when the agent is running online, the second link tracing data when the agent is being debugged, the third link tracing data when the agent's workflow is being debugged, and the fourth link tracing data when the agent is being evaluated.
[0049] Here, the entire lifecycle of an agent includes four stages: online operation, agent debugging, workflow debugging, and agent evaluation. Correspondingly, the link tracing data throughout the agent's lifecycle includes the first link tracing data during online operation, the second link tracing data during agent debugging, the third link tracing data during agent workflow debugging, and the fourth link tracing data during agent evaluation.
[0050] It should be noted that the first, second, third, and fourth link tracing data all include data generated by the agent during the process from receiving the request to outputting the corresponding response. In other words, the first, second, third, and fourth link tracing data will all include link tracing data for the entire process from the agent receiving the question to the agent outputting the answer to that question.
[0051] The first link tracing data is the link tracing data of the agent collected when the agent is running online.
[0052] For example, an event listener can be set in the interface used by the agent to interact with the user. When the event listener receives a trigger event, it collects first-link tracing data of the agent during online operation. The event listener is used to detect various interaction events, such as user request events and response return events.
[0053] For example, taking an online education intelligent tutoring platform as an example, when a user asks a question to the intelligent agent of the online education intelligent tutoring platform, the event listener detects the user request event. At this time, it begins to record the first link tracing data corresponding to the entire process from the question entering the intelligent agent to the intelligent agent retrieving the knowledge base, matching the problem-solving approach, generating the answer and returning it to the user. The recorded first link tracing data includes the user's basic information, the question content, the knowledge base documents called by the intelligent agent, the parameter settings of the problem-solving algorithm, the answer generation time, and other data.
[0054] Second-level link tracing data refers to the link tracing data of the agent collected during the debugging process. It should be understood that second-level link tracing data includes all historical records during the debugging of the agent (such as historical execution results, error messages, etc.).
[0055] Third-party tracing data refers to the link tracing data of the workflow collected during the debugging of the agent's workflow. It should be understood that third-party tracing data includes all historical records during workflow debugging (such as historical execution results, error messages, etc.).
[0056] The fourth link tracing data may include link tracing data of the agent for the evaluation task and link tracing data of the machine learning model scoring the inference results corresponding to the evaluation task output by the agent for the evaluation task. For example, for each inference sample included in the evaluation task, the inference sample can be input into the agent, the link tracing data of the agent for the inference sample can be recorded, and the link tracing data of the machine learning model scoring the inference results corresponding to the inference sample can be recorded.
[0057] The fourth link tracing data can include performance metrics for evaluating the agent. These metrics can be determined using first-packet latency, input token consumption, and output token consumption.
[0058] It should be understood that by using the first, second, third, and fourth link tracing data corresponding to the agent, link tracing data for the entire lifecycle of the agent can be obtained, covering link tracing during agent debugging, workflow debugging, online use, and evaluation phases, thus supporting full lifecycle link tracing of the agent. Furthermore, since the link tracing data includes data generated by the agent from receiving a request to outputting the corresponding response, it can comprehensively and meticulously record the link information of the agent at each key stage, thereby providing support for agent optimization.
[0059] In this disclosed embodiment, link tracing data of an agent throughout its entire lifecycle can be obtained through the OpenTelemetry (OTel, an open-source observability framework for the unified collection and transmission of metrics, logs and tracking data through standardized tools and protocols) framework.
[0060] In step 120, the link tracing data is stored based on the original identifier corresponding to the link tracing data; wherein the original identifier includes at least one of the session identifier, agent identifier, user identifier, and tenant identifier corresponding to the link tracing data.
[0061] Here, the agent's link tracing data throughout its entire lifecycle can be received through OpenTelemetry's Collector (an intermediate layer for centralized data management and distribution). Then, a distributed search engine stores the agent's link tracing data throughout its entire lifecycle. It's important to note that the first, second, third, and fourth link tracing data can be stored in the distributed search engine according to different indexes, and the original identifier corresponding to each link tracing data is recorded. The original identifier includes at least one of the following: session identifier, agent identifier, user identifier, and tenant identifier. The session identifier uniquely identifies the session to which the link tracing data belongs; the agent identifier uniquely identifies the agent to which the link tracing data belongs; the user identifier uniquely identifies the user to which the link tracing data belongs; and the tenant identifier uniquely identifies the tenant to which the link tracing data belongs.
[0062] It should be understood that storing link tracing data using the original identifier can mean storing the data with the original identifier as the key and the link tracing data as the value corresponding to that key. Users can query the required link tracing data using the corresponding original identifier.
[0063] In step 130, in response to the query request, the target link tracing data corresponding to the original identifier carried in the query request is determined from the stored link tracing data and displayed.
[0064] Here, a query server can provide query services to the front end to retrieve target link tracing data corresponding to the original identifier carried in the query request from a distributed search engine, and return it to the front end in a fixed format for display.
[0065] It should be understood that since link tracing data is stored using the original identifier corresponding to the link tracing data, the link tracing data corresponding to the original identifier can be retrieved by including the original identifier in the query request. For example, if there are multiple rounds of dialogue in a single session between a user and an agent, the link tracing data for all dialogues included in that session can be retrieved by including the session identifier in the query request.
[0066] For example, a user can query the link tracing data corresponding to the session associated with that session by using a session identifier. As another example, a user can query all link tracing data corresponding to a specific agent by using an agent identifier.
[0067] Therefore, by acquiring link tracing data of the agent throughout its entire lifecycle—including data generated by the agent from receiving a request to outputting the corresponding response—and storing the link tracing data based on its original identifier, the system can determine and display the target link tracing data corresponding to the original identifier carried in the query request from the stored link tracing data. This not only covers link tracing during agent debugging, workflow debugging, online use, and agent evaluation, but also supports full lifecycle link tracing of the agent. It can comprehensively and meticulously record the link information of the agent at each key stage, thus providing support for agent optimization.
[0068] In some feasible implementations, the following can be recorded: the original instructions received by the agent at the instruction layer; the data content generated by the agent at the intent layer, including at least one of the following: an intent recognition algorithm for identifying the original instructions, the intent category corresponding to the intent, and the confidence level corresponding to the intent; the data content generated by the agent at the tool layer, including at least one of the following: the tool name corresponding to the tool called by the agent, the input parameters of the tool, and the return result of the tool; the data content generated by the agent at the model layer, including at least one of the following: the input data of the model layer, the algorithm parameters used by the model layer, and the output result of the model layer; and the data content generated by the agent at the output layer, including the response result corresponding to the original instructions output by the agent and the method by which the agent integrates the response result; and link tracing data can be obtained based on the data content recorded at the instruction layer, intent layer, tool layer, model layer, and output layer.
[0069] Here, a layered recording architecture is used to collect link tracing data of the agent throughout its entire lifecycle. This layered recording architecture includes the agent's instruction layer, intent layer, tool layer, model layer, and output layer.
[0070] The instruction layer of the agent is used to receive raw instructions input by the user. Accordingly, the raw instructions input by the user can be recorded. These raw instructions can be text instructions, voice instructions, image instructions, etc. For example, if a user inputs the voice instruction "Check the weather for location X tomorrow," the corresponding audio file and the converted text content can be recorded.
[0071] For the agent's intent layer, the agent identifies the intent corresponding to the user's original input command. Accordingly, after the agent identifies the intent of the original command at the intent layer, it can use the identification algorithm for the original command's intent, the intent category corresponding to the intent, and the confidence level of the intent. For example, it can be recorded that the identification algorithm used by the agent to identify the intent is a natural language processing algorithm, the identified intent is to query the weather, and the confidence level of this intent is 0.95.
[0072] For the agent's tool layer, the agent invokes external tools to obtain intent-related results. Accordingly, the tool name, input parameters, and return result of the invoked tool are recorded. For example, if the agent invokes a weather query plugin, the plugin identifier (tool name), the parameters input to the weather query plugin (such as location and time), and the weather information returned by the plugin are recorded.
[0073] For the model layer of the agent, the agent processes the user's raw commands through a large language model. Accordingly, the input data of the large language model, the algorithm parameters used by the large language model, and the output results of the large language model can be recorded. For example, when the agent processes the intent of a weather query through the large language model, the user's raw commands, the large language model's thought process, and the large language model's output results can be recorded.
[0074] For the agent's output layer, the agent outputs the response result corresponding to the original command to the user. Correspondingly, the response result corresponding to the original command output by the agent and the method by which the agent integrates the response result can be recorded. For example, the agent's final output response result might be "Tomorrow at location X, sunny, temperature 20-25 degrees Celsius," and the method by which the agent integrates the weather information obtained from the tool layer and the processing result from the model layer could be recorded.
[0075] It should be understood that the data recorded at the instruction layer, intent layer, tool layer, model layer, and output layer can be used as the link tracing data for the agent. Of course, the recorded data can also be further processed to obtain link tracing data.
[0076] It is worth noting that the first link tracing data, the second link tracing data, the third link tracing data, and the fourth link tracing data can all be obtained through the above-mentioned hierarchical recording architecture.
[0077] Therefore, through the above-mentioned layered recording architecture, the data content generated by the agent at the instruction layer, intent layer, tool layer, model layer and output layer can be recorded, thereby comprehensively and in detail recording the link information of the agent at each key link, thus providing support for the optimization of the agent.
[0078] In some feasible implementations, the link tracing data includes at least a first call chain corresponding to the workflow invoked by the agent. Accordingly, when the workflow nodes included in the workflow invoked by the agent are executed, a span corresponding to the workflow node is created based on the node information corresponding to the workflow node, and then a first call chain corresponding to the workflow is constructed based on the topological relationship between the workflow nodes and the created span.
[0079] Here, when debugging the agent's workflow, the third-link tracing data obtained can be the first call chain corresponding to the workflow invoked by the agent. Of course, when the agent is running online, if the agent invokes a workflow, the first-link tracing data will also contain the first call chain corresponding to the workflow invoked by the agent.
[0080] The workflow invoked by the agent can include multiple workflow nodes. When a workflow node is executed, a span corresponding to that workflow node is created based on the node information. This node information can include the node identifier, node type, and the workflow identifier to which it belongs, among other things.
[0081] For example, node types may include start nodes, end nodes, large language model nodes, knowledge base nodes, question-answering database nodes, tool nodes, terminology database nodes, and so on. It should be understood that node types are predefined types to which workflow nodes belong.
[0082] Each executed workflow node corresponds to a span in the first call chain. The attributes of the corresponding span can be defined through the node information of the workflow node, thereby creating the span corresponding to the workflow node.
[0083] After the workflow is completed, the first call chain corresponding to the workflow can be constructed based on the topological relationship between workflow nodes and the created span.
[0084] The topological relationship between workflow nodes can refer to the parent-child relationship between each workflow node (represented by the `child_of` parameter). For all created spans, the parent-child relationship between the created spans can be determined through the topological relationship between workflow nodes, thus forming the first call chain corresponding to the workflow.
[0085] Therefore, through the above implementation method, a first call chain corresponding to the workflow can be constructed, thereby enabling workflow tracing. For example, through the first call chain, users can query whether the execution of the workflow meets expectations.
[0086] In some feasible implementations, log information corresponding to the session between the agent and the user can also be obtained from the first link tracing data. Then, based on the session identifier corresponding to the log information, the log information can be associated with the first link tracing data corresponding to the session, so as to view the log information corresponding to the session through the first link tracing data corresponding to the session.
[0087] Here, the agent can actively upload log information to the Collector, and then store the log information in the distributed search engine, specifically as the observe-real-time-log-YYYY-MM index.
[0088] Next, the log information is associated with the first link tracing data corresponding to the session through the session identifier, thereby linking the log information and the first link tracing data.
[0089] It should be understood that a single session can include multiple dialogues, and the first link tracing data will include the link tracing data corresponding to each dialogue. Associating the first link tracing data corresponding to a session with the session identifier corresponding to the log information can be achieved by associating the link tracing data corresponding to each dialogue with the log information corresponding to that dialogue.
[0090] Therefore, through the above implementation method, the log information of the session can be associated with the first link tracing data of the session, thereby facilitating the tracing of the complete execution path of the distributed request.
[0091] In some feasible implementations, the link tracing data includes a second call chain corresponding to the agent, which is the call chain corresponding to a single dialogue between the agent and the user. The second call chain includes the span executed by the agent and the topological relationships between the spans. Accordingly, the second call chain can be displayed.
[0092] Here, the second call chain corresponding to the agent refers to the execution path of each service when the agent executes a request. Figure 2 This is a schematic diagram of a second call chain according to some embodiments. For example... Figure 2As shown, a single session between an agent and a user comprises multiple consecutive dialogues. Each dialogue corresponds to a link tracing data point, and each link tracing data point includes a second call chain. Each second call chain includes the spans executed by one or more agents and the topological relationships between these spans. In the second call chain, the first span is the root node, each root node to the last node represents a call chain, and the topological relationship between spans is a parent-child relationship. For example, in... Figure 2 In the diagram, the first span 201 serves as the parent node of the second span 202, and the second span 202 serves as the child node of the first span 201.
[0093] It is worth noting that the first link tracing data, the second link tracing data, the third link tracing data, and the fourth link tracing data can all include the second call chain.
[0094] Therefore, by displaying the second call chain, the execution path of the agent can be clearly shown.
[0095] In some feasible implementations, step 130 may involve displaying a workflow debugging interface, which includes the workflow to be debugged. Then, in response to a query request triggered on the workflow debugging interface to view the target debugging record corresponding to the workflow, the target link tracing data corresponding to the target debugging record is determined from the stored link tracing data, and the target link tracing data corresponding to the target debugging record is displayed on the workflow debugging interface.
[0096] Here, the workflow debugging interface is used to adjust the workflow. The workflow debugging interface displays a history control; triggering this control displays the corresponding historical debugging records for the workflow. These historical debugging records can be sorted in descending order of time.
[0097] A query request to view the target debug record corresponding to a workflow can be triggered by selecting a historical debug record. For example, if a user selects a historical debug record, that selected record becomes the target debug record. Alternatively, a query request to view the target debug record corresponding to a workflow can be triggered by clicking the quick query control in the workflow debug interface. When a user clicks the quick query control in the workflow debug interface, the most recent historical debug record is used as the target debug record.
[0098] Then, the target link tracing data corresponding to the target debug record is retrieved from the stored link tracing data, and displayed in the workflow debugging interface. It's important to note that the target link tracing data corresponding to the target debug record can include the first call chain of the workflow and the runtime information corresponding to the target debug record.
[0099] Figure 3 This is a schematic diagram illustrating the target link tracing data corresponding to the target debugging record, based on some embodiments. For example... Figure 3 As shown, the first interface 300 can display the first call chain and runtime information 301 corresponding to the target debug record. The runtime information 301 includes information such as the execution time, consumed tokens, first character response time, start time, end time, and input for the target debug record. It should be noted that the first interface 300 can be displayed within a workflow debugging interface.
[0100] In some embodiments, the execution status and execution time of each span included in the first call chain can also be displayed in the first call chain.
[0101] like Figure 3 As shown, in the adjacent area of each span in the first call chain, the span execution status 303 and span execution time 302 corresponding to each span can be displayed. The span execution status 303 is used to indicate whether the span execution was successful. If the span execution is successful, it can be marked with a "√"; if the span execution fails, it can be marked with an "×". Of course, in other embodiments, the span execution status 303 can also be displayed using other identifiers, such as a "success" identifier or a "failure" identifier.
[0102] In some embodiments, when each span included in the first call chain is selected, the span details information corresponding to the selected span can also be displayed.
[0103] Each span in the first call chain can be selected. When a span is selected, its detailed information can be displayed. This information may include the span type, execution status, execution time, first character response time, token consumed, first character response time, name of the span, call type, start time, and end time.
[0104] like Figure 3 As shown, when the user selects the "User Input" span in the first call chain, the span details 304 corresponding to the "User Input" span can be displayed on the first interface 300.
[0105] In some embodiments, when the spans included in the first call chain are selected, the workflow node corresponding to the span can be positioned in the middle of the workflow debugging interface.
[0106] Following the above implementation method, the first interface 300 can be the interface displayed in the workflow debugging interface. Therefore, when a certain span is selected, the workflow node corresponding to the span can be positioned in the middle of the workflow debugging interface, so that the user can quickly locate the workflow node.
[0107] In some embodiments, when there are erroneous spans in each span of the first call chain, error information corresponding to the erroneous span can also be displayed.
[0108] An error span is a span that encounters an anomaly, which can be understood as a span whose execution status is failed. An error reporting control 305 can be provided in the first interface 300 to display the error information corresponding to the error span.
[0109] Therefore, through the above implementation method, rich link tracing information can be provided to users during workflow debugging to help them debug the workflow.
[0110] Figure 4 This is a schematic diagram illustrating the display of second link tracing data, according to some embodiments. For example... Figure 4 As shown, when a user views the second link tracing data corresponding to a specific debugging session of the agent, the call chain and operational information of the agent can be displayed on the second interface 400. Similarly, when a user selects a span on the second interface 400, the span details corresponding to the selected span will also be displayed, consistent with the display method of the first interface 300.
[0111] It is worth noting that the second interface 400 can be displayed in the agent debugging interface used to debug the agent.
[0112] Figure 5 This is a schematic diagram illustrating the display of fourth link tracing data, according to some embodiments. For example... Figure 5 As shown, the third interface 500 displays the evaluation results of agents (such as object A, object B, and object C). When the user views the fourth link tracing data corresponding to object B in the third interface 500, the fourth interface 501 can be displayed in the adjacent area of the third interface 500. The fourth interface 501 displays the call chain corresponding to the agent and the agent's running information. Of course, when the user selects a span in the call chain of the fourth interface 501, the span details corresponding to the selected span will also be displayed, consistent with the display method of the first interface 300. It is worth noting that the third interface 500 and the fourth interface 501 can be displayed in an agent evaluation interface used to evaluate agents.
[0113] Figure 6 This is a schematic diagram of the structure of a link tracing device for an intelligent agent, illustrated according to some embodiments. For example... Figure 6 As shown, this disclosure provides a link tracing device 600 for an intelligent agent, which includes:
[0114] The acquisition module 601 is configured to acquire link tracing data of the agent throughout its entire lifecycle. The link tracing data includes data generated by the agent during the process from receiving a request to the agent outputting a response result corresponding to the request. The link tracing data includes first link tracing data when the agent is running online, second link tracing data when the agent is being debugged, third link tracing data when the agent's workflow is being debugged, and fourth link tracing data when the agent is being evaluated.
[0115] Storage module 602 is configured to store the link tracing data based on the original identifier corresponding to the link tracing data; wherein the original identifier includes at least one of the session identifier, agent identifier, user identifier, and tenant identifier corresponding to the link tracing data;
[0116] Display module 603 is configured to, in response to a query request, determine the target link tracing data corresponding to the original identifier carried in the query request from the stored link tracing data, and display the target link tracing data.
[0117] Optionally, the acquisition module 601 is specifically configured as follows:
[0118] Record the original instructions received by the intelligent agent at the instruction layer;
[0119] Record the data content generated by the agent at the intent layer, the data content including at least one of the following: an intent recognition algorithm for recognizing the original instruction, an intent category corresponding to the intent, and a confidence level corresponding to the intent;
[0120] Record the data content generated by the agent at the tool layer. The data content includes at least one of the tool name corresponding to the tool called by the agent, the input parameters of the tool, and the return result of the tool.
[0121] Record the data content generated by the agent at the model layer, wherein the data content includes at least one of the input data of the model layer, the algorithm parameters used by the model layer, and the output results of the model layer;
[0122] Record the data content generated by the agent at the output layer. The data content includes the response result corresponding to the original instruction output by the agent and the way the agent integrates the response result.
[0123] The link tracing data is obtained based on the data recorded in the instruction layer, the intent layer, the tool layer, the model layer, and the output layer.
[0124] Optionally, the link tracing data includes at least the first call chain corresponding to the workflow invoked by the agent, and the acquisition module 601 is specifically configured to:
[0125] When the workflow nodes included in the workflow invoked by the agent are executed, a span corresponding to the workflow node is created based on the node information corresponding to the workflow node.
[0126] Based on the topological relationships between the workflow nodes and the created span, a first call chain corresponding to the workflow is constructed.
[0127] Optionally, if the link tracing data is the first link tracing data, the link tracing device 600 of the agent further includes:
[0128] The log acquisition unit is configured to acquire log information corresponding to the conversation between the agent and the user.
[0129] The association unit is configured to associate the log information with the first link tracing data corresponding to the session based on the session identifier corresponding to the log information, so as to view the log information corresponding to the session through the first link tracing data corresponding to the session.
[0130] Optionally, the link tracing data includes a second call chain corresponding to the agent, the second call chain being the call chain corresponding to a dialogue between the agent and the user, and the second call chain including the span executed by the agent and the topological relationship between the spans; the display module 603 is specifically configured to:
[0131] The second call chain is displayed.
[0132] Optionally, the display module 603 is specifically configured as follows:
[0133] Displays a workflow debugging interface, which includes the workflow to be debugged;
[0134] In response to a query request triggered on the workflow debugging interface for viewing the target debugging record corresponding to the workflow, the target link tracing data corresponding to the target debugging record is determined from the stored link tracing data, and the target link tracing data corresponding to the target debugging record is displayed on the workflow debugging interface. The target link tracing data includes the first call chain corresponding to the workflow and the running information corresponding to the target debugging record.
[0135] Optionally, the display module 603 is further configured to:
[0136] The first call chain displays the execution status and execution time of each span corresponding to the first span; and / or
[0137] When each span included in the first call chain is selected, display the span details corresponding to the selected span; and / or
[0138] When each span included in the first call chain is selected, the workflow node corresponding to the span is positioned in the middle of the workflow debugging interface; and / or
[0139] When there is an erroneous span in any of the spans included in the first call chain, the error information corresponding to the erroneous span is displayed.
[0140] The functional logic executed by each functional module in the link tracing device 600 of the aforementioned intelligent agent has been described in detail in the section on methods, and will not be repeated here.
[0141] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 700 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0142] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0143] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
[0144] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.
[0145] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
[0146] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.
[0147] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.
[0148] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:
[0149] Acquire link tracing data of the agent throughout its entire lifecycle. The link tracing data includes data generated by the agent from receiving a request to outputting a response corresponding to the request. This includes first link tracing data during online operation, second link tracing data during agent debugging, third link tracing data during workflow debugging, and fourth link tracing data during agent evaluation. Store the link tracing data based on its original identifier. The original identifier includes at least one of a session identifier, agent identifier, user identifier, and tenant identifier. In response to a query request, determine the target link tracing data corresponding to the original identifier carried in the query request from the stored link tracing data and display the target link tracing data.
[0150] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0152] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.
[0153] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.
[0154] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0155] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.
[0156] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.
[0157] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.
Claims
1. A link tracing method for an intelligent agent, characterized in that, include: Acquire link tracing data of the agent throughout its entire lifecycle. The link tracing data includes data generated by the agent from receiving a request to outputting the response result corresponding to the request. The link tracing data includes first link tracing data when the agent is running online, second link tracing data when the agent is being debugged, third link tracing data when the agent's workflow is being debugged, and fourth link tracing data when the agent is being evaluated. The link tracing data is stored based on the original identifier corresponding to the link tracing data; wherein, the original identifier includes at least one of the session identifier, agent identifier, user identifier, and tenant identifier corresponding to the link tracing data; In response to a query request, the target link tracing data corresponding to the original identifier carried in the query request is determined from the stored link tracing data, and the target link tracing data is displayed. Link tracing data of an agent throughout its entire lifecycle is collected using a hierarchical recording architecture. This link tracing data is obtained through the following steps: Record the original instructions received by the intelligent agent at the instruction layer; Record the data content generated by the agent at the intent layer, the data content including at least one of the following: an intent recognition algorithm for recognizing the original instruction, an intent category corresponding to the intent, and a confidence level corresponding to the intent; Record the data content generated by the agent at the tool layer. The data content includes at least one of the tool name corresponding to the tool called by the agent, the input parameters of the tool, and the return result of the tool. Record the data content generated by the agent at the model layer, wherein the data content includes at least one of the input data of the model layer, the algorithm parameters used by the model layer, and the output results of the model layer; Record the data content generated by the agent at the output layer. The data content includes the response result corresponding to the original instruction output by the agent and the way the agent integrates the response result. The link tracing data is obtained based on the data content recorded in the instruction layer, the intent layer, the tool layer, the model layer, and the output layer. The link tracing data includes at least a first call chain corresponding to the workflow invoked by the agent, and the first call chain is obtained through the following steps: When the workflow nodes included in the workflow invoked by the agent are executed, a span corresponding to the workflow node is created based on the node information corresponding to the workflow node; wherein, each executed workflow node corresponds to a span in a first call chain, and the attributes of the corresponding span are defined through the node information of the workflow node to create a span corresponding to the workflow node; Based on the topological relationship between the workflow nodes and the created span, construct the first call chain corresponding to the workflow; The link tracing data includes a second call chain corresponding to the agent. The second call chain is the call chain corresponding to a dialogue between the agent and the user. The second call chain includes the span executed by the agent and the topological relationship between the spans. The display of the target link tracing data includes: The second call chain is displayed.
2. The method according to claim 1, characterized in that, When the link tracing data is the first link tracing data, the method further includes: Obtain log information corresponding to the conversation between the intelligent agent and the user; Based on the session identifier corresponding to the log information, the log information is associated with the first link tracing data corresponding to the session, so as to view the log information corresponding to the session through the first link tracing data corresponding to the session.
3. The method according to claim 1 or 2, characterized in that, The step of responding to a query request by determining the target link tracing data corresponding to the original identifier carried in the query request from the stored link tracing data and displaying the target link tracing data includes: Displays a workflow debugging interface, which includes the workflow to be debugged; In response to a query request triggered on the workflow debugging interface for viewing the target debugging record corresponding to the workflow, the target link tracing data corresponding to the target debugging record is determined from the stored link tracing data, and the target link tracing data corresponding to the target debugging record is displayed on the workflow debugging interface. The target link tracing data includes the first call chain corresponding to the workflow and the running information corresponding to the target debugging record.
4. The method according to claim 1, characterized in that, The method further includes: The first call chain displays the execution status and execution time of each span corresponding to the first span; and / or When each span included in the first call chain is selected, display the span details corresponding to the selected span; and / or When each span included in the first call chain is selected, the workflow node corresponding to the span is positioned in the middle of the workflow debugging interface; and / or When there is an erroneous span in any of the spans included in the first call chain, the error information corresponding to the erroneous span is displayed.
5. A link tracing device for an intelligent agent, characterized in that, include: The acquisition module is configured to acquire link tracing data of the agent throughout its entire lifecycle. The link tracing data includes data generated by the agent during the process from receiving a request to the agent outputting the response result corresponding to the request. The link tracing data includes first link tracing data when the agent is running online, second link tracing data when the agent is being debugged, third link tracing data when the agent's workflow is being debugged, and fourth link tracing data when the agent is being evaluated. The storage module is configured to store the link tracing data based on the original identifier corresponding to the link tracing data; wherein the original identifier includes at least one of the session identifier, agent identifier, user identifier, and tenant identifier corresponding to the link tracing data; The display module is configured to, in response to a query request, determine the target link tracing data corresponding to the original identifier carried in the query request from the stored link tracing data, and display the target link tracing data; The acquisition module collects link tracing data of the intelligent agent throughout its entire lifecycle through a hierarchical recording architecture, specifically configured as follows: Record the original instructions received by the intelligent agent at the instruction layer; Record the data content generated by the agent at the intent layer, the data content including at least one of the following: an intent recognition algorithm for recognizing the original instruction, an intent category corresponding to the intent, and a confidence level corresponding to the intent; Record the data content generated by the agent at the tool layer. The data content includes at least one of the tool name corresponding to the tool called by the agent, the input parameters of the tool, and the return result of the tool. Record the data content generated by the agent at the model layer, wherein the data content includes at least one of the input data of the model layer, the algorithm parameters used by the model layer, and the output results of the model layer; Record the data content generated by the agent at the output layer. The data content includes the response result corresponding to the original instruction output by the agent and the way the agent integrates the response result. The link tracing data is obtained based on the data content recorded in the instruction layer, the intent layer, the tool layer, the model layer, and the output layer. The link tracing data includes at least the first call chain corresponding to the workflow invoked by the agent, and the acquisition module is specifically configured as follows: When the workflow nodes included in the workflow invoked by the agent are executed, a span corresponding to the workflow node is created based on the node information corresponding to the workflow node; wherein, each executed workflow node corresponds to a span in a first call chain, and the attributes of the corresponding span are defined through the node information of the workflow node to create a span corresponding to the workflow node; Based on the topological relationship between the workflow nodes and the created span, construct the first call chain corresponding to the workflow; The link tracing data includes a second call chain corresponding to the agent, which is the call chain corresponding to a dialogue between the agent and the user. The second call chain includes the span executed by the agent and the topological relationship between the spans; the display module is specifically configured as follows: The second call chain is displayed.
6. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processing device, it implements the steps of the method according to any one of claims 1-4.
7. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-4.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-4.