Intelligent agent link tracking method and device, medium, equipment and program product

Abstract

A link tracking method and apparatus for an agent, a medium, a device and a program product, relating to the technical field of computers, the method comprising: acquiring link tracking data of the agent in a full life cycle, the link tracking data comprising data content generated in a process from the agent receiving a request to the agent outputting a response result corresponding to the request; and then storing the link tracking data based on the original identifier corresponding to the link tracking data, in response to the query request, determining target link tracking data corresponding to the original identifier carried by the query request from the stored link tracking data, and displaying the target link tracking data, according to the method, intelligent agent debugging, workflow debugging, intelligent agent online use and link tracking in an intelligent agent evaluation stage can be covered, so that full-life-cycle link tracking of the intelligent agent is supported, link information of the intelligent agent in each key link can be comprehensively recorded in detail, and support is provided for optimization of the intelligent agent.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a link tracking method, apparatus, medium, device, and program product for an intelligent agent. Background Art

[0002] When it comes to tracking the links of intelligent agents, related technologies can only simply record user input and agent output, making it impossible to evaluate the performance and effectiveness of the intelligent agent at each stage, which seriously hinders the optimization and performance improvement of the intelligent agent. Moreover, related technologies only support link tracking of intelligent agents after they are released, and cannot achieve observation of the entire life cycle of the intelligent agent. Summary of the Invention

[0003] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a link tracking method for an intelligent agent, comprising: Acquire link tracking data for the agent throughout its life cycle, the link tracking data including data content generated by the agent from the time the agent receives a request to the time the agent outputs a response result corresponding to the request. The link tracking data includes first link tracking data when the agent is running online, second link tracking data when the agent is being debugged, third link tracking data when the agent's workflow is being debugged, and fourth link tracking data when the agent is being evaluated; storing the link tracking data based on an original identifier corresponding to the link tracking data; wherein the original identifier includes at least one of a session identifier, an agent identifier, a user identifier, and a tenant identifier corresponding to the link tracking data; In response to the query request, target link tracking data corresponding to the original identifier carried in the query request is determined from the stored link tracking data, and the target link tracking data is displayed.

[0005] In a second aspect, the present disclosure provides a link tracking device for an intelligent agent, comprising: an acquisition module configured to acquire link tracking data of an agent throughout its entire life cycle, the link tracking data including data content generated by the agent from the time the agent receives a request to the time the agent outputs a response result corresponding to the request, the link tracking data including first link tracking data when the agent is running online, second link tracking data when the agent is being debugged, third link tracking data when the agent's workflow is being debugged, and fourth link tracking data when the agent is being evaluated; A storage module is configured to store the link tracking data based on an original identifier corresponding to the link tracking data; wherein the original identifier includes at least one of a session identifier, an agent identifier, a user identifier, and a tenant identifier corresponding to the link tracking data; The display module is configured to determine, in response to a query request, target link tracking data corresponding to the original identifier carried in the query request from the stored link tracking data, and display the target link tracking data.

[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0007] In a fourth aspect, the present disclosure provides an electronic device, comprising: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method described in the first aspect.

[0008] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0009] Based on the above technical solution, by obtaining the link tracking data of the intelligent agent throughout its life cycle, the link tracking data includes the data content generated by the intelligent agent in the process from receiving the request to the intelligent agent outputting the response result corresponding to the request, and then based on the original identifier corresponding to the link tracking data, the link tracking data is stored, and in response to the query request, the target link tracking data corresponding to the original identifier carried by the query request is determined from the stored link tracking data, and the target link tracking data is displayed. It can not only cover the link tracking of the intelligent agent debugging, workflow debugging, intelligent agent online use and intelligent agent evaluation stages, thereby supporting link tracking of the intelligent agent throughout its life cycle, but also can comprehensively and detailedly record the link information of the intelligent agent in each key link, thereby providing support for the optimization of the intelligent agent.

[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 is a flowchart of a link tracking method of an intelligent agent according to some embodiments.

[0012] Figure 2 is a schematic diagram of a second call chain according to some embodiments.

[0013] Figure 3 This is a schematic diagram illustrating target link tracing data corresponding to a target debugging record according to some embodiments.

[0014] Figure 4 is a schematic diagram showing second link tracking data according to some embodiments.

[0015] Figure 5 is a schematic diagram showing fourth link tracking data according to some embodiments.

[0016] Figure 6 It is a structural diagram of a link tracking device of an intelligent body according to some embodiments.

[0017] Figure 7 is a schematic structural diagram of an electronic device according to some embodiments. DETAILED DESCRIPTION

[0018] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0019] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0020] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] At the same time, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0029] Before explaining in detail the link tracking method of the intelligent agent provided by the embodiment of the present disclosure, the actual terms of the embodiment of the present disclosure are explained.

[0030] Log: A text record of discrete events (such as error stacks, business operation logs, etc.) that provides detailed context for specific events.

[0031] Metrics: Metrics are aggregated statistics of system status (such as request volume and request success rate), which exist in the form of periodically sampled values ​​and are used for macro trend analysis.

[0032] A trace represents the complete execution process of a request or transaction from start to finish, recording the complete call chain of the request or transaction. The call chain is a directed acyclic graph (DAG) consisting of multiple spans. Each trace has a unique traceID, which identifies the entire call chain. For example, a trace is the process from a client initiating a request to the server processing it. Traces provide a global picture of what happens when a request is made to an application.

[0033] Span: A span is the smallest unit of distributed tracing, representing a single logical operation in a call chain. It can be a method call, a program block call, or a database access. A span is a named and timed continuous execution segment in a call chain. It records detailed information about the execution of a unit, including the span name, start and end times, status, and span attributes.

[0034] Span Attributes: Span attributes are key-value pairs containing metadata. Span can be annotated with metadata, such as model name, model version, application name, application version, etc.

[0035] A workflow is a series of steps or processes that an agent follows to perform a task or achieve a goal. Workflows can be predefined or dynamically generated, describing how an agent responds to environmental changes, makes decisions, executes actions, and learns and improves.

[0036] Session: A session represents multiple consecutive conversations between a user and an agent, and each conversation corresponds to a trace.

[0037] The link tracking method of the intelligent agent provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0038] Figure 1 FIG. 1 is a flow chart of a link tracking method of an intelligent agent according to some embodiments. Figure 1 As shown, the embodiment of the present disclosure provides a link tracking method for an intelligent agent, which can be specifically performed by a link tracking device for an intelligent agent, and the device can be implemented in software and / or hardware. Figure 1 As shown, the method may include the following steps.

[0039] In step 110, the link tracking data of the intelligent agent throughout its entire life cycle is obtained. The link tracking data includes the data content generated by the intelligent agent in the process from receiving a request to the intelligent agent outputting a response result corresponding to the request. The link tracking data includes the first link tracking data when the intelligent agent is running online, the second link tracking data when the intelligent agent is debugged, the third link tracking data when the intelligent agent's workflow is debugged, and the fourth link tracking data when the intelligent agent is evaluated.

[0040] Here, the full lifecycle of an agent includes four stages: online operation, agent debugging, workflow debugging, and agent evaluation. Accordingly, the link tracking data for an agent throughout its full lifecycle includes the first link tracking data for online operation, the second link tracking data for debugging, the third link tracking data for workflow debugging, and the fourth link tracking data for evaluation.

[0041] It should be noted that the first, second, third, and fourth link tracking data all include data generated by the agent from the time it receives a request to the time it outputs a response to the request. In other words, the first, second, third, and fourth link tracking data all include link tracking data covering the entire process from the time the agent receives a question to the time the agent outputs the answer to the question.

[0042] Among them, the first link tracking data is the link tracking data of the intelligent agent collected when the intelligent agent is running online.

[0043] For example, an event listener can be set up in the interface used by the agent to interact with the user. When the event listener receives a trigger event, it collects the first link tracking data of the agent when it is running online. The event listener is used to detect various interaction events, such as user request events and response return events.

[0044] For example, in the case of an online education intelligent tutoring platform, when a user asks a question to the platform's agent, the event listener detects the user request event. At this point, it begins recording first-link tracking data for the entire process, from the question entering the agent, to the agent searching the knowledge base, matching solution ideas, generating an answer, and returning it to the user. This first-link tracking data includes basic user information, question content, knowledge base documents invoked by the agent, parameter settings for the problem-solving algorithm, and answer generation time.

[0045] The second link tracking data is the link tracking data of the agent collected when debugging the agent. It should be understood that the second link tracking data includes all historical records (such as historical execution results, error messages, etc.) when debugging the agent.

[0046] The third link tracking data is the workflow link tracking data collected when debugging the workflow of the agent. It should be understood that the third link tracking data includes all historical records (such as historical execution results, error messages, etc.) when debugging the workflow.

[0047] The fourth link tracking data may include link tracking data of the agent for the evaluation task and link tracking data of the machine learning model scoring the inference result corresponding to the evaluation task output by the agent. For example, for each inference sample included in the evaluation task, the inference sample may be input into the agent, and the link tracking data of the agent for the inference sample and the link tracking data of the machine learning model scoring the inference result corresponding to the inference sample may be recorded.

[0048] The fourth link tracking data may include performance indicators for evaluating the agent, which may be determined by first packet delay, input token consumption, and output token consumption.

[0049] It should be understood that through the first link tracking data, second link tracking data, third link tracking data, and fourth link tracking data corresponding to the agent, link tracking data for the entire life cycle of the agent can be obtained, covering link tracking during the agent debugging, workflow debugging, agent online use, and agent evaluation stages, thereby supporting link tracking of the agent throughout its life cycle. Moreover, because link tracking data includes data content generated by the agent from the time it receives a request to the time it outputs a response corresponding to the request, it can comprehensively and detailedly record the link information of the agent at each key link, thereby providing support for the optimization of the agent.

[0050] In the disclosed embodiment, the link tracking data of the intelligent agent throughout its life cycle can be obtained through the OpenTelemetry (OTel, an open source observability framework for unified collection and transmission of indicators, logs and tracking data through standardized tools and protocols) framework.

[0051] In step 120, the link tracking data is stored based on the original identifier corresponding to the link tracking data; wherein the original identifier includes at least one of a session identifier, an agent identifier, a user identifier, and a tenant identifier corresponding to the link tracking data.

[0052] Here, the link tracking data of the agent throughout its life cycle can be received through OpenTelemetry's Collector (an intermediate layer for centralized management and distribution of data). Then, the link tracking data of the agent throughout its life cycle is stored through a distributed search engine. It should be noted that the first link tracking data, the second link tracking data, the third link tracking data, and the fourth link tracking data can be stored in a distributed search engine according to different indexes, and the original identifier corresponding to each link tracking data is recorded. Among them, the original identifier includes at least one of the session identifier, agent identifier, user identifier, and tenant identifier corresponding to the link tracking data. The session identifier is used to uniquely identify the session to which the link tracking data belongs, the agent identifier is used to uniquely identify the agent to which the link tracking data belongs, the user identifier is used to uniquely identify the user to which the link tracking data belongs, and the tenant identifier is used to uniquely identify the tenant to which the link tracking data belongs.

[0053] It should be understood that storing link tracking data using the original identifier may mean storing the link tracking data using the original identifier as a key and the link tracking data as a value corresponding to the key. A user may query the required link tracking data using the corresponding original identifier.

[0054] In step 130 , in response to the query request, target link tracking data corresponding to the original identifier carried in the query request is determined from the stored link tracking data, and the target link tracking data is displayed.

[0055] Here, the query server can provide query services to the front end, which is used to query the target link tracking data corresponding to the original identifier carried by the query request from the distributed search engine, and return it to the front end in a fixed format to display the target link tracking data on the front end.

[0056] It should be understood that since link tracking data is stored using the original identifier corresponding to the link tracking data, the original identifier can be included in a query request to query the link tracking data corresponding to the original identifier. For example, if a conversation between a user and an agent includes multiple rounds of dialogue, the link tracking data for all rounds of dialogue included in the conversation can be retrieved by including the session identifier in the query request.

[0057] For example, a user can query the link tracking data corresponding to the session corresponding to the session identifier by using the session identifier. For another example, a user can query all link tracking data corresponding to a certain agent by using the agent identifier.

[0058] Therefore, by obtaining the link tracking data of the intelligent agent throughout its life cycle, the link tracking data includes the data content generated by the intelligent agent in the process from receiving the request to the intelligent agent outputting the response result corresponding to the request, and then based on the original identifier corresponding to the link tracking data, the link tracking data is stored, and in response to the query request, the target link tracking data corresponding to the original identifier carried by the query request is determined from the stored link tracking data, and the target link tracking data is displayed. It can not only cover the link tracking of the intelligent agent debugging, workflow debugging, intelligent agent online use and intelligent agent evaluation stages, thereby supporting link tracking of the intelligent agent throughout its life cycle, but also can comprehensively and detailedly record the link information of the intelligent agent in each key link, thereby providing support for the optimization of the intelligent agent.

[0059] In some feasible implementations, the original instructions received by the agent at the instruction layer can be recorded; the data content generated by the agent at the intention layer can be recorded, and the data content includes the recognition algorithm for identifying the intention of the original instruction, the intention category corresponding to the intention, and at least one of the confidence level corresponding to the intention; the data content generated by the agent at the tool layer can be recorded, and the data content includes the tool name corresponding to the tool called by the agent, the tool input parameters, and at least one of the tool return results; the data content generated by the agent at the model layer can be recorded, and the data content includes the input data of the model layer, the algorithm parameters used by the model layer, and at least one of the output results of the model layer; the data content generated by the agent at the output layer can be recorded, and the data content includes the response results corresponding to the original instructions output by the agent and the way in which the agent integrates the response results; based on the data content recorded at the instruction layer, intention layer, tool layer, model layer, and output layer, link tracking data can be obtained.

[0060] Here, a layered recording architecture is used to collect link tracking data throughout the life cycle of the agent. The layered recording architecture includes the agent's instruction layer, intent layer, tool layer, model layer, and output layer.

[0061] The agent's command layer is used to receive original commands input by the user into the agent. Accordingly, the original commands input by the user can be recorded. These original commands can be text commands, voice commands, image commands, and so on. For example, if a user inputs the voice command "Help me check tomorrow's weather in X location," the corresponding audio file and the converted text content of the voice command can be recorded.

[0062] The agent's intent layer identifies the intent corresponding to the original user input command. Accordingly, after the agent identifies the original command's intent at the intent layer, it can use the recognition algorithm used to identify the original command's intent, the corresponding intent category, and the corresponding confidence level. For example, the agent's recognition algorithm used to identify the intent is a natural language processing algorithm, the recognized intent is to query the weather, and the confidence level of the intent is 0.95.

[0063] For the agent's tool layer, the agent calls external tools through the tool layer to obtain results related to the intent. Accordingly, the tool name, input parameters, and return results corresponding to the tool called by the agent are recorded. For example, if the agent calls a weather query plugin, the plugin identifier (tool name) corresponding to the weather query plugin, the parameters input to the weather query plugin (such as location and time), and the weather information returned by the weather query plugin are recorded.

[0064] At the agent's model layer, the agent processes the user's original instructions through a large language model. Accordingly, the large language model's input data, algorithm parameters used by the large language model, and the large language model's output can be recorded. For example, when the agent processes a weather query intent through a large language model, it can record the user's original input instruction, the large language model's thought process, and the large language model's output.

[0065] The agent's output layer outputs the response to the original command to the user. This layer records the response to the original command output by the agent and how the agent integrates the response. For example, the agent's final output might be "Tomorrow, the weather at X will be sunny, with a temperature of 20-25 degrees Celsius." This layer also records how the agent integrates the weather information obtained by the tool layer with the processing results of the model layer.

[0066] It should be understood that the data content recorded at the instruction layer, intent layer, tool layer, model layer, and output layer can be used as the link tracking data corresponding to the intelligent agent. Of course, the recorded data content can also be further processed to obtain link tracking data.

[0067] It is worth noting that the first link tracking data, the second link tracking data, the third link tracking data and the fourth link tracking data can all be obtained through the above-mentioned layered recording architecture.

[0068] Therefore, through the above-mentioned layered recording architecture, the data content generated by the intelligent agent at the instruction layer, intention layer, tool layer, model layer and output layer can be recorded, so as to comprehensively and in detail record the link information of the intelligent agent in each key link, thereby providing support for the optimization of the intelligent agent.

[0069] In some possible implementations, the link tracking data includes at least a first call chain corresponding to a workflow invoked by an agent. Accordingly, when a workflow node included in the workflow invoked by the agent is executed, a span corresponding to the workflow node is created based on the node information corresponding to the workflow node. Furthermore, based on the topological relationships between the workflow nodes and the created spans, the first call chain corresponding to the workflow is constructed.

[0070] Here, when debugging an agent's workflow, the third-link tracking data obtained can be the first call chain corresponding to the workflow called by the agent. Of course, when the agent is running online, if the agent calls a workflow, the first-link tracking data will also include the first call chain corresponding to the workflow called by the agent.

[0071] The workflow invoked by the agent can include multiple workflow nodes. When a workflow node in the workflow is executed, a span corresponding to the workflow node is created based on the node information corresponding to the workflow node. The node information can include the node identifier, node type, and the workflow identifier to which the node belongs.

[0072] Exemplarily, node types may include start nodes, end nodes, large language model nodes, knowledge base nodes, question and answer base nodes, tool nodes, term base nodes, etc. It should be understood that the node type is a predefined type of workflow node.

[0073] Each executed workflow node corresponds to a span in the first call chain, and the properties of the corresponding span can be defined through the node information of the workflow node, thereby creating a span corresponding to the workflow node.

[0074] After the workflow is executed, a first call chain corresponding to the workflow may be constructed based on the topological relationship between the workflow nodes and the created spans.

[0075] The topological relationship between workflow nodes can refer to the parent-child relationship between each workflow node (represented by the child_of parameter). For all created spans, the parent-child relationship between created spans can be determined through the topological relationship between workflow nodes, thereby forming the first call chain corresponding to the workflow.

[0076] Thus, through the above implementation, the first call chain corresponding to the workflow can be constructed, thereby tracing the workflow. For example, through the first call chain, the user can query whether the execution of the workflow meets expectations.

[0077] In some possible implementations, log information corresponding to a session between the agent and the user can also be obtained for the first link tracking data. Then, based on the session identifier corresponding to the log information, the log information can be associated with the first link tracking data corresponding to the session, so that the log information corresponding to the session can be viewed through the first link tracking data corresponding to the session.

[0078] Here, the agent can actively upload log information to the Collector, and then store the log information in the distributed search engine, specifically as an observe-real-time-log-YYYY-MM index.

[0079] Next, the log information is associated with the first link tracking data corresponding to the session through the session identifier corresponding to the log information, thereby connecting the log information and the first link tracking data in series.

[0080] It should be understood that a session may include multiple conversations, and the first link tracking data may include the link tracking data corresponding to each conversation. Associating the first link tracking data corresponding to a session with the session identifier corresponding to the log information may be associating the link tracking data corresponding to each conversation with the log information corresponding to the conversation.

[0081] Therefore, through the above implementation, the log information of the session can be associated with the first link tracking data of the session, thereby facilitating the tracking of the complete execution path of the distributed request.

[0082] In some possible implementations, the link tracking data includes a second call chain corresponding to the agent, the second call chain corresponding to a conversation between the agent and the user, and the second call chain including spans executed by the agent and the topological relationships between the spans. Accordingly, the second call chain can be displayed.

[0083] Here, the second call chain corresponding to the agent refers to the execution path of each service when the agent executes the request. Figure 2 FIG. 1 is a schematic diagram of a second call chain according to some embodiments. Figure 2As shown, a conversation between an agent and a user includes multiple consecutive conversations. Each conversation corresponds to a link tracking data. Each link tracking data includes a second call chain. Each second call chain includes one or more spans executed by the agent and the topological relationship between spans. In the second call chain, the first span is the root node, and each root node to the last node represents a call chain. The topological relationship between spans is the parent-child relationship between spans. For example, in Figure 2 In the example, the first span 201 serves as the parent node of the second span 202 , and the second span 202 serves as the child node of the first span 201 .

[0084] It is worth noting that the first link tracing data, the second link tracing data, the third link tracing data and the fourth link tracing data may all include the second call chain.

[0085] Therefore, by displaying the second call chain, the execution path of the agent can be clearly displayed.

[0086] In some possible implementations, in step 130, a workflow debugging interface may be displayed, the workflow debugging interface including the workflow to be debugged. Then, in response to a query request triggered on the workflow debugging interface for viewing a target debugging record corresponding to the workflow, target link tracking data corresponding to the target debugging record is determined from the stored link tracking data, and the target link tracking data corresponding to the target debugging record is displayed on the workflow debugging interface.

[0087] Here, the workflow debugging interface is used to adjust the workflow. The workflow debugging interface can display a run history control. By triggering the run history control, the workflow debugging interface can display the workflow's corresponding historical debugging records. The historical debugging records can be sorted in descending time order.

[0088] A query request for viewing the target debug record corresponding to a workflow can be triggered by selecting a historical debug record. For example, if a user selects a historical debug record, the selected historical debug record becomes the target debug record. Alternatively, a query request for viewing the target debug record corresponding to a workflow can be triggered by clicking a quick query control in the workflow debugging interface. When a user clicks the quick query control in the workflow debugging interface, the most recent historical debug record is selected as the target debug record.

[0089] Then, the target link tracking data corresponding to the target debugging record is retrieved from the stored link tracking data, and the target link tracking data corresponding to the target debugging record is displayed in the workflow debugging interface. It should be noted that the target link tracking data corresponding to the target debugging record may include the first call chain corresponding to the workflow and the running information corresponding to the target debugging record.

[0090] Figure 3 FIG. 1 is a schematic diagram showing target link tracking data corresponding to a target debugging record according to some embodiments. Figure 3 As shown, first interface 300 may display the first call chain and execution information 301 corresponding to the target debugging record. Execution information 301 includes information such as the duration, tokens consumed, time taken to reply to the first character, start time, end time, and input corresponding to the target debugging record. It should be noted that first interface 300 may be displayed within the workflow debugging interface.

[0091] In some embodiments, the span execution status and span running time corresponding to each span included in the first call chain may also be displayed in the first call chain.

[0092] like Figure 3 As shown, the span execution status 303 and span running time 302 corresponding to each span can be displayed in the vicinity of each span in the first call chain. The span execution status 303 is used to indicate whether the span execution is successful. If the span execution is successful, it can be marked with a "√" mark, and if the span execution fails, it can be marked with an "×" mark. Of course, in other embodiments, the span execution status 303 can also be displayed with other marks, for example, with a "success" mark or a "failure" mark.

[0093] In some embodiments, when each span included in the first call chain is selected, span detail information corresponding to the selected span may be displayed.

[0094] Each span included in the first call chain can be selected. When a span is selected, the span details corresponding to the selected span can be displayed. The span details can include the span type, span execution status, span duration, span first character response duration, span tokens consumed, span first character response time, span name, span call type, span start time, span end time, etc.

[0095] like Figure 3 As shown, when the user selects the “user input” span in the first call chain, the span detail information 304 corresponding to the “user input” span may be displayed on the first interface 300 .

[0096] In some embodiments, when each span included in the first call chain is selected, the workflow node corresponding to the span may be positioned in the middle of the workflow debugging interface.

[0097] Following the above embodiment, the first interface 300 may be an interface displayed in the workflow debugging interface. Therefore, when a span is selected, the workflow node corresponding to the span may be positioned in the middle of the workflow debugging interface, so that the user can quickly locate the workflow node.

[0098] In some embodiments, when there is an error span among the spans included in the first call chain, error information corresponding to the error span may be displayed.

[0099] The error span is an abnormal span, that is, it can be understood as a span whose execution status is failed. An error reporting control 305 can be provided in the first interface 300, and the error reporting control 305 is used to display error information corresponding to the error span.

[0100] Therefore, through the above implementation, rich link tracking information can be provided to the user during workflow debugging to help the user debug the workflow.

[0101] Figure 4 FIG. 1 is a schematic diagram showing second link tracking data according to some embodiments. Figure 4 As shown, when a user views the second link tracing data corresponding to a particular debugging session of an agent, the corresponding call chain and the agent's running information can be displayed in the second interface 400. Of course, when the user selects a span in the second interface 400, the span details corresponding to the selected span will also be displayed, which is consistent with the display method of the first interface 300.

[0102] It is worth noting that the second interface 400 can be displayed in an agent debugging interface for debugging the agent.

[0103] Figure 5 FIG. 1 is a schematic diagram showing fourth link tracking data according to some embodiments. Figure 5 As shown, in the third interface 500, the evaluation results of the intelligent agent (such as object A, object B, and object C) are displayed. When the user views the fourth link tracking data corresponding to object B in the third interface 500, the fourth interface 501 can be displayed in the adjacent area of ​​the third interface 500. The call chain corresponding to the intelligent agent and the operation information of the intelligent agent are displayed in the fourth interface 501. Of course, when the user selects a span in the call chain of the fourth interface 501, the span details information corresponding to the selected span will also be displayed, which is consistent with the display method of the first interface 300. It is worth noting that the third interface 500 and the fourth interface 501 can be displayed in the intelligent agent evaluation interface used to evaluate the intelligent agent.

[0104] Figure 6 FIG. 1 is a schematic diagram of a link tracking device for an intelligent agent according to some embodiments. Figure 6 As shown, the embodiment of the present disclosure provides a link tracking device 600 of an intelligent agent, and the link tracking device 600 of an intelligent agent includes: Acquisition module 601 is configured to acquire link tracking data of an agent throughout its entire life cycle. The link tracking data includes data content generated by the agent from receiving a request to outputting a response result corresponding to the request. The link tracking data includes first link tracking data when the agent is running online, second link tracking data when the agent is being debugged, third link tracking data when the agent's workflow is being debugged, and fourth link tracking data when the agent is being evaluated. The storage module 602 is configured to store the link tracking data based on the original identifier corresponding to the link tracking data; wherein the original identifier includes at least one of a session identifier, an agent identifier, a user identifier, and a tenant identifier corresponding to the link tracking data; The display module 603 is configured to, in response to the query request, determine the target link tracking data corresponding to the original identifier carried in the query request from the stored link tracking data, and display the target link tracking data.

[0105] Optionally, the acquisition module 601 is specifically configured to: Recording the original instructions received by the agent at the instruction layer; Recording data content generated by the agent at the intent layer, the data content including at least one of a recognition algorithm for identifying the intent of the original instruction, an intent category corresponding to the intent, and a confidence level corresponding to the intent; Recording data content generated by the agent at the tool layer, the data content including at least one of a tool name corresponding to a tool called by the agent, input parameters of the tool, and a return result of the tool; Recording data content generated by the agent at the model layer, the data content including at least one of input data of the model layer, algorithm parameters used by the model layer, and output results of the model layer; Recording the data content generated by the agent at the output layer, including the response result corresponding to the original instruction output by the agent and the way in which the agent integrates the response result; The link tracking data is obtained based on the data content recorded in the instruction layer, the intent layer, the tool layer, the model layer and the output layer.

[0106] Optionally, the link tracking data includes at least a first call chain corresponding to the workflow called by the agent, and the acquisition module 601 is specifically configured to: When a workflow node included in the workflow called by the agent is executed, creating a span corresponding to the workflow node based on the node information corresponding to the workflow node; Based on the topological relationship between the workflow nodes and the created span, a first call chain corresponding to the workflow is constructed.

[0107] Optionally, when the link tracking data is the first link tracking data, the link tracking device 600 of the intelligent agent further includes: a log acquisition unit, configured to acquire log information corresponding to a conversation between the agent and the user; The associating unit is configured to associate the log information with the first link tracking data corresponding to the session based on the session identifier corresponding to the log information, so as to view the log information corresponding to the session through the first link tracking data corresponding to the session.

[0108] Optionally, the link tracking data includes a second call chain corresponding to the agent, the second call chain being a call chain corresponding to a conversation between the agent and the user, and the second call chain including spans executed by the agent and topological relationships between spans; the display module 603 is specifically configured to: The second call chain is displayed.

[0109] Optionally, the display module 603 is specifically configured to: Displaying a workflow debugging interface, wherein the workflow debugging interface includes the workflow to be debugged; In response to a query request for viewing the target debugging record corresponding to the workflow triggered in the workflow debugging interface, target link tracking data corresponding to the target debugging record is determined from the stored link tracking data, and the target link tracking data corresponding to the target debugging record is displayed in the workflow debugging interface, wherein the target link tracking data includes the first call chain corresponding to the workflow and the running information corresponding to the target debugging record.

[0110] Optionally, the display module 603 is further configured to: Displaying the span execution status and span running time corresponding to each span included in the first call chain in the first call chain; and / or When each span included in the first call chain is selected, displaying span detail information corresponding to the selected span; and / or When each span included in the first call chain is selected, positioning the workflow node corresponding to the span in the middle of the workflow debugging interface; and / or When an error span exists among the spans included in the first call chain, error information corresponding to the error span is displayed.

[0111] The functional logic executed by each functional module in the link tracking device 600 of the above-mentioned intelligent entity has been described in detail in the part about the method, and will not be repeated here.

[0112] Reference below Figure 7 , which shows a schematic structural diagram of an electronic device (e.g., a terminal device or server) 700 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0113] like Figure 7 As shown, electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or programs loaded from a storage device 708 into a random access memory (RAM) 703. Various programs and data required for the operation of electronic device 700 are also stored in RAM 703. Processing device 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0114] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0115] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0116] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0117] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0118] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0119] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: Acquire link tracking data of an agent throughout its entire life cycle, the link tracking data including data content generated by the agent from the time the agent receives a request to the time the agent outputs a response result corresponding to the request, the link tracking data including first link tracking data when the agent is running online, second link tracking data when the agent is debugged, third link tracking data when the agent's workflow is debugged, and fourth link tracking data when the agent is evaluated; store the link tracking data based on an original identifier corresponding to the link tracking data; wherein the original identifier includes at least one of a session identifier, an agent identifier, a user identifier, and a tenant identifier corresponding to the link tracking data; in response to a query request, determine target link tracking data corresponding to the original identifier carried by the query request from the stored link tracking data, and display the target link tracking data.

[0120] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The modules involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a module does not necessarily limit the module itself.

[0123] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0124] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the scope of the above disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0126] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0127] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. A link tracking method for an intelligent agent, characterized in that: include: Acquire link tracking data for the agent throughout its life cycle, the link tracking data including data content generated by the agent from the time the agent receives a request to the time the agent outputs a response result corresponding to the request. The link tracking data includes first link tracking data when the agent is running online, second link tracking data when the agent is being debugged, third link tracking data when the agent's workflow is being debugged, and fourth link tracking data when the agent is being evaluated; storing the link tracking data based on an original identifier corresponding to the link tracking data; wherein the original identifier includes at least one of a session identifier, an agent identifier, a user identifier, and a tenant identifier corresponding to the link tracking data; In response to the query request, target link tracking data corresponding to the original identifier carried in the query request is determined from the stored link tracking data, and the target link tracking data is displayed.

2. The method according to claim 1, characterized in that The link tracking data is obtained by the following steps: Recording the original instructions received by the agent at the instruction layer; Recording data content generated by the agent at the intent layer, the data content including at least one of a recognition algorithm for identifying the intent of the original instruction, an intent category corresponding to the intent, and a confidence level corresponding to the intent; Recording data content generated by the agent at the tool layer, the data content including at least one of a tool name corresponding to a tool called by the agent, input parameters of the tool, and a return result of the tool; Recording data content generated by the agent at the model layer, the data content including at least one of input data of the model layer, algorithm parameters used by the model layer, and output results of the model layer; Recording the data content generated by the agent at the output layer, including the response result corresponding to the original instruction output by the agent and the way in which the agent integrates the response result; The link tracking data is obtained based on the data content recorded in the instruction layer, the intent layer, the tool layer, the model layer and the output layer.

3. The method according to claim 1, characterized in that The link tracking data includes at least a first call chain corresponding to the workflow called by the agent, and the first call chain is obtained by the following steps: When a workflow node included in the workflow called by the agent is executed, creating a span corresponding to the workflow node based on the node information corresponding to the workflow node; Based on the topological relationship between the workflow nodes and the created span, a first call chain corresponding to the workflow is constructed.

4. The method according to claim 1, wherein In a case where the link tracking data is the first link tracking data, the method further includes: Obtaining log information corresponding to the conversation between the agent and the user; Based on the session identifier corresponding to the log information, the log information is associated with the first link tracking data corresponding to the session, so that the log information corresponding to the session can be viewed through the first link tracking data corresponding to the session.

5. The method according to any one of claims 1 to 4, characterized in that The link tracking data includes a second call chain corresponding to the agent, the second call chain is a call chain corresponding to a conversation between the agent and the user, and the second call chain includes spans executed by the agent and topological relationships between spans; The displaying of the target link tracking data includes: The second call chain is displayed.

6. The method according to any one of claims 1 to 4, characterized in that The step of determining, in response to the query request, target link tracking data corresponding to the original identifier carried in the query request from the stored link tracking data, and displaying the target link tracking data, includes: Displaying a workflow debugging interface, wherein the workflow debugging interface includes the workflow to be debugged; In response to a query request for viewing the target debugging record corresponding to the workflow triggered in the workflow debugging interface, target link tracking data corresponding to the target debugging record is determined from the stored link tracking data, and the target link tracking data corresponding to the target debugging record is displayed in the workflow debugging interface, wherein the target link tracking data includes the first call chain corresponding to the workflow and the running information corresponding to the target debugging record.

7. The method according to claim 6, characterized in that The method further comprises: Displaying the span execution status and span running time corresponding to each span included in the first call chain in the first call chain; and / or When each span included in the first call chain is selected, displaying span detail information corresponding to the selected span; and / or When each span included in the first call chain is selected, positioning the workflow node corresponding to the span in the middle of the workflow debugging interface; and / or When an error span exists among the spans included in the first call chain, error information corresponding to the error span is displayed.

8. A link tracking device for an intelligent agent, characterized in that: include: an acquisition module configured to acquire link tracking data of an agent throughout its entire life cycle, the link tracking data including data content generated by the agent from the time the agent receives a request to the time the agent outputs a response result corresponding to the request, the link tracking data including first link tracking data when the agent is running online, second link tracking data when the agent is being debugged, third link tracking data when the agent's workflow is being debugged, and fourth link tracking data when the agent is being evaluated; A storage module is configured to store the link tracking data based on an original identifier corresponding to the link tracking data; wherein the original identifier includes at least one of a session identifier, an agent identifier, a user identifier, and a tenant identifier corresponding to the link tracking data; The display module is configured to determine, in response to a query request, target link tracking data corresponding to the original identifier carried in the query request from the stored link tracking data, and display the target link tracking data.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images