Method and device for evaluating processing entity system, equipment and storage medium

Abstract

The embodiment of the invention provides a method and device for evaluating a processing entity system, equipment and a storage medium. The method comprises the steps of obtaining operation data generated by the processing entity system in a task execution process; an extraction mode is determined according to the development framework type of the processing entity system, track data is extracted from the operation data according to the extraction mode, the track data at least comprises processing entity node data, and the processing entity node data records information related to the operation process of a processing entity participating in task execution; and evaluating the performance of the processing entity system using the trajectory data. According to the method, a universal and effective trajectory data normal form compatible with various development frameworks is defined for processing entity system evaluation. According to the method, the operation data with large differentiation generated when the entity system executes tasks on various development frameworks can be collected and processed, and the track data is extracted from the collected operation data in a special extraction mode.

Description

Technical Field

[0001] This disclosure relates to a method, apparatus, electronic device, computer program product, and non-transitory computer-readable storage medium for evaluating processing physical systems. Background Technology

[0002] In recent years, breakthroughs in artificial intelligence technology, particularly in Large Language Models (LLMs), have greatly propelled the development and application of agent technology (also known as intelligent agent technology). An agent system, as an AI system capable of perceiving its environment, making autonomous decisions, and invoking tools to execute complex tasks, typically comprises key components such as multiple agents, a Large Language Model (LLM), tool calling capabilities, and a domain-specific knowledge base. Benefiting from its high degree of autonomy and task processing capabilities, agent technology has demonstrated broad application prospects in numerous fields, including customer service, data analysis, automated processes, and scientific research assistance, and continues to expand into deeper and more complex application scenarios. Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] At least one embodiment of this disclosure provides a method for evaluating a processing entity system, comprising: obtaining runtime data generated by the processing entity system during task execution; determining an extraction method based on the development framework type of the processing entity system; extracting trajectory data from the runtime data according to the extraction method, wherein the trajectory data includes at least processing entity node data, the processing entity node data recording information related to the runtime process of the processing entity participating in the task execution; and using the trajectory data to evaluate the performance of the processing entity system.

[0005] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to perform the method as described above.

[0006] At least one embodiment of this disclosure provides a computer program product, including a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the method described above.

[0007] At least one embodiment of this disclosure provides an electronic device, including: one or more processors; and one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the method as described above.

[0008] At least one embodiment of this disclosure provides an apparatus for evaluating a processing entity system, comprising: an acquisition module configured to acquire runtime data generated by the processing entity system during task execution; an extraction module configured to determine an extraction method based on the development framework type of the processing entity system, and extract trajectory data from the runtime data according to the extraction method, wherein the trajectory data includes at least processing entity node data, the processing entity node data recording information related to the runtime of the processing entity participating in the task execution; and an evaluation module configured to evaluate the performance of the processing entity using the trajectory data. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 This illustration schematically depicts an application scenario of a method and apparatus for evaluating a processing entity system provided in at least one embodiment of this disclosure;

[0011] Figure 2 A flowchart illustrating a method for evaluating a processing entity system provided in at least one embodiment of this disclosure is shown schematically.

[0012] Figures 3A-3D The illustration shows a schematic diagram of a page for evaluating a processing entity system provided in at least one embodiment of the present disclosure;

[0013] Figure 4 This schematically illustrates a structural block diagram of an apparatus for evaluating a processing entity system provided in at least one embodiment of the present disclosure;

[0014] Figure 5 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0017] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0019] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0020] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and authorization should be obtained from the relevant users. Among them, relevant users may include any type of rights holder, such as individuals, enterprises, and groups.

[0023] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly inform the user that the requested operation will require obtaining and using the user's information, thereby enabling the relevant user to choose whether to provide information to the software or hardware such as the electronic device, application, server, or storage medium that performs the operation of the technical solution disclosed herein based on the prompt message.

[0024] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide information to the electronic device.

[0025] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] With the development of computer and electronic technologies, artificial intelligence (AI) technology has rapidly developed and been widely applied. Taking AI models with a large number of parameters (also known as large AI models, or simply "large models," such as large language models (LLMs)) constructed by AI networks as an example, AI models have been widely used in search engines, entity processing, related vertical industries, and basic disciplines, promoting the intelligent development of various industries. For example, the accuracy of content generated by AI models can be improved by combining knowledge bases with AI models. It should be noted that in this embodiment, the AI ​​model includes any one or a combination of multiple types of large language models, visual large models, audio large models, and multimodal large models.

[0027] With the development of artificial intelligence (AI) models, AI-driven processing entity (AI agent) systems have been widely applied. Especially with the emergence of AI models, various industries have begun deploying AI model-driven processing entity systems (also known as AI model agent systems) to accelerate output. For example, the content generated by AI model processing entity systems can be constrained to ensure that the system outputs desired content. AI model processing entity systems are typically able to perceive information in their environment, make decisions, and take actions to achieve specific goals or tasks.

[0028] Processing entity systems, as artificial intelligence systems capable of perceiving the environment, making autonomous decisions, and calling tools to perform complex tasks, may include key components such as multiple processing entities, large language models (LLMs), tool calling components, and domain-specific knowledge bases.

[0029] For example, an LLM (Limited Language Management) can be the core inference engine of a processing entity system, acquiring general language understanding and generation capabilities through pre-training on massive amounts of text data. For instance, an LLM can undertake key functions such as task parsing, logical reasoning, and decision planning within a processing entity system. Tool invocation can be a crucial mechanism for extending the capabilities of a processing entity system. For example, through predefined application programming interfaces (APIs), a processing entity system can actively invoke external tools (e.g., calculators, databases, specialized software, or application data interfaces (APIs) to perform operations that an LLM cannot directly execute (e.g., real-time data queries, complex calculations). A knowledge base can be a dedicated information storage and retrieval system for a processing entity system, storing structured / unstructured domain knowledge (e.g., product manuals, industry rules, private data). For instance, through vectorized embedding and similarity matching techniques, a processing entity system can dynamically retrieve precise task-related information, supplementing the static knowledge deficiencies of an LLM.

[0030] For example, during user interaction with an AI model or a processing entity system, it is necessary to define the role, capabilities, rules, and behavioral boundaries of the AI ​​model or processing entity system through interactive guidance information (or sequences of interactive instructions, behavioral guidance parameters (sequences), model control instructions (sequences), etc.), thereby constraining the content generated by the AI ​​model in processing the entity system. Interactive guidance information (or sequences of interactive instructions, behavioral guidance parameters (sequences), model control instructions (sequences)) may include prompts. In the following text, interactive guidance information, sequences of interactive instructions, sequences of behavioral guidance parameters (sequences), and sequences of model control instructions (sequences) are used interchangeably.

[0031] As the application scenarios of processing entity systems become increasingly diversified, the complexity of tasks increases significantly, and the interaction logic of their internal components (e.g., LLM, tool calls, knowledge bases) becomes more complex, it is necessary to effectively evaluate the performance and quality of processing entity systems and adjust or improve them based on the evaluation results.

[0032] The performance of processing entity systems can be evaluated using manual assessment methods. For example, professional evaluators can subjectively judge the output results of the processing entity system based on preset scoring criteria (e.g., task completion, answer accuracy, logical coherence, etc.). While manual assessment can capture some subtle differences at the semantic level, its inherent limitations severely restrict its application effectiveness. For instance, manual assessment methods are highly subjective, inefficient, difficult to apply on a large scale, and struggle to assess the quality of the internal execution processes of the processing entity.

[0033] Furthermore, end-to-end automated evaluation methods can be used to assess the performance of processing entity systems. For example, automated methods can be used to directly evaluate the inputs and final outputs of the processing entity system. By using pre-defined rules or LLM-based evaluation methods, the performance evaluation results of the processing entity system can be obtained by analyzing its inputs and final outputs. However, such methods have a single evaluation dimension, primarily focusing on the final output of the processing entity and neglecting the performance evaluation of the internal execution process. Moreover, the significant differences in runtime data generated by processing entity systems across different development frameworks during task execution greatly complicate automated evaluation.

[0034] At least one embodiment of this disclosure provides a technical solution that enables automated evaluation of processing entity systems across development frameworks and guides the identification of problems and continuous optimization of processing entity systems, thereby overcoming the obvious shortcomings of using manual or automated evaluation methods to assess the performance of processing entity systems in terms of cost, efficiency, and development framework compatibility.

[0035] To at least partially solve or alleviate at least one of the above-mentioned technical problems, at least one embodiment of this disclosure provides a method for evaluating a processing entity system, comprising: obtaining runtime data generated by the processing entity system during task execution; determining an extraction method based on the development framework type of the processing entity system; extracting trajectory data from the runtime data according to the extraction method, wherein the trajectory data includes at least processing entity node data, the processing entity node data recording information related to the runtime of the processing entity participating in the task execution; and using the trajectory data to evaluate the performance of the processing entity system.

[0036] Based on the method for evaluating a processing entity system provided in at least one embodiment of the present disclosure, at least one embodiment of the present disclosure also provides an apparatus, electronic device, computer program product, and non-transitory computer-readable storage medium for evaluating a processing entity system.

[0037] According to at least one embodiment of the present disclosure, a method, apparatus, electronic device, computer program product, and non-transitory computer-readable storage medium for evaluating processing entity systems define a general and effective trajectory data paradigm compatible with various development frameworks for evaluating processing entity systems. For example, it is possible to collect significantly differentiated operational data generated by the processing entity system when performing tasks on various development frameworks, and extract the aforementioned trajectory data from the collected operational data using a dedicated extraction method. In this way, the applicability of the evaluation of the processing entity system can be enhanced, and the compatibility with development frameworks can be improved, solving the problem of heterogeneous data sources and reducing the difficulty and cost of evaluating processing entity systems across development frameworks.

[0038] The solutions in at least one embodiment of this disclosure can be applied to products used for evaluating processing entity systems or other types of evaluation products, or they can be a functional module of a processing entity or artificial intelligence model. It is understood that before using the technical solutions disclosed in the embodiments of this disclosure, relevant users should be informed of the type, scope of use, and usage scenarios of the information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and their authorization should be obtained. Relevant users include, but are not limited to: the processing entity or artificial intelligence model business entity and its users, as well as other possible related parties.

[0039] The embodiments and some examples of this disclosure will now be described in detail with reference to the accompanying drawings.

[0040] Figure 1 The illustration shows an application scenario of the method and apparatus for evaluating the interaction process of an entity system, provided in at least one embodiment of the present disclosure.

[0041] like Figure 1 As shown, the application scenario 100 of this embodiment includes a user 101 and a client device 102. The user 101 may include, for example, a user of the processing entity system, an evaluator of the processing entity system, or other relevant parties. The client device 102 may include, for example, any electronic device capable of providing services to the user based on the processing entity system, evaluating the performance of the processing entity system, and providing an interactive page for user operation and use. Examples include mobile phones, tablets, portable computers, desktop computers, smart wearable devices, smart home appliances, or smart vehicle terminals, etc. The embodiments disclosed herein do not limit this.

[0042] For example, in response to user 101's interaction with the interactive page, client device 102 can obtain the interaction guidance information provided by the interaction, and based on the interaction guidance information, obtain the response content corresponding to the interaction guidance information, and output the response content to user 101. For example, the interaction guidance information may include information entered on the interactive page, or it may include information obtained by converting speech. For example, the interaction guidance information may instruct a test task to be executed by the processing entity system, and the processing entity system may execute the corresponding test task based on the interaction guidance information and return the execution result of the test task to user 101.

[0043] For example, the client device 102 runs an operating system, which may have instant messaging clients, search clients, AI assistants, and other clients installed. These clients may be clients used to execute the methods according to embodiments of this disclosure. These clients may include at least a target application capable of acquiring response content generated by the processing entity system in response to interactive guidance information, or a target application integrating a mini-program capable of acquiring response content generated by the processing entity in response to interactive guidance information, or a web page integrating the function of acquiring response content generated by an AI model in response to interactive guidance information. Embodiments of this disclosure do not limit this. According to at least one embodiment of this disclosure, the processing entity may include an intelligent agent (e.g., a main intelligent agent participating in the execution of a task and additional intelligent agents invoked by the main intelligent agent), a part of an intelligent agent, or other entities capable of executing the task to be processed. This disclosure is not limited thereto.

[0044] For example, in some embodiments, the client device 102 may have a server locally deployed to support the operation of the processing entity system, so that the client on the terminal device can send instructions / requests to the server, so that the server responds to the instructions / requests by using the processing entity system to generate response content with interactive guidance information, and sends the response content to the client.

[0045] In at least one embodiment of this disclosure, the client device 102 can obtain runtime data generated by the processing entity system used by the server in the client device 102 during the execution of corresponding tasks or test tasks. The runtime data of the processing entity system can refer to information recording the interaction between the processing entity and the external environment (including user 101, other processing entities, tools, knowledge bases, etc.) and changes in its internal decision-making state during the execution of test tasks, including but not limited to trace data and log information. The runtime data of the processing entity system can include the specific processing path and intermediate results generated when the processing entity executes tasks or test tasks, showing the flow of interactive guidance information within the processing entity system. For example, a software development kit (SDK) can be deployed in the server of the client device 102 to collect or obtain the runtime data of the processing entity, but this disclosure is not limited to this; other runtime data collection methods are also possible.

[0046] In other embodiments, such as Figure 1 As shown, application scenario 100 may also involve a server 103, which can communicate with the client device 102 via wired or wireless communication links. For example, it can be a local area network server, a wide area network server, or a cloud server. For instance, the server 103 may run a background server that supports the client installed on the client device 102. The client on the terminal device can respond to interactive operations by sending instructions / requests to the server on the server 103, or by sending or receiving data, so that the server on the server responds to the instructions / requests by using the processing entity system to generate response content corresponding to the interactive guidance information, and sends the response content to the client on the client device 102. For example, the interactive guidance information can instruct the processing entity system to perform a task or test task. The processing entity system can execute the corresponding task or test task based on the interactive guidance information and return the execution result to the user 101.

[0047] In at least one embodiment of this disclosure, the application scenario 100 may further include a server on server 103, providing interactive guidance information to the processing entity system used by the server, enabling the processing entity system to generate response content based on the task or test task indicated in the interactive guidance information. In at least one embodiment of this disclosure, client device 102 can obtain runtime data generated by the processing entity system used by the server on server 103 during the execution of corresponding tasks. For example, an SDK can be deployed on the server on server 103 to collect or obtain runtime data from the processing entity system. Figure 1 The processing entity system in the process can be developed using various development frameworks, so the variability of the collected runtime data can be very large.

[0048] For example, the method for evaluating a processing entity system provided in at least one embodiment of this disclosure can be implemented in software, hardware, firmware, or any combination thereof.

[0049] For example, the method for evaluating a processing entity system provided in at least one embodiment of this disclosure is applicable to a client device 102 or a server 103, which can load and execute the method for evaluating a processing entity system. The embodiments of this disclosure do not limit this.

[0050] For example, the client device 102 may include a central processing unit (CPU) or graphics processing unit (GPU), digital signal processor (DSP), neural network processing unit (NPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, storage units, etc. The client device 102 is also equipped with an operating system, application programming interfaces (APIs) (e.g., OpenGL (Open Graphics Library), Metal, etc.), etc., and implements the method for evaluating processing entity systems provided in this disclosure embodiment by running code or instructions.

[0051] For example, the client device 102 may also include an output component, such as a display component, which may be a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a quantum dot light-emitting diode (QLED) display, or a light-emitting diode display (e.g., a micro-LED display), etc., and the embodiments disclosed herein are not limited thereto. For example, the display component may display an interactive page, and show a configuration page for configuring the evaluation process for the processing entity, and a result presentation page for displaying the evaluation results of the processing entity.

[0052] The following will combine Figures 2-3D A method for evaluating a processing entity system provided by at least one embodiment of the present disclosure will be described in detail.

[0053] Figure 2 A flowchart illustrating a method for evaluating a processing entity system provided in at least one embodiment of the present disclosure is shown.

[0054] like Figure 2As shown, the method 200 for evaluating a processing entity system in this embodiment includes steps S210 to S240. For example, the executing entity of this method for evaluating a processing entity system can be an electronic device with a corresponding client deployed, or any electronic device communicating with the client; the embodiments of this disclosure do not limit this. According to at least one embodiment of this disclosure, the processing entity may include an intelligent agent (e.g., a main intelligent agent participating in the execution of a task and additional intelligent agents invoked by the main intelligent agent), a part of an intelligent agent, or other entities capable of executing the task to be processed; this disclosure is not limited thereto.

[0055] In step S210, the running data generated by the processing entity system during the execution of the task can be obtained.

[0056] According to at least one embodiment of this disclosure, interactive guidance information can be sent to a processing entity system to instruct the processing entity system to perform a task such as a test task. The interactive guidance information may be, for example, information obtained in response to a user's interactive operation, such as multimedia information in text, image, and / or audio form. The interactive guidance information may be a prompt (also called a prompt) guiding the processing entity system to generate specific output, and may be a question, a piece of text, or a formatted instruction.

[0057] According to at least one embodiment of this disclosure, interactive guidance information can guide a processing entity system to perform tasks such as test tasks. The tasks can be configured by the user according to testing needs (e.g., evaluation metrics for processing entity systems developed using various development frameworks). Additionally or alternatively, the tasks can be routine tasks performed by processing entity systems developed using various development frameworks.

[0058] According to at least one embodiment of this disclosure, an SDK can be deployed on the server side of the processing entity system. For example, the corresponding SDK can be used to collect or obtain the runtime data of the processing entity. This disclosure is not limited thereto, and other methods of collecting runtime data are also possible. For example, processing entities developed using various development frameworks can automatically report their runtime data. As mentioned above, the runtime data of the processing entity system can refer to records of the interaction between the processing entity system and the external environment and changes in its internal decision-making state during the execution of test tasks.

[0059] In step S220, the extraction method can be determined according to the development framework type of the processing entity system. According to the extraction method, trajectory data is extracted from the running data. The trajectory data includes at least processing entity node data, which records information related to the running process of the processing entity participating in the task execution.

[0060] According to at least one embodiment of this disclosure, the processing entity system can be developed using various development frameworks. Development frameworks may include tool libraries, specifications, and application programming interfaces (APIs), but this disclosure is not limited to these. Development frameworks can assist developers in building processing entity systems, and have a significant impact on the programming model, tool invocation methods, and state management logic of the processing entity system. Development frameworks may include, but are not limited to, LangChain / LangGraph, LlamaIndex, AutoGen, Semantic Kernel, and Eino, but this disclosure is not limited to these and may include more or fewer development frameworks.

[0061] According to at least one embodiment of this disclosure, the differences in runtime data generated by processing entity systems under different development frameworks can be very large, and such highly variable runtime data may be difficult to directly use for evaluating the performance of the processing entity system. Furthermore, the runtime data is massive in volume and contains a large amount of information irrelevant to the evaluation of the processing entity system. Such irrelevant information may increase the computational load of the performance evaluation process and interfere with the evaluation results.

[0062] According to at least one embodiment of this disclosure, the extraction method can be determined based on the development framework type of the processing entity system, and trajectory data can be extracted from the running data according to the extraction method.

[0063] According to at least one embodiment of this disclosure, a corresponding adapter can be determined based on the development framework type of the processing entity system, and the corresponding adapter can be used to extract trajectory data from the runtime data. For example, different adapters can be designed for different development frameworks, and the adapter can understand the runtime data generated by the processing entity system under the corresponding development framework and extract it as trajectory data.

[0064] According to at least one embodiment of this disclosure, a corresponding field mapping rule can be determined based on the development framework type of the processing entity system, and trajectory data can be extracted from the runtime data using the corresponding field mapping rule. For example, different field mapping rules can be designed for different development frameworks, and the runtime data generated by the processing entity system under the corresponding development framework can be extracted as trajectory data using the field mapping rule. The above-described method of determining the extraction method based on the development framework type of the processing entity system is merely an example, and this disclosure is not limited thereto.

[0065] According to at least one embodiment of this disclosure, the method defines a standardized trajectory data paradigm. By extracting the highly diverse operational data generated by processing entity systems under different development frameworks into trajectory data of the standardized trajectory data paradigm, the evaluation difficulty of processing entity systems can be reduced, and the cross-development framework compatibility of the processing entity system evaluation mechanism can be increased. The standardized trajectory data paradigm can specify that the trajectory data includes at least processing entity node data. The processing entity node data can record information related to the operation process of the processing entity. For example, during the execution of a task such as a test, the processing entity system can execute the task through a main processing entity or by calling multiple additional processing entities through the main processing entity. The processing entity node data can include the input and output data of processing entities participating in the execution of the task, such as the main processing entity and multiple additional processing entities that may be called, as well as the input and output data of intermediate inference / tool ​​calls and other steps.

[0066] In step S230, the performance of the processing entity system can be evaluated using trajectory data.

[0067] According to at least one embodiment of this disclosure, the performance of a processing entity system can be evaluated using standardized trajectory data extracted from highly differentiated operational data generated by processing entity systems through different development framework types. In this way, the development framework compatibility of the processing entity system evaluation mechanism can be improved. Furthermore, since a large amount of data irrelevant to the evaluation process is ignored or removed during extraction, the computational load can be reduced and the accuracy of the evaluation results improved.

[0068] Figures 3A-3D The illustration shows a schematic diagram of a page for evaluating a processing entity system provided in at least one embodiment of the present disclosure.

[0069] According to at least one embodiment of this disclosure, referring to Figure 3A In response to obtaining runtime data generated by the processing entity system during task execution, a first area 3100 may be provided. For example, a user can import the obtained runtime data generated by the processing entity system during task execution into a test platform 3000. The test platform 3000 may provide the first area 3100 to display runtime data 3110 and trajectory data 3120. For example, the first area 3100 may include sub-areas for displaying runtime data 3110 and trajectory data 3120 to facilitate comparison of runtime data 3110 and trajectory data 3120.

[0070] According to at least one embodiment of this disclosure, runtime data 3110 may include at least one of tracking data and log data generated by the processing entity system during the execution of tasks. For example, tracking data may be a structured and high-fidelity record of the processing entity system's execution generated based on observations. For example, log data may be general and flexible, but relatively raw and possibly unstructured runtime information output from the processing entity system.

[0071] According to at least one embodiment of this disclosure, for tracking data, span information such as span type and span name can be identified. A standard trajectory paradigm (which may be predefined or user-defined, but is not limited thereto) can be used to extract trajectory data of the standard paradigm from the tracking data based on the aforementioned span information.

[0072] According to at least one embodiment of this disclosure, a large language model can be used to understand the standard paradigm of log data and trajectory data. Furthermore, the large language model can be used to extract trajectory data from the aforementioned log data. Additionally or alternatively, the test platform can support user-written scripts to extract trajectory data from the aforementioned log data.

[0073] According to at least one embodiment of this disclosure, the standardized trajectory data paradigm may specify that the trajectory data may also include at least one of tool node data, model node data, and root node data.

[0074] According to at least one embodiment of this disclosure, the tool node data may record information related to the execution process of the invoked tool. For example, the tool node data may record the name of the invoked tool, the tool invocation parameters used to invoke the tool, the input and output results returned after the tool is executed, and the execution status of the tool invocation (success, failure, timeout, etc.).

[0075] According to at least one embodiment of this disclosure, model node data may record information related to the operation of the invoked model (e.g., a large language model, but this disclosure is not limited thereto). For example, model node data may record the inputs and outputs of the invoked LLM, as well as some optional information (e.g., internal "thinking process," such as thought chain steps, self-verification, candidate option evaluation, etc., and the number of tokens consumed).

[0076] According to at least one embodiment of this disclosure, the root node data may record the inputs and outputs of the processing entity system for a task. For example, the root node data may record the initial inputs and final outputs of the processing entity system under test for a task such as a test task, the start and end times of the processing entity system under test executing the test task, the final execution status (e.g., success, failure, abort, etc.), and some other optional information (e.g., total time).

[0077] According to at least one embodiment of this disclosure, the trajectory data can be, for example, as follows:

[0078] namespace XXXX.loop.trajectory

[0079] struct Trajectory {

[0080] / / trace_id

[0081] 1: optional string id

[0082] / / Root node, records information about the entire trajectory

[0083] 2: optional RootStep root_step

[0084] / / Agent step list, recording execution information of processed entities in the trajectory.

[0085] 3: optional list <agentstep>agent_steps

[0086] }

[0087] struct RootStep {

[0088] 1: optional string id / / Unique identifier, retrieved as span_id when importing tracking data

[0089] 2: optional string name / / Name, used when importing tracking data (span_name)

[0090] 3: optional string input / / Input

[0091] 4: optional string output / / Output

[0092] / / System Properties

[0093] 100: optional map<string, string> metadata / / Reserved fields that can hold custom attributes defined by business logic.

[0094] 101: optional BasicInfo basic_info

[0095] 102: optional MetricsInfo metrics_info

[0096] }

[0097] struct AgentStep {

[0098] / / Basic properties

[0099] 1: optional string id / / Unique identifier, obtained by retrieving span_id when importing trajectory data

[0100] 2: optional string parent_id / / Parent identifier, retrieved as parent_span_id when importing trajectory data

[0101] 3: optional string name / / Name, taken as span_name when importing trajectory data

[0102] 4: optional string input / / Input

[0103] 5: optional string output / / Output

[0104] 20: optional list <step>steps / / Child node, indicating the internal steps the entity processes.

[0105] / / System Properties

[0106] 100: optional map<string, string> metadata / / Reserved fields that can hold custom attributes defined by business logic.

[0107] 101: optional BasicInfo basic_info

[0108] 102: optional MetricsInfo metrics_info

[0109] }

[0110] struct Step {

[0111] / / Basic properties

[0112] 1: optional string id / / Unique identifier, obtained by retrieving span_id when importing trajectory data

[0113] 2: optional string parent_id / / Parent identifier, retrieved as parent_span_id when importing trajectory data

[0114] 3: optional StepType type / / Type

[0115] 4: optional string name / / Name, taken as span_name when importing trajectory data

[0116] 5: optional string input / / Input

[0117] 6: optional string output / / Output

[0118] / / Various types of supplementary information

[0119] 20: optional ModelInfo model_info / / Filled when type=model

[0120] / / System Properties

[0121] 100: optional map<string, string> metadata / / Reserved fields that can hold custom attributes defined by business logic.

[0122] 101: optional BasicInfo basic_info

[0123] }

[0124] typedef string StepType(ts.enum="true")

[0125] const StepType StepType_Agent = "agent"

[0126] const StepType StepType_Model = "model"

[0127] const StepType StepType_Tool = "tool"

[0128] struct ModelInfo {

[0129] 1: optional i32 input_tokens

[0130] 2: optional i32 output_tokens

[0131] 3: optional string latency_first_resp / / Time taken for the first data packet, in milliseconds

[0132] 4: optional i32 reasoning_tokens

[0133] 5: optional i32 input_read_cached_tokens

[0134] 6: optional i32 input_creation_cached_tokens

[0135] }

[0136] struct BasicInfo {

[0137] 1: optional string started_at / / Unit: milliseconds

[0138] 2: optional string duration / / Unit: milliseconds

[0139] 3: optional Error error

[0140] }

[0141] struct Error {

[0142] 1: optional i32 code

[0143] 2: optional string msg

[0144] }

[0145] struct MetricsInfo {

[0146] 1: optional string llm_duration / / Unit: milliseconds

[0147] 2: optional string tool_duration / / Unit: milliseconds

[0148] 3: optional map <i32, list <string>tool_errors / / Tool error distribution, formatted as: error code --> list <toolstepid>

[0149] 4: optional double tool_error_rate / / Tool error rate

[0150] 5: optional map <i32, list <string>> model_errors / / Model error distribution, formatted as: error code --> list <modelstepid>

[0151] 6: optional double model_error_rate / / model error rate

[0152] 7: optional double tool_step_proportion / / tool step proportion (denominator is total sub-step)

[0153] 8: optional i32 input_tokens / / input token number

[0154] 9: optional i32 output_tokens / / output token number

[0155] }

[0156] The above examples of trajectory data are provided for reference and understanding only.

[0157] According to at least one embodiment of the present disclosure, the first area 3110 can display a first control 3130. The first control 3130 can be referred to as a preview control or a visualization control, and the present disclosure is not limited thereto. In response to a selection operation for the first control 3130 (for example, by clicking the first control 3130 using a cursor, and the present disclosure is not limited thereto), a second area for previewing or providing a visual representation of the running data 3110 or the trajectory data 3120 can be provided.

[0158] According to at least one embodiment of the present disclosure, with reference to Figure 3B , the second area 3200 can be configured to display one of the trajectory data and the running data. For example, the second area 3200 can include a first sub-area 3210, a second sub-area 3220, and a third sub-area 3230. The first sub-area 3210 can be configured to display at least one of the running data and the trajectory data. For example, the second area is further configured to display a second control 3240 and a third control 3250. Although Figure 3B the second control 3240 and the third control 3250 are shown as independent controls, the present disclosure is not limited thereto, for example, the second control and the third control can be different operating parts of a single control.

[0159] According to at least one embodiment of the present disclosure, as Figure 3C indicated, the second control 3240 can be in a first state or a second state. In response to the second control 3240 being in the first state, the running data can be displayed in the first sub-area 3210. Figure 3B In response to the second control 3240 being in the second state, the trajectory data can be displayed in the second sub-area 3220. Figure 3B The trajectory data is displayed in the first sub-region 3210. Although the correspondence between the first state and the second state and the displayed data is shown above, the present disclosure is not limited thereto. For example, the second state can correspond to displaying the running data, and the first state can correspond to displaying the trajectory data. According to at least one embodiment of the present disclosure, the second control can be switched between the first state and the second state in response to a selection operation (for example, by clicking the second control 3240 by a cursor) on the second control 3240. Additionally or alternatively, in response to the switching of the state of the second control 3240, the display form of the third control 3250 can be changed accordingly, and then such a change in the display form does not mean that the third control 3250 is subjected to a selection operation. That is, the third control 3250 can only follow the second control 3240 to change the display form, without triggering the operation corresponding to the third control 3250.

[0160] According to at least one embodiment of the present disclosure, in response to the pointing component such as a cursor staying on the second control 3240 and the third control 3250 for a predetermined time, the state of the second control 3240 and the function of the third control 3250 can be displayed in the form of a bubble. For example, in response to the pointing component such as a cursor staying on the second control 3240 for a predetermined time, the state of the second control 3240 (for example, the first state of currently displaying the running data or the second state of currently displaying the trajectory data) can be displayed in the form of a bubble. For example, in response to the pointing component such as a cursor staying on the third control 3250 for a predetermined time, the function of the third control 3250 (for example, modifying the configuration rule of the trajectory data) can be displayed in the form of a bubble.

[0161] According to at least one embodiment of the present disclosure, in response to a selection operation on the third control 3250, a third region 3310 as shown in Figure 3D may be provided. The third region 3310 can be configured to display the configuration rule of the trajectory data, for example, configuration rule 1, configuration rule 2. Although the configuration rule 1 and the configuration rule 2 are shown, Figure 3D the present disclosure is not limited thereto, and more or less configuration rules can be included. The configuration rule of the trajectory data can be used to extract the trajectory data from the running data. In the case of the running data being tracking data, the configuration rule can indicate the type or name of the span to be extracted in the tracking data, and the relationship between multiple configuration rules can be an or relationship. The modification of the configuration rule of the trajectory data can be received through the third region 3310. For example, more configuration rules can be added by selecting the "+" symbol, and the configuration rule can be deleted by selecting the "x" symbol. In addition, the configuration rule 1 and the configuration rule 2 can be modified by selecting the configuration rule 1 or the configuration rule 2.

[0162] According to at least one of the embodiments of the present disclosure, the performance of the processing entity system can be evaluated using the trajectory data extracted according to the configuration rule of the trajectory data. Further, in response to receiving a modification of the configuration rule of the trajectory data through the third area 3310, the performance of the processing entity system can be further evaluated using the trajectory data extracted from the running data according to the modified configuration rule of the trajectory data. In this way, in the case that the user is provided with the standard trajectory data paradigm, the user can be further supported to adjust the trajectory data. For example, the user can increase or delete the corresponding node data in batches according to the use demand by modifying the configuration rule. By modifying the configuration rule, the user personalized trajectory data paradigm can be generated, so as to better meet the test requirements of the user. The third area 3310 can further include a control (not shown in Figure 3D ) for applying the above-mentioned configuration rule, and in response to selecting the control for applying the above-mentioned configuration rule, the trajectory data can be extracted from the running data using the above-mentioned configuration condition. The third area 3310 can further include a control (not shown in Figure 3D ) for canceling the above-mentioned modification of the configuration rule, and in response to selecting the control for canceling the above-mentioned modification of the configuration rule, the above-mentioned modification can be canceled.

[0163] According to at least one of the embodiments of the present disclosure, referring to Figure 3B , the second sub-area 3220 can be configured to display a structured diagram obtained according to the data displayed in the first sub-area 3210. According to at least one of the embodiments of the present disclosure, the structured diagram can be a tree diagram, but the present disclosure is not limited thereto, and other forms of structured diagrams are also possible. The structured diagram can include nodes corresponding to the data displayed in the first sub-area 3210. For example, in response to displaying the running data in the first sub-area 3210, the structured diagram corresponding to the running data can be displayed in the second sub-area 3220. For example, in response to displaying the trajectory data in the first sub-area 3210, the structured diagram corresponding to the trajectory data can be displayed in the second sub-area 3220.

[0164] According to at least one of the embodiments of the present disclosure, the root node data of the trajectory data, the processing entity node data, the tool node data, and the model node data can be organized according to a hierarchical structure. Accordingly, the second sub-area 3220 can display a structured diagram generated according to the above-mentioned trajectory data and the hierarchical structure. According to at least one of the embodiments of the present disclosure, in the hierarchical structure, the child node data corresponding to the processing entity node data can include at least one of the tool node data and the model node data, and the child node data corresponding to the root node data can include at least one of the processing entity node data, the tool node data, and the model node data. For example, as Figure 3B As shown, the child nodes of root node 1 can include processing entity node 1 and processing entity node 2. The child nodes of processing entity node 1 can include model node 1 and tool node 1.

[0165] According to at least one embodiment of this disclosure, each of the root node data, processing entity node data, tool node data, and model node data includes a node identifier field. Further, the node data in the processing entity node data, tool node data, and model node data that has parent node data may also include a node identifier field for the parent node data. This configuration of the trajectory data facilitates subsequent evaluation and analysis, as well as the generation of structured diagrams. Displaying the trajectory data in the form of a structured diagram makes it easier for users to understand and reduces the difficulty of user operation.

[0166] According to at least one embodiment of this disclosure, the third sub-region 3230 can be configured to display analysis content for a portion of trajectory data. The analysis content may include the node type, node name, and detailed information indicated by the node data for the selected portion of trajectory data. According to at least one embodiment of this disclosure, in response to a selection operation for at least one node in a structured diagram displayed in the second sub-region, analysis content for the data corresponding to that node can be displayed in the third sub-region. For example, in response to a selection operation for processing entity node 1, analysis content for node data associated with processing entity node 1 can be displayed in the third sub-region 3230.

[0167] Figure 4 The schematic diagram illustrates a structural block diagram of an apparatus for evaluating a processing entity system provided in at least one embodiment of the present disclosure.

[0168] like Figure 4 As shown, the apparatus 400 for evaluating a processing entity system in this embodiment includes an acquisition module 410, an extraction module 420, and an evaluation module 430. For example, these units or modules can be implemented using hardware (e.g., circuit) modules, software modules, firmware modules, etc. The following embodiments are similar and will not be repeated. For example, these units or modules can be implemented using a central processing unit (CPU), a general-purpose graphics processor (GPGPU), a graphics processing unit (GPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, along with corresponding computer instructions.

[0169] The acquisition module 410 can be configured to acquire runtime data generated by the processing entity system during the execution of a task.

[0170] The extraction module 420 can be configured to determine the extraction method based on the development framework type of the processing entity system, and extract trajectory data from the running data according to the extraction method. The trajectory data includes at least processing entity node data, which records information related to the running process of the processing entity participating in the task execution.

[0171] The evaluation module 430 can be configured to evaluate the performance of the processed entity using trajectory data.

[0172] In at least one embodiment of this disclosure, the trajectory data further includes at least one of the following: tool node data, which records information related to the operation of the invoked tool; model node data, which records information related to the operation of the invoked model; and root node data, which records the inputs and outputs of the processing entity system for the task.

[0173] In at least one embodiment of this disclosure, the root node data, processing entity node data, tool node data, and model node data are organized according to a hierarchical structure.

[0174] In at least one embodiment of this disclosure, in the hierarchical structure, the child node data corresponding to the processing entity node data includes at least one of tool node data and model node data; wherein, the child node data corresponding to the root node data includes at least one of processing entity node data, tool node data, and model node data.

[0175] In at least one embodiment of this disclosure, each of the root node data, processing entity node data, tool node data, and model node data includes a node identifier field; and the node data of the processing entity node data, tool node data, and model node data that has parent node data also includes a node identifier field of the parent node data.

[0176] In at least one embodiment of this disclosure, the runtime data includes at least one of tracking data and log data generated by the processing entity system during the execution of tasks.

[0177] In at least one embodiment of this disclosure, the apparatus 400 for evaluating a processing entity system further includes a display module configured to provide a first area in response to obtaining operational data generated by the processing entity system during the execution of a task, wherein the first area is configured to display operational data, trajectory data, and a first control; and to provide a second area in response to a selection operation on the first control, wherein the second area is configured to display one of the trajectory data and the operational data.

[0178] In at least one embodiment of this disclosure, the second region includes a first sub-region and a second sub-region, wherein the first sub-region is configured to display at least one of running data and trajectory data, and the second sub-region is configured to display a structured diagram obtained based on the data displayed in the first sub-region, wherein the structured diagram includes nodes corresponding to the data displayed in the first sub-region.

[0179] In at least one embodiment of this disclosure, the second region is further configured to display a second control, and the display module is further configured to: display running data in the first sub-region in response to the second control being in a first state; and display trajectory data in the first sub-region in response to the second control being in a second state.

[0180] In at least one embodiment of this disclosure, the display module is further configured to switch the second control between a first state and a second state in response to a selection operation on the second control.

[0181] In at least one embodiment of this disclosure, the second region is further configured to display a third control, and the display module is further configured to: provide a third region in response to a selection operation on the third control, wherein the third region is configured to display configuration rules for trajectory data, the configuration rules for trajectory data being used to extract trajectory data from running data; and receive modifications to the configuration rules for trajectory data through the third region.

[0182] In at least one embodiment of this disclosure, the evaluation module 430 is further configured to evaluate the performance of the processing entity system using trajectory data extracted from the running data according to the configuration rules of the modified trajectory data.

[0183] In at least one embodiment of this disclosure, the second region further includes a third sub-region, wherein the third sub-region is configured to display analysis content for a portion of the trajectory data, and the display module is further configured to: in response to a selection operation of at least one node in the structured diagram displayed in the second sub-region, display analysis content for the data corresponding to the node in the third sub-region.

[0184] It should be noted that, for clarity and brevity, this disclosure does not show all the constituent units of the apparatus 400 for evaluating the processing entity system. To achieve the necessary functions of the apparatus for evaluating the processing entity system, those skilled in the art can provide or configure other constituent units (not shown) according to specific needs, and this disclosure does not impose any limitations on this.

[0185] Figure 5 A schematic diagram of the structure of an electronic device (e.g., a terminal device or a server) 500 suitable for implementing embodiments of the present disclosure is shown.

[0186] refer to Figure 5 The terminal devices in this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0187] like Figure 5 As shown, electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. RAM 503 also stores various programs and data required for the operation of electronic device 500. Processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0188] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0189] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0190] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0191] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0192] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0193] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: obtain runtime data generated by the processing entity system during the execution of a task; determine an extraction method based on the development framework type of the processing entity system; extract trajectory data from the runtime data according to the extraction method, wherein the trajectory data includes at least processing entity node data, the processing entity node data recording information related to the runtime of the processing entity participating in the execution of the task; and evaluate the performance of the processing entity system using the trajectory data.

[0194] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0195] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0196] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily constitute a limitation on the unit or module itself.

[0197] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0198] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0199] One embodiment of this disclosure provides a non-transitory readable storage medium having computer instructions stored thereon that, when executed by a processor, perform one or more steps of the various methods and additional aspects described above.

[0200] For example, the non-temporarily readable storage medium may be any combination of one or more computer-readable storage media, such as a computer-readable storage medium containing program code for performing the various methods described above.

[0201] For example, when the program code is read by a computer, the computer can execute the program code stored in the computer storage medium to perform one or more steps of the various methods and additional aspects described above, such as those according to at least one embodiment of the present disclosure.

[0202] For example, the non-transitory readable storage medium may include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, and other non-transitory readable storage media or any combination thereof.

[0203] Embodiments of this disclosure also provide a computer program product. The computer program product may include a computer program or instructions. When executed by a processor, the computer program or instructions can implement the methods described above, which will not be repeated here for the sake of brevity.

[0204] According to one or more embodiments of this disclosure, Example 1 provides a method for evaluating a processing entity system, comprising:

[0205] Obtain the runtime data generated by the processing entity system during the execution of tasks;

[0206] The extraction method is determined based on the development framework type of the processing entity system. According to the extraction method, trajectory data is extracted from the operational data. The trajectory data includes at least processing entity node data, which records information related to the operational process of the processing entities participating in the task execution.

[0207] The performance of the processing entity system is evaluated using the trajectory data.

[0208] According to one or more embodiments of this disclosure, Example 2 provides a method wherein the trajectory data in Example 1 further includes at least one of the following:

[0209] Tool node data records information related to the execution process of the invoked tool;

[0210] Model node data records information related to the execution process of the called model; and

[0211] The root node data records the inputs and outputs of the processing entity system for the task.

[0212] According to one or more embodiments of this disclosure, Example 3 provides a method in which the root node data, the processing entity node data, the tool node data, and the model node data in Example 2 are organized according to a hierarchical structure.

[0213] According to one or more embodiments of this disclosure, Example 4 provides a method, wherein, in the hierarchical structure described in Example 3, the child node data corresponding to the processed entity node data includes at least one of the tool node data and the model node data; and

[0214] The child node data corresponding to the root node data includes at least one of the processing entity node data, the tool node data, and the model node data.

[0215] According to one or more embodiments of this disclosure, Example 5 provides a method wherein each of the root node data, processing entity node data, tool node data, and model node data in Example 3 includes a node identifier field; and

[0216] The node data with parent node data in the processing entity node data, the tool node data, and the model node data also includes the node identifier field of the parent node data.

[0217] According to one or more embodiments of this disclosure, Example Six provides a method in which the runtime data in Example One includes at least one of tracking data and log data generated by the processing entity system during the execution of a task.

[0218] According to one or more embodiments of this disclosure, Example 7 provides a method, wherein the method in any one of Examples 1 to 6 further includes:

[0219] In response to obtaining the runtime data generated by the processing entity system during the execution of the task, a first area is provided, wherein the first area is configured to display the runtime data, the trajectory data, and a first control; and

[0220] In response to a selection operation on the first control, a second area is provided, wherein the second area is configured to display one of the trajectory data and the running data.

[0221] According to one or more embodiments of this disclosure, Example Eight provides a method wherein the second region in Example Seven includes a first sub-region and a second sub-region, and

[0222] The first sub-region is configured to display at least one of the running data and the trajectory data, and

[0223] The second sub-region is configured to display a structured diagram obtained based on the data displayed in the first sub-region, wherein the structured diagram includes nodes corresponding to the data displayed in the first sub-region.

[0224] According to one or more embodiments of this disclosure, Example Nine provides a method wherein the second region in Example Eight is further configured to display a second control, and the method further includes:

[0225] In response to the second control being in the first state, the running data is displayed in the first sub-area; and

[0226] In response to the second control being in the second state, the trajectory data is displayed in the first sub-area.

[0227] According to one or more embodiments of this disclosure, Example 10 provides a method, wherein the method in Example 9 further includes:

[0228] In response to a selection operation on the second control, the second control is switched between the first state and the second state.

[0229] According to one or more embodiments of this disclosure, Example 11 provides a method wherein the second area in Example 7 is further configured to display a third control, and the method further includes:

[0230] In response to a selection operation on the third control, a third region is provided, wherein the third region is configured to display configuration rules for the trajectory data, the configuration rules for extracting the trajectory data from the running data; and

[0231] Modifications to the configuration rules for the trajectory data are received through the third region.

[0232] According to one or more embodiments of this disclosure, Example Twelve provides a method wherein the evaluation of the performance of the processing entity system using the trajectory data in Example Eleven includes:

[0233] The performance of the processing entity system is evaluated using trajectory data extracted from the running data according to the modified configuration rules of the trajectory data.

[0234] According to one or more embodiments of this disclosure, Example Thirteen provides a method in which the second region in Example Eight further includes a third sub-region, wherein the third sub-region is configured to display analysis content for a portion of trajectory data, and the method further includes:

[0235] In response to a selection operation on at least one node in the structured diagram displayed in the second sub-region, analysis content of the data corresponding to the node is displayed in the third sub-region.

[0236] According to one or more embodiments of the present disclosure, Example Fourteen provides a non-transitory computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of Examples One to Thirteen.

[0237] According to one or more embodiments of this disclosure, Example Fourteen provides an electronic device, including:

[0238] One or more processors; and

[0239] One or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the method as described in any one of Examples 1 to 13.

[0240] According to one or more embodiments of this disclosure, Example Fifteen provides an apparatus for evaluating a processing entity system, comprising:

[0241] The acquisition module is configured to acquire runtime data generated by the processing entity system during task execution.

[0242] An extraction module is configured to determine an extraction method based on the development framework type of the processing entity system, and extract trajectory data from the runtime data according to the extraction method. The trajectory data includes at least processing entity node data, which records information related to the runtime process of the processing entities participating in the task execution.

[0243] The evaluation module is configured to evaluate the performance of the processing entity using the trajectory data.

[0244] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0245] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0246] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.< / modelstepid> < / string> < / toolstepid> < / string> < / step> < / agentstep>

Claims

1. A method for evaluating a processing entity system, comprising: Obtain the runtime data generated by the processing entity system during the execution of tasks; The extraction method is determined according to the development framework type of the processing entity system. Trajectory data is extracted from the running data according to the extraction method. The trajectory data includes at least processing entity node data, and the processing entity node data records information related to the running process of the processing entity participating in the execution of the task. as well as The performance of the processing entity system is evaluated using the trajectory data.

2. The method according to claim 1, wherein, The trajectory data also includes at least one of the following: Tool node data records information related to the execution process of the invoked tool; Model node data records information related to the execution process of the called model; as well as The root node data records the inputs and outputs of the processing entity system for the task.

3. The method according to claim 2, wherein, The root node data, the processing entity node data, the tool node data, and the model node data are organized according to a hierarchical structure.

4. The method according to claim 3, wherein, In the hierarchical structure, the child node data corresponding to the processed entity node data includes at least one of the tool node data and the model node data; and The child node data corresponding to the root node data includes at least one of the processing entity node data, the tool node data, and the model node data.

5. The method according to claim 3, wherein, Each of the root node data, processing entity node data, tool node data, and model node data includes a node identifier field; and The node data with parent node data in the processing entity node data, the tool node data, and the model node data also includes the node identifier field of the parent node data.

6. The method according to claim 1, wherein, The operational data includes at least one of the tracking data and log data generated by the processing entity system during the execution of tasks.

7. The method according to any one of claims 1-6, further comprising: In response to obtaining the runtime data generated by the processing entity system during the execution of the task, a first area is provided, wherein the first area is configured to display the runtime data, the trajectory data, and a first control; and In response to a selection operation on the first control, a second area is provided, wherein the second area is configured to display one of the trajectory data and the running data.

8. The method according to claim 7, wherein, The second region includes a first sub-region and a second sub-region. The first sub-region is configured to display at least one of the running data and the trajectory data. The second sub-region is configured to display a structured diagram obtained based on the data displayed in the first sub-region, wherein the structured diagram includes nodes corresponding to the data displayed in the first sub-region.

9. The method according to claim 8, wherein, The second area is also configured to display a second control, and the method further includes: In response to the second control being in the first state, the running data is displayed in the first sub-area; and In response to the second control being in the second state, the trajectory data is displayed in the first sub-area.

10. The method of claim 9, further comprising: In response to a selection operation on the second control, the second control is switched between the first state and the second state.

11. The method according to claim 7, wherein, The second area is also configured to display a third control, and the method further includes: In response to a selection operation on the third control, a third region is provided, wherein the third region is configured to display configuration rules for the trajectory data, the configuration rules for extracting the trajectory data from the running data; and Modifications to the configuration rules for the trajectory data are received through the third region.

12. The method according to claim 11, wherein, The evaluation of the performance of the processing entity system using the trajectory data includes: The performance of the processing entity system is evaluated using trajectory data extracted from the running data according to the modified configuration rules of the trajectory data.

13. The method according to claim 8, wherein, The second region further includes a third sub-region, wherein the third sub-region is configured to display analysis content for a portion of the trajectory data, and the method further includes: In response to a selection operation on at least one node in the structured diagram displayed in the second sub-region, analysis content of the data corresponding to the node is displayed in the third sub-region.

14. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-13.

15. An electronic device comprising: One or more processors; as well as One or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the method as described in any one of claims 1-13.

16. An apparatus for evaluating a processing entity system, comprising: The acquisition module is configured to acquire runtime data generated by the processing entity system during task execution. The extraction module is configured to determine the extraction method based on the development framework type of the processing entity system, and extract trajectory data from the running data according to the extraction method. The trajectory data includes at least processing entity node data, which records information related to the running process of the processing entity participating in the task execution. as well as The evaluation module is configured to evaluate the performance of the processing entity using the trajectory data.

Images