Information processing device, information processing method, and information processing program

Abstract

To support auditing operations for multi-agent systems using graph structures. [Solution] The information processing device according to the present invention comprises a distribution unit, an aggregation unit, and a synchronization processing unit. The distribution unit distributes event data relating to events that occur in conjunction with the execution of a workflow. The aggregation unit aggregates the event data based on identification information that identifies the request. The synchronization processing unit transmits control data to a terminal device for synchronized display of information relating to the response to the request, information relating to the progress of the workflow, and information relating to the basis of the response, based on the aggregated event data.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] In recent years, multi-agent systems that operate by combining a plurality of models have been widely used to improve the performance of machine learning models. For example, Patent Document 1 proposes a method and system for generating a unified knowledge graph from multimodal data sources using a collaborative multi-agent LLM.

[0003] Further, in relation to multi-agent systems, a design method is known for constructing workflows of complex AI agents and Large Language Models (LLM) by modeling each program as a graph structure including "nodes" that are specific processing steps and "edges" that indicate transitions between nodes. According to this design method, it is expected to reduce the burden required for implementing a control framework including "loops (cycles)" and "complex conditional branches".

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0005] However, there is room for improvement in supporting the audit work of the above-described conventional multi-agent system using a graph structure.

[0006] This application was made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that can support audit work for a multi-agent system using a graph structure. [Means for solving the problem]

[0007] The information processing device according to the present invention is an information processing device that processes requests by a workflow including multiple nodes using a large-scale language model, and comprises a distribution unit, an aggregation unit, and a synchronization processing unit. The distribution unit distributes event data relating to events that occur in conjunction with the execution of the workflow. The aggregation unit aggregates the event data based on identification information for identifying requests. The synchronization processing unit transmits control data to a terminal device for synchronously displaying information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response, based on the event data aggregated by the aggregation unit. [Effects of the Invention]

[0008] According to one embodiment, auditing operations for a multi-agent system using a graph structure can be supported. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a diagram illustrating the overview of the information processing according to the embodiment. [Figure 2] Figure 2 shows an example of a user interface display related to information processing according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of an information processing system according to the embodiment. [Figure 4] Figure 4 shows an example of the configuration of an information processing device according to the present invention. [Figure 5] Figure 5 is a diagram showing an overview of the event information stored in the event information storage unit according to the embodiment. [Figure 6] Figure 6 is a diagram showing an overview of the view information stored in the view information storage unit according to the embodiment. [Figure 7] Figure 7 is a diagram showing an overview of the correspondence information stored in the correspondence information storage unit according to the embodiment. [Figure 8] Figure 8 is a flowchart showing an example of the information processing procedure according to the embodiment. [Figure 9] Figure 9 shows a specific example of information processing by the information processing device according to the embodiment. [Figure 10] Figure 10 is a hardware configuration diagram showing an example of a computer that implements information processing according to an embodiment or modified example. [Modes for carrying out the invention]

[0010] The following describes in detail, with reference to the drawings, the embodiments for implementing the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing apparatus, information processing method, and information processing program. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted.

[0011] [Embodiment] [1. Overview of Information Processing Based on the Embodiment] The following describes the overview of the information processing performed by the information processing device 100 according to the embodiment, using Figure 1. Figure 1 is a diagram illustrating the overview of the information processing according to the embodiment. The information processing according to the embodiment is based on a multi-agent system that processes user requests using a workflow defined by a graph structure that includes multiple nodes, which are processing units using a large-scale language model, and edges that indicate transitions between nodes.

[0012] The information processing according to the embodiment is implemented by an information processing system SYS including a terminal device 10 and an information processing apparatus 100. The terminal device 10 and the information processing apparatus 100 are each connected to a network N (for example, see FIG. 3) by wire or wirelessly. The terminal device 10 and the information processing apparatus 100 can communicate with other devices via the network N.

[0013] The terminal device 10 is an information processing terminal used by a user U such as a system administrator who manages the information processing apparatus 100. The terminal device 10 corresponds to a tablet PC (Personal Computer), a notebook or desktop PC, a smartphone, or other wearable devices, but is not particularly limited to such examples.

[0014] The user U can operate a web browser to access, for example, a platform for system auditing, thereby performing an audit of the multi-agent system using a graph structure as an auditor.

[0015] The information processing apparatus 100 includes a multi-agent system that processes external requests according to a workflow defined by a graph structure including a plurality of nodes that are processing units using a large language model and edges indicating transitions between the nodes. The external requests are not limited to, for example, user requests from users who use the multi-agent system, and may include calls from external APIs (Application Programming Interfaces) of cooperation destinations. The information processing apparatus 100 is typically a server apparatus, but may be implemented by a mainframe, a workstation, or the like. When the information processing apparatus 100 is implemented by a server apparatus, it may be implemented by a single server apparatus, or may be implemented by a cloud system or the like in which a plurality of server apparatuses and a plurality of storage apparatuses operate in cooperation.

[0016] Further, the information processing apparatus 100 executes, as the information processing according to the embodiment, information processing for supporting audit work of a multi-agent system using a graph structure through a platform for system audit.

[0017] For example, in a conventional multi-agent system, respective data such as events indicating the progress status of each node (process), generated answers, and reference sources serving as the basis for the answers are dispersedly output as asynchronous or independent separate log data (or data streams). Therefore, it is not possible to accurately associate the temporal or contextual dependencies between the respective data, and there is a technical problem that it is technically difficult to centrally track and identify the transition process of a dynamically changing workflow and the output basis of an answer.

[0018] Accordingly, the information processing apparatus 100 according to the embodiment executes information processing capable of supporting audit work by simultaneously displaying, in a streaming manner, an answer to a request, the progress status of a workflow, and the basis of the answer.

[0019] For example, as illustrated in FIG. 1, the terminal device 10 transmits a search query for specifying an event to be monitored input by the user U to the information processing apparatus 100 in accordance with an operation input of the user U (step S01). Note that the search query may include, for example, a trace ID that is identification information for uniquely specifying a user request (such as a question) to the multi-agent system, a user ID that is identification information for specifying the user U, text information indicating the content of the question, and the like.

[0020] Upon receiving the search query from the terminal device 10, the information processing apparatus 100 specifies a workflow corresponding to the search query, and executes event distribution processing for distributing event data related to an event that occurs along with execution of the specified workflow (step S02). The event data includes "node start", "node end", "tool call", or "LLM response".

[0021] Specifically, the information processing device 100 detects event data output asynchronously from the multi-agent system in real time. The information processing device 100 then serializes the detected event data as streaming data while maintaining the order in which it was received (generated), and autonomously distributes it to the event aggregation process via streaming.

[0022] Next, the information processing device 100 aggregates the distributed event data based on a trace ID used to identify identical requests. For example, the information processing device 100 identifies event data related to identical requests based on the trace ID, and performs event aggregation processing to convert the identified event data into a predetermined data format and arrange it chronologically (step S03).

[0023] First, the information processing device 100 temporarily stores the event data, which is continuously sent in streaming format, in an internal buffer (memory area). Then, the information processing device 100 reads each event data from the internal buffer and first performs a "normalization schema conversion process" on each event data.

[0024] Next, the information processing device 100 analyzes the unique data format for each event data source (for example, different types of external tools or different LLM providers) and converts it into a unique normalization schema (common format) that has been predefined as a predetermined data format within the system. In this conversion to the normalization schema, the information processing device 100 also preserves the trace ID included in the event data header in each of the converted event data.

[0025] Following normalization, the information processing device 100 performs a "time-series sorting process" using the trace ID for each event data that has been converted into the normalized schema. Specifically, the information processing device 100 extracts and groups only the event data that have the same trace ID from the event data group after conversion to the normalized schema. Then, the information processing device 100 refers to the timestamp (occurrence time information, or event sequence number) contained in each event data that has the same trace ID and sorts them so that they are in the correct occurrence order (time-series) within the workflow execution session. The information processing device 100 then passes the converted and sorted event data to the synchronization process or the detection process.

[0026] Next, the information processing device 100 performs a detection process based on the event data after transformation and sorting processing (hereinafter referred to as the "normalized event sequence") aggregated by the event aggregation process, to detect the occurrence of a predetermined abnormality, including processing delays or stagnation, at each node included in the execution session, for each execution session identified by the trace ID (step S04).

[0027] For example, the information processing device 100 monitors the normalized event sequence output sequentially from the event aggregation process and detects a predetermined anomaly for each execution session running in parallel for individual requests, according to the processing status of each node included in the execution session.

[0028] Specifically, the information processing device 100 determines that a process has "timed out (stagnated, hung up)" if it exceeds the maximum allowable processing time pre-set for each node executed in each execution session, or the allowable waiting time for an external API (Application Programming Interface) call.

[0029] Furthermore, for each execution session, after receiving an event (node ​​start) indicating that processing has started at a specific node, the information processing device 100 determines that the node is in an "unresponsive state (not updated)" if an event (node ​​end) indicating that the node has finished, or if intermediate output events (for example, text data fragments during streaming, interruption of image or audio streaming, etc.) are interrupted for a predetermined period (for example, several seconds).

[0030] Furthermore, the information processing device 100 may constantly monitor the interval for receiving event data executed in each execution session, and may determine that the node in question in each execution session has frozen (abnormally stopped) if the interval for receiving events exceeds a predetermined first threshold (timeout period), or if the execution time of the same node exceeds a predetermined second threshold (long period of no updates).

[0031] If the information processing device 100 determines that an abnormality such as "unresponsive state" or "stagnation (hang-up)" has occurred in a node within any execution session, it dynamically generates an "alert event" that includes a trace ID to identify the relevant execution session. The information processing device 100 then passes the generated "alert event" to the subsequent synchronization process.

[0032] Next, the information processing device 100, based on the normalized event sequence obtained by the event aggregation process, sends control data to the terminal device 10, which is the source of the request, to display information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response in a synchronous manner according to the progress of the workflow. This enables the terminal device 10 to perform a synchronization process to control the synchronous display of each piece of information (step S05).

[0033] Specifically, the synchronization processing unit 135 analyzes the event type and payload data included in the normalized event column, and based on the analysis results, generates various view data for each trace ID, corresponding to the three display areas: the "Output tab," the "Flow tab," and the "Metadata tab" located on the user interface displayed on the terminal device 10.

[0034] For example, the information processing device 100 extracts LLM response data, which is the final answer (message) to user U, from the normalized event sequence, generates view data for displaying the answer in the user interface, and saves it as view data for the output tab, associated with a timestamp.

[0035] Furthermore, for example, the information processing device 100 extracts data such as identification information of nodes currently running or completed, and processing transition states from the normalized event column, generates view data for visualizing the progress of the workflow in the user interface, and saves it as view data for the flow tab, associated with a timestamp.

[0036] Furthermore, for example, the information processing device 100 extracts data such as tool node call details (input queries and execution parameters) and referenced document data (source information) from the normalized event column, generates view data for displaying metadata in the user interface, and saves it as view data for the metadata tab, associated with a timestamp.

[0037] The information processing device 100 then streams the view data (an example of "control data") corresponding to the "output tab," "flow tab," and "metadata tab" to the terminal device 10 in real time, either in batches or in parallel, as synchronous distribution data via the same communication session corresponding to the trace ID. As a result, each event that occurs and is organized asynchronously in conjunction with the execution of the workflow is simultaneously and synchronously received by the terminal device 10 as "screen display update data" belonging to a single, identical execution session (trace ID), and the display areas (views) of the three corresponding tabs are updated simultaneously in real time (synchronous distribution) without delay.

[0038] Furthermore, when the information processing device 100 receives an alert event from the detection process, it generates display data such as a pop-up that notifies of processing delays or errors, and dynamically inserts the generated display data into the communication session corresponding to the trace ID included in the alert event, thereby controlling real-time synchronous distribution.

[0039] Furthermore, the information processing device 100 performs a synchronization process to synchronize the display changes on the terminal device 10 by sending control data to the terminal device 10 to control the display changes to the corresponding object (an example of a "specific data location") in response to the operation of the user U on the terminal device 10 on the citation tab (an example of a "citation element") (step S06).

[0040] Specifically, the information processing device 100 analyzes the LLM response (answer text) contained in the view data for the output tab and detects "quote tags (for example, identification symbols such as [1] and [2])" embedded in the text. At the same time, the information processing device 100 identifies the "corresponding object (for example, a specific reference document or search result data contained in the view data for the metadata tab)" that is the source of the quote tag. The basis synchronization unit 136 then manages the correspondence between the identified quote tag and the corresponding object as correspondence relationship information by saving it in association with a trace ID.

[0041] Furthermore, when the information processing device 100 receives an operation event from the terminal device 10 indicating, for example, that a specific "quote tag" on the output tab has been selected by user U (e.g., clicked or moused over), it refers to the correspondence information and identifies the corresponding "corresponding object". The information processing device 100 then generates "highlight state update data" (an example of "control data") to highlight the area in the metadata tab where the "corresponding object" is displayed (e.g., scroll or change the background color), and transmits it to the terminal device 10 via the communication unit 110 as synchronous distribution data for the same communication session (step S07).

[0042] As described above, the information processing device 100 according to the embodiment displays the response to a request, the progress status of the workflow, and the basis for the response in a synchronized manner. Therefore, the information processing device 100 according to the embodiment can support audit operations for multi-agent systems using a graph structure.

[0043] Figure 2 is a diagram showing an example of a user interface display for information processing according to an embodiment. Screen W of the audit-enabled streaming UI shown in Figure 2 is an example of a user interface displayed on the terminal device 10. Screen W shown in Figure 2 displays streaming data processed by the information processing device 100 in real time, synchronized for each trace ID.

[0044] Furthermore, at the top of screen W, in addition to the trace ID (trace_id) used to identify the audit target, the current status of the process (e.g., "running"), the elapsed time, and alert information indicating any anomalies detected by the information processing device 100 are displayed. In the example in Figure 2, a response delay was detected at the third step node (3): "retrieve_policy") of the processing flow in the execution session corresponding to the trace ID (trace_id): "helpdesk-000123," and the "delay detected" alert and risk level ("medium") are highlighted.

[0045] Furthermore, the output tab TB-1, located on the right side of the center of screen W, streams information about the question text entered by the user of the multi-agent system and the answer from the multi-agent system to that question. If a specific source document is cited during the answer generation process, citation tags (for example, [FAQ-1021], [POLICY-88]) are inserted into the text in real time.

[0046] Furthermore, the flow tab TB-2, located in the center of screen W, displays the execution status of multiple nodes that make up the graph structure (in Figure 2, the workflow from node (1): "classify" corresponding to the first step node to node (6): "final_answer" corresponding to the sixth step node) in order from top to bottom. Each node is displayed with a different identifier (check mark, loading icon, warning icon, etc.) depending on its status, such as execution complete, streaming, waiting, or not started.

[0047] Additionally, the metadata tab TB-3, located on the right side of the center of screen W, displays detailed information about the supporting data corresponding to the citation tags inserted in the response (document name, source node, confidence level, usage location, etc.). It also displays metadata such as the name of the agent executed, the model used, and the status of saving to the audit log.

[0048] The information processing device 100 synchronizes and displays event information corresponding to the trace ID (trace_id): "helpdesk-000123" on screen W shown in Figure 2. That is, the information in the output tab TB-1, the flow tab TB-2, and the metadata tab TB-3 is displayed synchronized in an interconnected manner. Specifically, as shown in Figure 2, the citation tag: [FAQ-1021] included in the answer in the output tab TB-1, the node (2): [search_faq] above which the basis indicated by the citation tag: [FAQ-1021] was obtained in the processing flow of the flow tab TB-2, and the detailed card of the basis log / meta information [FAQ-1021] in the metadata tab TB-3 are displayed interconnected by visual connectors (e.g., linked lines and highlights).

[0049] Similarly, the warning display for node (3):[retrieve_policy] where the information processing device 100 has detected an "acquisition delay" is immediately synchronized and propagated to the corresponding quoted tag:[POLICY-88] being generated in the output tab TB-1 and to the display of [POLICY-88 (delayed)] in the metadata tab TB-3. This allows user U to instantly track (trace) which part of the multi-agent system's output is experiencing delays or anomalies.

[0050] Furthermore, the event timeline display area TL located at the bottom of screen W displays events such as the start and end of each node, data acquisition, and delay detection, which have been normalized and sorted chronologically by the information processing device 100, along with timestamps, arranged horizontally from left to right on screen W. This allows user U to accurately understand the context of each process and the location of bottlenecks along the time axis.

[0051] Furthermore, when user U presses buttons such as "Play," "Export," or "Audit Memo" located at the top of screen W, a request for subsequent processing is sent from terminal device 10 to information processing device 100. The "lines (connectors)" connecting the quote tags and each view may be displayed at all times, or they may be displayed in conjunction with operations such as mouse hover.

[0052] [2. System Configuration] The configuration of the information processing system SYS that performs the information processing method according to this application will be described below. Figure 3 is a diagram showing an example of the configuration of the information processing system SYS according to the embodiment.

[0053] As shown in Figure 3, the information processing system SYS according to this embodiment is configured to include a terminal device 10 and an information processing device 100. Note that the configuration of the information processing system SYS shown in Figure 3 is just one example, and other devices not illustrated in Figure 3 may be included.

[0054] The terminal device 10 and the information processing device 100 are connected to the network N by wire or wireless connection. The terminal device 10 and the information processing device 100 can communicate with other devices via the network N.

[0055] Network N includes, for example, WANs (Wide Area Networks) such as the Internet, and mobile communication networks such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation: 5th Generation Mobile Communication System).

[0056] Furthermore, terminal device 10 is an information processing terminal used by user U, such as a system administrator who manages the information processing device 100. Terminal device 10 may include a tablet PC (Personal Computer), a notebook or desktop PC, a smartphone, or other wearable device, but is not limited to these examples.

[0057] The terminal device 10 has a control function unit that controls each functional unit provided in the terminal device 10 to realize input / output, display, or communication. The terminal device 10 connects to the network N via a mobile communication network or short-range wireless communication such as Bluetooth® or wireless LAN (Local Area Network), and can communicate with other devices such as the information processing device 100 through the network N.

[0058] User U can act as an auditor, performing audits of multi-agent systems using graph structures, for example, by operating a web browser and accessing a platform for system auditing.

[0059] Furthermore, when the terminal device 10 receives control information from the information processing device 100 to perform predetermined information processing, it performs the information processing according to the control information. Here, the control information is written in, for example, a scripting language such as JavaScript (registered trademark), a stylesheet language such as CSS (Cascading Style Sheets), a programming language such as Java (registered trademark), or a markup language such as HTML (HyperText Markup Language). Alternatively, the predetermined application itself distributed from the information processing device 100 may be considered as the control information.

[0060] The information processing device 100 has a multi-agent system that processes requests from external sources through a workflow defined by a graph structure that includes multiple nodes, which are processing units using a large-scale language model, and edges that indicate transitions between nodes. The information processing device 100 is typically a server device, but may also be implemented as a mainframe or workstation. Furthermore, when the information processing device 100 is implemented as a server device, it may be implemented as a single server device, or as a cloud system in which multiple server devices and multiple storage devices work together.

[0061] Furthermore, as information processing related to the embodiment, the information processing device 100 performs information processing to support audit operations of a multi-agent system using a graph structure through a system audit platform.

[0062] [3. Equipment configuration] Hereinafter, an example of the functional configuration of the information processing device 100 according to the embodiment will be described using Figure 4. Figure 4 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Figure 4, the information processing device 100 according to the embodiment has a communication unit 110, a storage unit 120, and a control unit 130.

[0063] (Communications Department 110) The communication unit 110 is implemented, for example, by a communication module or a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection and transmits and receives information with other devices such as the terminal device 10.

[0064] (Storage unit 120) The storage unit 120 stores, for example, programs and data used for control and calculations by the control unit 130. The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) or flash memory, or storage devices such as hard disks or optical discs. As shown in Figure 4, the storage unit 120 has an event information storage unit 121, a view information storage unit 122, and a correspondence relationship information storage unit 123.

[0065] (Event information storage unit 121) The event information storage unit 121 stores event information relating to events that occur in the multi-agent system. Figure 5 is a diagram showing an overview of the event information stored in the event information storage unit 121 according to this embodiment.

[0066] As shown in Figure 5, the event information stored in the event information storage unit 121 has a "trace ID" item, a "timestamp" item, and an "event information" item. These items in the event information are interconnected.

[0067] The "Trace ID" field stores the trace ID, which is identification information used to uniquely identify user requests (such as questions) to the multi-agent system. The "Timestamp" field stores timestamp information indicating when each event occurred in the execution session corresponding to the relevant trace ID. The "Event Information" field stores information about each event in the execution session corresponding to the relevant trace ID (for example, "Node Start," "Node End," "Tool Invocation," or "LLM Response").

[0068] (View information storage unit 122) The view information storage unit 122 stores view information to be displayed synchronously in three display areas located on the user interface: the "Output Tab," the "Flow Tab," and the "Meta Information Tab." Figure 6 is a diagram showing an overview of the view information stored in the view information storage unit 122 according to this embodiment.

[0069] As shown in Figure 6, the view information stored in the view information storage unit 122 includes the following items: "Trace ID", "Timestamp", "View data for output tab", "View data for flow tab", and "View data for meta information tab". These items in the view information are interconnected.

[0070] The "Trace ID" field stores the trace ID, which is identification information used to uniquely identify the corresponding user request (such as a question). The "Timestamp" field stores timestamp information indicating when the corresponding event occurred. The "Output Tab View Data" field stores display information for the output tab, which is placed in the user interface displayed on the terminal device 10. The "Flow Tab View Data" field stores display information for the flow tab, which is placed in the user interface displayed on the terminal device 10. The "Meta Information Tab View Data" field stores display information for the meta information tab, which is placed in the user interface displayed on the terminal device 10.

[0071] (Relationship information storage unit 123) The correspondence information storage unit 123 stores correspondence information that shows the correspondence between citation tags (an example of "citation elements") displayed along with the information about the answer and corresponding objects (an example of "specific data locations") included in the information about the evidence. Figure 7 is a diagram showing an overview of the correspondence information stored in the correspondence information storage unit 123 according to this embodiment.

[0072] As shown in Figure 7, the correspondence information stored in the correspondence information storage unit 123 includes the items "Trace ID," "Citation Tag," and "Corresponding Object." These items in the correspondence information are mutually associated.

[0073] The "Trace ID" field stores the trace ID, which is identification information used to uniquely identify the corresponding user request (such as a question). The "Quote Tag" field stores information about identification symbols embedded in the answer text to identify information about the basis of the answer. The "Corresponding Object" field stores information indicating the reference documents, search result data, etc., that serve as the basis for the answer.

[0074] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs (an example of an "information processing program" according to this embodiment) stored in the internal memory of the information processing device 100 using RAM as the working area.

[0075] Furthermore, the control unit 130 can be implemented using an integrated circuit such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a GPGPU (General Purpose Graphic Processing Unit).

[0076] As shown in Figure 4, the control unit 130 includes a graph processing unit 131, a distribution unit 132, an aggregation unit 133, a detection unit 134, a synchronization processing unit 135, and a basis synchronization unit 136. These units realize or execute the information processing functions and operations described below. Note that the control unit 130 may also be equipped with new functional units different from those shown in Figure 7, for example, in response to the expansion of various processes performed by the information processing device 100.

[0077] (Graph processing unit 131) When the graph processing unit 131 receives an external request, it interprets the intent of the request and dynamically executes a workflow that has been predefined in the graph structure.

[0078] When the graph processing unit 131 receives a user request from the terminal device 10 via the communication unit 110, it issues a "trace ID (trace_id)," which is an identifier to uniquely identify the execution session, and establishes a real-time communication channel (such as "WebSocket") for exchanging data with the terminal device 10 during workflow execution. The graph processing unit 131 also embeds the issued trace ID into the header (header area) of event data related to various events that occur each time the workflow is executed (such as "node start / end," "tool call," and "LLM response"), and stores it in the event information storage unit 121.

[0079] (Distribution Section 132) The distribution unit 132 executes the process of distributing event data received from the graph processing unit 131. Specifically, the distribution unit 132 detects event data output asynchronously from the graph processing unit 131 in real time. The distribution unit 132 then serializes the detected event data as streaming data while maintaining the order in which it was received (generated), and autonomously distributes it in streaming format to the downstream component, the aggregation unit 133.

[0080] (Aggregation section 133) The aggregation unit 133 aggregates event data distributed by the distribution unit 132 based on the trace ID, which is identification information that identifies the request. Specifically, the aggregation unit 133 identifies event data related to the same request based on the trace ID, and aggregates the identified event data by converting it to a predetermined data format and arranging it chronologically. The aggregation unit 133 then passes the aggregated, converted, and arranged event data ("normalized event sequence") to the subsequent detection unit 134 and synchronization processing unit 135.

[0081] Specifically, the aggregation unit 133 temporarily stores the event data continuously sent from the distribution unit 132 in streaming format in an internal buffer (memory area). Then, the aggregation unit 133 reads each event data from the internal buffer and first performs a "normalization schema conversion process" on each event data.

[0082] Specifically, the aggregation unit 133 analyzes the unique data format for each event data source (for example, different types of external tools or different LLM providers) and converts it into a unique normalization schema (common format) that is predefined as a predetermined data format within the system. During this conversion to the normalization schema, the aggregation unit 133 carries over and retains the trace ID included in the event data header to each of the converted event data.

[0083] Next, the aggregation unit 133 performs a "time-series sorting process" using the trace ID for each event data that has been converted to the normalized schema. Specifically, the aggregation unit 133 extracts and groups only the event data that have the same trace ID from the event data group after conversion to the normalized schema. Then, the aggregation unit 133 refers to the timestamp (occurrence time information, or event sequence number) contained in each event data that has the same trace ID and sorts them to the correct occurrence order (time-series) within the workflow execution session.

[0084] (Detection unit 134) Based on the event data after conversion and sorting processing aggregated by the aggregation unit 133, the detection unit 134 detects the occurrence of predetermined abnormalities, including processing delays or stagnation, in each node included in each execution session identified by the trace ID. Specifically, the detection unit 134 monitors the event data output sequentially from the aggregation unit 133 and detects predetermined abnormalities according to the processing status of each node included in each execution session executed in parallel for individual requests.

[0085] Specifically, the detection unit 134 determines that a process has "timed out (stagnated, hung up)" if it exceeds the maximum allowable processing time or the allowable waiting time for an external API (Application Programming Interface) call, which are pre-set for each node executed in each execution session. In addition, for each execution session, after receiving an event (start event) indicating that processing has started at a specific node, the detection unit 134 determines that the node is in an "unresponsive state (not updated)" if the node's end event or intermediate output event (for example, a text data fragment during streaming (including interruptions in image or audio streaming)) is interrupted for a predetermined period (for example, several seconds).

[0086] Alternatively, the detection unit 134 constantly monitors the interval between receiving event data executed in each execution session. The detection unit 134 may then determine that a node in a given execution session has frozen (abnormally stopped) if the interval between receiving events exceeds a predetermined first threshold (timeout period), or if the execution time of the same node exceeds a predetermined second threshold (long period of no updates).

[0087] If the detection unit 134 determines that an abnormality such as "unresponsive state" or "stagnation (hang-up)" has occurred in any node within an execution session, it dynamically generates an "alert event" that includes a trace ID to identify the relevant execution session. The detection unit 134 then passes the generated "alert event" to the subsequent synchronization processing unit 135.

[0088] (Synchronization processing unit 135) The synchronization processing unit 135 transmits control data to the terminal device 10 for synchronously displaying information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response, based on the event data aggregated by the aggregation unit 133. In other words, the synchronization processing unit 135 transmits control data to the terminal device 10, which is the source of the request, for synchronously displaying each piece of information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response, based on the event data after conversion and sorting processing aggregated by the aggregation unit 133, in accordance with the progress of the workflow, thereby executing processing to control the synchronous display of each piece of information in the terminal device 10.

[0089] Specifically, the synchronization processing unit 135 analyzes the event type and payload data contained in the event data after conversion and sorting processing, and based on the analysis results, generates various view data corresponding to the three display areas (views) that constitute the user interface displayed on the terminal device 10: the "output tab," the "flow tab," and the "meta information tab," for each trace ID.

[0090] For example, the synchronization processing unit 135 extracts LLM response data, which is the final answer (message) to user U, from the event data after conversion and sorting processing, generates view data for displaying the answer in the user interface, and stores the generated view data in the view information storage unit 122, as view data for the output tab, which is a display area for displaying the answer in the user interface, associated with a timestamp.

[0091] Furthermore, for example, the synchronization processing unit 135 extracts data such as identification information of nodes currently running or completed, and processing transition status from the event data after the conversion and sorting process, generates view data for visualizing the progress of the workflow in the user interface, and stores the generated view data in the view information storage unit 122, as view data for the flow tab, which is a display area for displaying the view data in the user interface, associated with a timestamp.

[0092] Furthermore, for example, the synchronization processing unit 135 extracts data such as tool node call details (input queries and execution parameters) and referenced document data (source information) from the event data after the conversion and sorting process, generates view data for displaying metadata in the user interface, and stores the generated view data in the view information storage unit 122, as view data for the metadata tab, which is a display area for displaying metadata in the user interface, as it is associated with a timestamp.

[0093] The synchronization processing unit 135 then streams the view data (an example of "control data") corresponding to the "output tab," "flow tab," and "metadata tab" to the terminal device 10 in real time, either in batches or in parallel, via the communication unit 110 through the same communication session (communication path) corresponding to the trace ID. As a result, each event that is asynchronously generated and organized in conjunction with the execution of the workflow is simultaneously and synchronously received by the terminal device 10 as "screen display update data" belonging to a single, identical execution session (trace ID), and the display areas (views) of the three corresponding tabs are updated simultaneously in real time (synchronously distributed) without delay.

[0094] Furthermore, when the detection unit 134 detects the occurrence of a predetermined abnormality, the synchronization processing unit 135 generates display data to display the detected abnormality on the user interface displayed on the terminal device 10, and transmits it to the terminal device 10 via the same communication channel identified by the identification information corresponding to the same request, through the communication unit 110. For example, when the synchronization processing unit 135 receives an alert event from the detection unit 134, it generates display data such as a pop-up notifying of processing delays or errors, dynamically inserts the generated display data into the communication session corresponding to the trace ID included in the alert event, and controls the real-time synchronous distribution through the communication unit 110.

[0095] Furthermore, the synchronization processing unit 135 may distribute control information to change the display mode of each tab according to the content of the alert event. For example, the synchronization processing unit 135 distributes control information to highlight the border of the relevant node in red or to display a warning icon in the flow tab. Also, the synchronization processing unit 135 distributes control information to prioritize the display of the last recorded stack trace or API status code in the metadata tab. In this way, it helps user U to immediately determine "why it is stopped (e.g., an API failure or an infinite loop in the logic)."

[0096] (Base Synchronization Section 136) The evidence synchronization unit 136 manages the correspondence between citation tags (an example of a "citation element") displayed along with the answer information and corresponding objects (an example of a "specific data location") included in the evidence information. Furthermore, based on the correspondence, the evidence synchronization unit 136 synchronizes the display changes on the terminal device 10 by transmitting control data to the terminal device 10 to control the display changes to specific data locations in response to operations on the citation elements by the user U on the terminal device 10.

[0097] Specifically, the evidence synchronization unit 136 analyzes the LLM response (answer text) contained in the view data for the output tab and detects "quote tags (for example, identification symbols such as [1] and [2])" embedded in the text. At the same time, the evidence synchronization unit 136 identifies the "corresponding object (for example, a specific reference document or search result data contained in the view data for the metadata tab)" from which the quote tags originated. The evidence synchronization unit 136 then manages the correspondence between the identified quote tags and the corresponding object by associating it with a trace ID and saving it as correspondence relationship information.

[0098] Furthermore, when the base synchronization unit 136 receives an operation event from the communication unit 110 indicating that, for example, user U has selected a specific "quote tag" on the output tab in the terminal device 10 (for example, by clicking or mouseover), it refers to the correspondence information and identifies the corresponding "corresponding object". The base synchronization unit 136 then generates "highlight state update data" (an example of "control data") to highlight the area in the metadata tab where the "corresponding object" is displayed (for example, by scrolling or changing the background color), and transmits it to the terminal device 10 via the communication unit 110 as data for the same communication session.

[0099] [4. Example of processing procedure] Hereinafter, an example of the processing procedure for information processing performed by the information processing device 100 according to the embodiment will be described using Figure 8. Figure 8 is a flowchart of an example of the processing procedure for information processing according to the embodiment. Note that the processing procedure shown in Figure 8 is executed by the control unit 130 of the information processing device 100.

[0100] The distribution unit 132 distributes event data related to events that occur in conjunction with the execution of the workflow in streaming format (step S101).

[0101] Furthermore, the aggregation unit 133 converts the event data distributed by the distribution unit 132 into a predetermined data format and arranges it in chronological order (step S102).

[0102] Furthermore, the detection unit 134 detects the occurrence of a predetermined abnormality, including processing delays or stagnation, at each node included in the execution session, for each execution session identified by the identification information, based on the event data after conversion and sorting processing output from the aggregation unit 133 (step S103).

[0103] Furthermore, the synchronization processing unit 135, based on the event data after conversion and sorting processing by the aggregation unit 133, sends control data to the terminal device 10, which is the source of the request, to synchronize the display of information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response, according to the progress of the workflow, thereby executing processing to control the synchronized display of each piece of information in the terminal device 10 (step S104).

[0104] Furthermore, the evidence synchronization unit 136 synchronizes the display changes in the terminal device 10 (step S105) by transmitting control data to the terminal device 10 to control the display changes to specific data locations in response to operations on the quoted elements by the user U of the terminal device 10, based on the correspondence between the quoted elements displayed with the answer information and specific data locations contained in the evidence information. This completes the processing procedure shown in Figure 8.

[0105] [5. Variant] The following describes a specific example of information processing according to the embodiment, using Figure 9. Figure 9 is a diagram showing a specific example of information processing by the information processing device according to the embodiment. In Figure 9, a specific example of information processing is described when the multi-agent system installed in the information processing device 100 is a "travel planning agent".

[0106] As shown in Figure 9, user U sends a user request to the information processing device 100, for example, an audit question for the "travel planning agent": "Create a one-night trip plan from Tokyo to Kyoto next week with a budget of 30,000 yen."

[0107] The graph processing unit 131 assigns a trace ID (trace_999) to the user request and executes a workflow to process the user request. The distribution unit 132 detects asynchronous events in real time that occur as a result of the execution of the workflow by the graph processing unit 131. For example, the distribution unit 132 detects an event such as "Node N1 [Intention Interpretation] has been activated." The information processing unit 100 also detects a tool call event where Node N2 has been activated and "a search query for hotels in Kyoto costing less than 15,000 yen per night has been sent to an external travel API." The information processing unit 100 also detects an event where Node N3 has been activated and is sequentially outputting (streaming) the final response text (token) one character at a time, such as "Day 1: Took the Shinkansen to Kyoto, stayed at Hotel X. Day 2: Sightseeing at Kiyomizu-dera Temple..." The distribution unit 132 streams the event data to the aggregation unit 133 in the order in which they occur, in a non-blocking manner.

[0108] The aggregation unit 133 extracts and groups only the events with a trace ID (trace_999) from the large amount of event data distributed by the distribution unit 132, and converts the disparate data structures returned from different travel APIs into a system-common format (normalized schema). The aggregation unit 133 also sorts the event data after conversion to the normalized schema based on the timestamp, rearranging it into the correct chronological order to complete the "normalized event column," and passes it on to the next stage.

[0109] The detection unit 134 determines that the execution session is "normal" as long as fragments of text such as "Day 1: By Shinkansen..." are continuously received from node N3 at regular intervals (for example, every 0.1 seconds). On the other hand, the detection unit 134 detects an anomaly if the [Hotel Search Tool] on node N2 is launched but the response from the external travel API is delayed and no events are received for more than 3 seconds (the difference in timestamps exceeds the threshold). The detection unit 134 determines that "the hotel search node has become unresponsive" and dynamically generates an "alert event" including the trace ID (trace_999), passing it on to the next stage.

[0110] The synchronization processing unit 135, based on the normalized event sequence received from the aggregation unit 133, delivers view information in real time to the three tabs on the terminal device 10's screen via a communication session corresponding to the trace ID.

[0111] The synchronization processing unit 135 transmits, for example, the text being generated, such as "Kyoto 1-night trip plan: Day 1..." output from node N3, in real time, like an animation of characters being typed out. The synchronization processing unit 135 also transmits update data of the processing flow, such as "Node N2 [Tool Call] is currently running." Furthermore, the synchronization processing unit 135 transmits specific numerical data and document source data that support calculations, such as "Shinkansen one-way: approximately 14,000 yen" and "Hotel X: 7,500 yen per night (with breakfast)" that node N2 actually searched.

[0112] Furthermore, if the synchronization processing unit 135 receives an "alert event," it can distribute interruption pop-up data to the entire screen that reads, "Connection to the travel API is congested. Please wait a moment."

[0113] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, and conversely, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0114] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0115] Furthermore, the embodiments described above can be combined as appropriate, provided that the processing content is not contradictory.

[0116] [6. Hardware Configuration] Furthermore, the information processing device 100 according to the embodiments or modified examples described above can be implemented by a computer 1000 having the configuration shown in Figure 10. Figure 10 is a hardware configuration diagram showing an example of a computer that implements the information processing according to the embodiments or modified examples.

[0117] Computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0118] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The primary storage device 1040 is a memory device, such as RAM, that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and is implemented using ROM (Read Only Memory), HDD, flash memory, etc.

[0119] Output IF1060 is an interface for transmitting information to be output to output devices 1010, such as monitors and printers, and is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). Input IF1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, and scanners, and is implemented using, for example, USB.

[0120] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), tape media, magnetic recording media, or semiconductor memory. Furthermore, the input device 1020 may be an external storage medium such as a USB memory stick.

[0121] Network IF1080 receives data from other devices via network N and sends it to the arithmetic unit 1030, and also transmits data generated by the arithmetic unit 1030 to other devices via network N.

[0122] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0123] For example, when the computer 1000 functions as an information processing device 100 according to an embodiment or modified example, the arithmetic unit 1030 of the computer 1000 performs the same function as the control unit 130 by executing a program loaded on the primary storage device 1040. That is, the arithmetic unit 1030 performs processing by the information processing device 100 according to an embodiment or modified example in cooperation with the information processing program loaded on the primary storage device 1040.

[0124] [7. Effects] As described above, the information processing device 100 according to the embodiment is an information processing device that processes requests by a workflow including multiple nodes using a large-scale language model, and comprises a distribution unit 132, an aggregation unit 133, and a synchronization processing unit 135. The distribution unit 132 distributes event data related to events that occur in conjunction with the execution of the workflow. The aggregation unit 133 aggregates the event data based on identification information (for example, "trace ID") for identifying the request. The synchronization processing unit 135 transmits control data to the terminal device 10 for synchronously displaying information related to the response to the request, information related to the progress of the workflow, and information related to the basis of the response, based on the event data aggregated by the aggregation unit 133.

[0125] In other words, the distribution unit 132 distributes event data in streaming format. The aggregation unit 133 identifies event data related to the same request from the event data distributed by the distribution unit 132 based on identification information, converts the identified event data into a predetermined data format, and aggregates it by arranging it chronologically. The synchronization processing unit 135 then executes processing to control the synchronous display of each piece of information in the terminal device 10, which is the source of the request, by sending control data to the terminal device 10, which is the source of the request, to synchronously display information related to the response to the request, information related to the progress of the workflow, and information related to the basis of the response, according to the progress of the workflow, based on the event data aggregated by the aggregation unit 133 ("normalized event sequence").

[0126] As described above, the information processing device 100 according to this embodiment can synchronize and display information such as information regarding responses to external requests (such as questions from users or requests from other systems through system integration), information regarding the progress of the workflow, and information regarding the basis for the responses on the same screen in accordance with the progress of the workflow, when executing a workflow using a graph-structured multi-agent system with a large-scale language model (LLM). This makes it possible to grasp the progress, responses, and basis information, which were previously scattered in individual logs, etc., in a unified and real-time manner, thereby eliminating information fragmentation.

[0127] Furthermore, according to the information processing device 100 of this embodiment, since each piece of information is displayed synchronously in chronological order on the terminal device 10, it is easy to track progress bottlenecks, such as "where the workflow processing stopped," and the reasoning process, such as "why that answer was reached." As a result, the time required to investigate system troubles or unexpected answers can be significantly reduced, and the efficiency of troubleshooting can be improved.

[0128] Furthermore, according to the information processing device 100 of this embodiment, during audits and system evaluations, it becomes possible to clearly present, at a glance, the progress of the workflow that led to a particular answer and the supporting data such as the referenced supporting documents. This has the excellent effect of easily fulfilling accountability for the output of a system using LLM.

[0129] For these reasons, the information processing device 100 according to this embodiment can support audit operations for multi-agent systems using a graph structure.

[0130] Furthermore, the synchronization processing unit 135 analyzes the event data aggregated by the aggregation unit 133, generates view data to be displayed in each of the multiple display areas that constitute the user interface displayed on the terminal device 10, stores each of the generated view data in the view information storage unit 122 in association with the display area, and transmits each of the view data stored in the view information storage unit 122 to the terminal device 10 as control data, either all at once or in parallel, via the same communication channel identified by the identification information corresponding to the same request. This reduces communication resources and connection overhead on the terminal side compared to establishing multiple communication channels individually to send and receive data, minimizes display discrepancies between multiple display areas, and efficiently achieves more precise synchronized display.

[0131] Furthermore, the information processing device 100 according to the embodiment further includes a detection unit 134. Based on the event data aggregated by the aggregation unit 133, the detection unit 134 detects the occurrence of a predetermined abnormality, including processing delays or stagnation, at each node included in each execution session, for each execution session identified by the identification information. In addition, when the detection unit 134 detects the occurrence of a predetermined abnormality, the synchronization processing unit 135 dynamically generates display data to display the detected predetermined abnormality on the user interface displayed on the terminal device 10, and transmits it to the terminal device 10 via the same communication channel identified by the identification information corresponding to the same request. As a result, when an abnormality occurs during the execution of a workflow, the information processing device 100 according to the embodiment allows an auditor, such as user U, to grasp (visualize) in real time and visually "what kind of abnormality (such as a hang-up or delay) is occurring at which node in which execution session" when an abnormality occurs.

[0132] Furthermore, the detection unit 134 determines that a predetermined abnormality has occurred if the allowable time set in advance for each node executed in the execution session, or the allowable waiting time for calling an external API (Application Programming Interface), is exceeded. As a result, the information processing device 100 according to the embodiment can reliably and quickly detect response delays of specific external services (external APIs) or processing delays at specific nodes.

[0133] Furthermore, the detection unit 134 determines that a predetermined abnormality has occurred if, after processing has started at a specific node in the execution session, the data fragments delivered in streaming format are interrupted for a predetermined period of time. As a result, the information processing device 100 according to the embodiment can detect serious abnormalities such as when the processing itself remains in the "running" status but the system completely stops (freezes, hangs up) and no error event is even issued.

[0134] Furthermore, the detection unit 134 determines that a predetermined abnormality has occurred in an execution session if the interval between receiving event data in the execution session exceeds a predetermined first threshold, or if the execution time of the same node among the nodes exceeds a predetermined second threshold. As a result, the information processing device 100 according to the embodiment can monitor abnormalities from multiple perspectives, both in terms of the overall health of the streaming communication and the processing delay of individual nodes, thereby improving the accuracy of detection.

[0135] Furthermore, when the detection unit 134 detects the occurrence of a predetermined anomaly, it dynamically generates an alert event containing the identification information for identifying the execution session in which the detected anomaly occurred and passes it to the synchronization processing unit 135. The synchronization processing unit 135 then generates display data to display the occurrence of the anomaly on the user interface displayed on the terminal device 10, based on the alert event that was passed to it. As a result, the information processing device 100 according to the embodiment can seamlessly link the processing from the occurrence of an anomaly to the notification of an alert to the terminal side in an event-driven manner, thereby improving the real-time performance and implementation flexibility of the system.

[0136] Furthermore, the information processing device 100 according to the embodiment further includes a basis synchronization unit 136. The basis synchronization unit 136 manages the correspondence between citation elements (e.g., "citation tags") displayed together with the information about the answer and specific data locations (e.g., "corresponding objects") included in the information about the basis. The basis synchronization unit 136 also synchronizes the display changes on the terminal device 10 by transmitting control data to the terminal device 10 to control the display changes to specific data locations in response to user U's operation on the citation elements of the terminal device 10, based on the correspondence. As a result, the information processing device 100 according to the embodiment can intuitively confirm "which part of the generated answer is based on which basis" through the operation of bidirectional links, thereby improving the convenience of basis tracing.

[0137] The effects described above can be achieved by any combination of the processes performed by the respective parts described above, or by any combination of the processes performed by the respective parts.

[0138] [8. Other] Although some embodiments of the present invention have been described in detail above with reference to the drawings, these are illustrative examples, and the present invention can be implemented in various other forms with modifications and improvements based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

[0139] Furthermore, the configuration of the aforementioned information processing device 100 can be flexibly changed, for example, by calling external platforms via APIs (Application Programming Interfaces) or network computing depending on the function.

[0140] Furthermore, the term "part" in the claims can be replaced with "means," "circuit," etc. For example, "control unit" can be replaced with "control means" or "control circuit." [Explanation of symbols]

[0141] SYS Information Processing System 10 Terminal devices 100 Information Processing Devices 110 Communications Department 120 Storage section 121 Event Information Storage Unit 122 View Information Storage Unit 123 Correspondence information storage unit 130 Control Unit 131 Graph Processing Unit 132 Distribution Department 133 Aggregation Section 134 Detection unit 135 Synchronization Processing Unit 136 Reason Synchronization Section

Claims

1. An information processing device that processes requests using a workflow involving multiple nodes with a large-scale language model, A distribution unit that distributes event data related to events that occur in connection with the execution of the aforementioned workflow, An aggregation unit aggregates the event data based on identification information that identifies the request, A synchronization processing unit transmits control data to a terminal device for synchronously displaying information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response, based on the event data aggregated by the aggregation unit. An information processing device characterized by comprising:

2. The aforementioned distribution unit, The aforementioned event data is distributed in streaming format. The aforementioned aggregation unit is From the event data distributed by the distribution unit, the event data relating to the same request is identified based on the identification information, the identified event data is converted to a predetermined data format, and aggregated by arranging it in chronological order. The synchronization processing unit, Based on the event data aggregated by the aggregation unit, the terminal device that sent the request executes a process to control the synchronized display of the information, by sending control data to the terminal device that sent the request, which is used to synchronize the display of the information, information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response, in accordance with the progress of the workflow. The information processing apparatus according to feature 1.

3. The synchronization processing unit, The event data aggregated by the aggregation unit is analyzed, and view data is generated to be displayed in each of the multiple display areas that constitute the user interface displayed on the terminal device. Each of the generated view data is associated with the display area and stored in a predetermined storage unit. Each of the view data stored in the predetermined storage unit is transmitted to the terminal device as control data, either collectively or in parallel, via the same communication channel identified by the identification information corresponding to the same request. The information processing apparatus according to feature 2.

4. Based on the event data aggregated by the aggregation unit, a detection unit detects the occurrence of a predetermined abnormality, including processing delays or stagnation, in each node included in the execution session, for each execution session identified by the identification information. Furthermore, The synchronization processing unit, When the detection unit detects the occurrence of the predetermined abnormality, it dynamically generates display data to display the detected predetermined abnormality on the user interface displayed on the terminal device, and transmits it to the terminal device via the same communication channel identified by the identification information corresponding to the same request. The information processing apparatus according to feature 2.

5. The detection unit, If the maximum allowable processing time or the allowable waiting time for calling an external API (Application Programming Interface) is exceeded for each node executed in the aforementioned execution session, it is determined that the predetermined abnormality has occurred. The information processing apparatus according to feature 4.

6. The detection unit, If, after processing has started at a specific node in the execution session, the data fragments delivered in streaming format are interrupted for a predetermined period of time, it is determined that the predetermined abnormality has occurred. The information processing apparatus according to feature 4.

7. The detection unit is If the interval for receiving the event data in the execution session exceeds a predetermined first threshold, or if the execution time of the same node among the nodes exceeds a predetermined second threshold, it is determined that the predetermined abnormality has occurred in the execution session. The information processing apparatus according to feature 4.

8. The detection unit, When the occurrence of the predetermined abnormality is detected, an alert event containing the identification information for identifying the execution session in which the detected predetermined abnormality occurred is dynamically generated and passed to the synchronization processing unit. The synchronization processing unit, Based on the alert event that was delivered, the display data is generated. The information processing apparatus according to feature 4.

9. The system further includes a basis synchronization unit that manages the correspondence between quoted elements displayed along with the information regarding the answer and specific data points contained in the information regarding the basis, The aforementioned basis synchronization unit is, Based on the aforementioned correspondence, control data for controlling the display changes to specific data locations in response to user operations on the quoted elements is transmitted to the terminal device, thereby synchronizing the display changes in the terminal device. The information processing apparatus according to feature 2.

10. An information processing method performed by a computer that processes requests using a workflow involving multiple nodes with a large-scale language model, A distribution process for distributing event data related to events that occur in conjunction with the execution of the aforementioned workflow, An aggregation step of aggregating the event data based on identification information that identifies the request, A synchronization processing step that transmits control data to a terminal device for synchronously displaying information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response, based on the event data aggregated in the aggregation step. An information processing method characterized by including

11. A computer that processes requests using a workflow involving multiple nodes with a large-scale language model, A distribution procedure for distributing event data related to events that occur in connection with the execution of the aforementioned workflow, An aggregation procedure for aggregating the event data based on identification information that identifies the request, A synchronization processing procedure that transmits control data to a terminal device for synchronously displaying information regarding the response to the request, information regarding the progress of the workflow, and information regarding the basis for the response, based on the event data aggregated by the aggregation procedure. An information processing program characterized by causing the execution of [a specific action].

Images