AI agent-to-agent decision-making information sharing device and program

Abstract

When using multiple artificial intelligence agents, user modifications, rejections, acceptances, evaluations, and subsequent decisions are not carried over to subsequent agents, resulting in redundant explanations and inconsistent responses. [Solution] The information processing device 30 comprises a processor 31, memory 32, storage device 33, and communication interface 34. The processor 31 writes feedback events, which associate the output presented by an artificial intelligence agent or artificial intelligence tool with user response information to said output, as structured data to the storage device 33, and generates or updates a user-specific decision-making dataset. Furthermore, in response to requests from subsequent artificial intelligence agents or artificial intelligence tools, it provides contextual information, judgment criterion information, ranking information, or output adjustment information based on the dataset.

Description

Technical Field

[0001] The present invention relates to information processing technology that accumulates information on human judgment, modification, rejection, adoption, evaluation, status reporting, and post - decision review obtained through interaction or task execution with an artificial intelligence agent or an artificial intelligence tool as structured data on a storage device and provides it to a plurality of heterogeneous artificial intelligence agents or artificial intelligence tools.

[0002] In particular, the present invention relates to a user - specific decision - making dataset generation device that generates feedback events, decision - making reviews, and preference pairs from the daily use of a plurality of artificial intelligence agents or artificial intelligence tools and uses them for subsequent context search, ranking, and output adjustment of artificial intelligence agents or artificial intelligence tools.

Background Art

[0003] In recent years, users may use a plurality of artificial intelligence agents such as coding agents on terminals, agents within integrated development environments, in - browser agents, cloud - type agents, and in - company agents according to tasks, work areas, model performance, and contract conditions.

[0004] In this usage form, since each artificial intelligence agent has an independent session history, prompt, rule file, memory function, or evaluation history, the user's judgment, policy, preference, exception rule, past decision - making reasons, or current state formed in the interaction with one agent is not naturally carried over to another agent.

[0005] Conventionally, memory layers, knowledge graphs, vector search, dialogue history search, and life log technologies for artificial intelligence agents are known. These are technologies for storing, searching, or summarizing factual information, conversation history, screen or voice observation records.

[0006] Furthermore, developer platforms are known for observing, evaluating, prompting, and human-scoring LLM applications. These are technologies for collecting trace or evaluation data that developers or providers use to improve the overall quality of their applications.

[0007] Furthermore, techniques for tuning artificial intelligence models using preference pairs are known, such as RLHF, DPO, reward model learning, and preference model learning. However, these are primarily techniques related to learning algorithms or large-scale annotation infrastructure, and do not adequately provide a concrete device configuration for continuously generating preference datasets corresponding to user entities such as the user themselves, user groups, departments, teams, projects, roles, organizations, tenants, or work areas, and using them for contextual search, reranking, and output adjustment during inference for subsequent agents.

[0008] In addition, in collaborative support technologies such as video review, techniques are known that maintain review comments, drawings, or approval information in chronological order or in conjunction with the target object's position. While such techniques are useful in that they maintain human review judgments on the target object, they do not convert the output, modifications, rejections, acceptances, and post-evaluations obtained from multiple artificial intelligence agents into user-specific decision-making datasets for use in ranking subsequent agents or adjusting their output. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Special Publication No. 2018-522343

[0010] [Patent Document 2] Special Publication No. 2023-513146

[0011] [Patent Document 3] Patent No. 7760209 [Non-patent literature]

[0012] [Non-Patent Document 1] Mem0, https: / / mem0.ai /

[0013] [Non-Patent Document 2] Letta, https: / / www.letta.com /

[0014] [Non-Patent Document 3] Zep / Graphiti, https: / / www.getzep.com /

[0015] [Non-Patent Document 4] Langfuse, https: / / langfuse.com /

[0016] [Non-Patent Document 5] LangSmith, https: / / www.langchain.com / langsmith

[0017] [Non-Patent Document 6] OpenAI memory, https: / / help.openai.com /

[0018] [Non-Patent Document 7] Anthropic memory tool and MCP related materials, https: / / modelcontextprotocol.io /

[0019] [Non-Patent Document 8] Rewind / Limitless, https: / / www.limitless.ai / [Summary of the Invention] [Problems to be Solved by the Invention]

[0020] The problem to be solved by the present invention is to reduce the problem that when a user uses a plurality of different artificial intelligence agents in parallel, each agent cannot inherit the user's past judgments, corrections, rejections, acceptances, and post-evaluation, resulting in duplicate explanations and inconsistent judgments for each session.

[0021] Another problem to be solved by the present invention is to convert human corrections and evaluations of the output of artificial intelligence agents into structured data that can be used for subsequent searches, rankings, and output adjustments, rather than just searching for simple facts or conversation histories.

[0022] Yet another problem to be solved by the present invention is to make available to a plurality of artificial intelligence agents the user's decision-making data, preference data, business judgments, technical judgments, and evaluation histories while storing them in a storage device or storage resource that is not limited to any of a terminal, server, cloud, or distributed system.

Means for Solving the Problems

[0023] A first aspect of the present invention is an information processing apparatus that generates a user-specific decision-making data set corresponding to a user subject from a conversation history or task execution history with an artificial intelligence agent or an artificial intelligence tool and provides it to a plurality of artificial intelligence agents or artificial intelligence tools. The information processing apparatus includes one or more processors, a memory, a storage device, and a communication interface.

[0024] The processor executes a process of receiving, via the communication interface, or generating using the memory, a feedback event in which at least one of an output presented by the artificial intelligence agent or artificial intelligence tool to the user and a user correction operation, acceptance or rejection, evaluation input, final adoption result, or user response information corresponding thereto for the output is associated with a conversation or task execution with the first artificial intelligence agent or artificial intelligence tool.

[0025] The processor transforms the feedback event according to a predetermined data schema and writes it to the storage device as structured data relating to the judgment or decision of a user belonging to the user entity, thereby generating or updating at least a portion of the user entity-specific decision dataset corresponding to the user entity on the storage device.

[0026] The processor provides contextual information, decision criteria information, ranking information, or output adjustment information based on the user-specific decision-making dataset via the communication interface in response to requests from one or more subsequent artificial intelligence agents, artificial intelligence tools, or subsequent sessions of artificial intelligence agents that are the same as or different from the first artificial intelligence agent or artificial intelligence tool.

[0027] The processor may receive or generate a decision review, including the results, lessons learned, final judgment, outcome, confidence level, or re-evaluation, after a predetermined time has elapsed for the feedback event, and may record the decision review in the storage device.

[0028] The processor may generate preference pairs, which associate an accepted output indicating an accepted output with a rejected output indicating an rejected output, based on the feedback event and the decision review, and record the preference pairs in the storage device.

[0029] In a second aspect of the present invention, the processor searches for candidate contexts using vector representations, full-text indexes, or combinations thereof held in a memory device, scores the candidate contexts based on past feedback events, decision reviews, and preference pairs, and reconstructs the ranking of the candidate contexts.

[0030] In a third aspect of the present invention, the communication interface includes loopback HTTP, WebSocket, Unix Domain Socket, named pipe, standard input / output, MCP, gRPC, shared memory, function call, HTTP API, HTTPS, message queue, or equivalent intra-device or network communication paths. In one embodiment, the communication interface may be configured to restrict connections from an external network.

[0031] In a fourth aspect of the present invention, the processor does not read the authentication information stored in the confidential information storage area provided by the operating system at startup, but reads it as needed when receiving a search request, a vector representation generation request, or an external embedded provider usage request.

[0032] In a fifth aspect of the present invention, the processor uses an embedded model on the same device or a managed server as the default provider, and uses an external embedded provider only when explicitly configured by the user or administrator. This allows the user's decision data and preference data to be stored in a managed storage device or memory resource according to the execution environment being used. [Effects of the Invention]

[0033] According to the present invention, even when a user uses multiple artificial intelligence agents in parallel, past modifications, rejections, acceptances, evaluations, and post-reviews are provided to subsequent agents, thereby improving the consistency of the responses of each artificial intelligence agent.

[0034] According to the present invention, instead of simply retrieving conversation history or facts, the relationship between the output of the artificial intelligence agent and the user's modified output can be recorded as a preference pair, making it easier to reflect user-specific decision-making criteria in subsequent agents.

[0035] According to the present invention, by using a storage device and a communication interface, user decision data and preference data can be utilized by multiple agents in any execution environment, whether it be a terminal, server, cloud, or distributed system.

[0036] According to the present invention, even if the initial evaluation is reversed later through a decision review, the weights or acceptance / rejection flags of the preference pairs can be updated, so that the actual results after time has elapsed can be reflected in subsequent ranking or output adjustments.

[0037] According to the present invention, even without immediately retraining a heavy generative model by reranking search results, user-specific decision criteria can be reflected in context selection using computing resources on terminals, servers, the cloud, or distributed systems. [Brief explanation of the drawing]

[0038] [Figure 1] This is an overall configuration diagram of an information processing device according to one embodiment of the present invention.

[0039] [Figure 2] This figure shows an example of the main data structures held in a storage device.

[0040] [Figure 3] This sequence diagram illustrates an example of a process in which contextual information is inherited from a first artificial intelligence agent to a second artificial intelligence agent.

[0041] [Figure 4] This figure shows an example of a process for acquiring or generating feedback events.

[0042] [Figure 5] This figure shows an example of a process for recording decision reviews and handling evaluation reversals.

[0043] [Figure 6]This figure shows an example of a process for generating preference pairs from feedback events and decision reviews.

[0044] [Figure 7] This figure shows an example of a process that reranks search results based on past reviews and preference pairs.

[0045] [Figure 8] This figure shows an example of an API configuration using an internal communication channel or a network communication channel within the same device. [Modes for carrying out the invention]

[0046] Embodiments of the present invention will now be described with reference to the drawings. In this embodiment, an artificial intelligence agent means software that performs investigation, implementation, document creation, decision support, operation support, or equivalent processing based on natural language, commands, tool calls, file operations, or work area information from a user. In this embodiment, an artificial intelligence tool means an application, extension, command line tool, or functional module that includes an artificial intelligence model or processing that utilizes an artificial intelligence model and performs interaction, search, code execution, file operations, document generation, external API access, or decision support. An artificial intelligence tool may operate as independent software or may operate as part of an artificial intelligence agent. In this embodiment, a user entity means a single user, a group of users consisting of multiple users, a department, a team, a project, a role, a job title, an organization, a tenant, a work area, an account, a terminal, or a combination thereof.

[0047] As shown in Figure 1, the information processing device 30 comprises a processor 31, memory 32, storage device 33, communication interface 34, and secure information storage area access means 35. The information processing device 30 is implemented, for example, as a desktop application, server application, cloud service, container, virtual machine, or distributed system running on a user's personal computer.

[0048] The processor 31 functions as an event acquisition module 50, a decision review management module 60, a preference pair generation module 70, a context search module 80, a reranking module 90, and an API endpoint 100 by executing programs deployed in memory 32.

[0049] As shown in Figure 2, the storage device 33 holds the database 40. The storage device 33 may be a storage device within the information processing device 30, an external storage device, a storage device on a server, cloud storage, a distributed data store, or a combination thereof. The database 40 may include a session history table 41, an agent status table 42, an inter-agent message table 43, a search index table 44, a feedback event table 45, a decision review table 46, and a preference pair table 47. The database 40 is physically recorded on the storage device 33 as a relational database, a vector database, a full-text search index, a graph database, or a combination thereof, and is subject to join, search, scoring, and ranking calculations by the processor 31. The user-specific decision dataset is a structured data set corresponding to a user, including at least a portion of the feedback event table 45, and depending on the embodiment, including the decision review table 46, the preference pair table 47, the search index table 44, vector representations, and their reference relationships. The user-specific decision-making dataset may be an individual dataset corresponding to a single user, or it may be a dataset corresponding to a group of users, a department, a team, a project, a role, a job title, an organization, a tenant, a work area, an account, a device, or a combination thereof.

[0050] The feedback event table 45 may include an event identifier, input context, searched context, artificial intelligence output, user response information, user modification, modified output, status, reason for rejection or acceptance, originating artificial intelligence agent identifier, work area identifier, and time. User response information may include the user's modification operation, acceptance / rejection, evaluation input, final acceptance result, hold, re-execution instruction, or equivalent responses.

[0051] The decision review table 46 may include a review identifier, feedback event reference, final decision, outcome, confidence level, review time, and evaluation reversal flag. The evaluation reversal flag is set when the initial evaluation and the post-evaluation differ.

[0052] The preference pair table 47 may include preference pair identifiers, query context, accept output, reject output, weights, training output status, referenced decision review, and update time. The training output status may indicate states such as unused, output for training, disabled, or awaiting re-evaluation.

[0053] As shown in Figure 3, the first artificial intelligence agent 10 sends output, user corrections, evaluations, status reports, or session termination information to the API endpoint 100 during task execution or at the end of a session. The information processing device 30 normalizes this information and records it in the storage device 33. When the subsequent second artificial intelligence agent 20 sends a context request at the start of work, the context search module 80 and the reranking module 90 extract relevant past records, rank them, and provide them to the second artificial intelligence agent 20.

[0054] As shown in Figure 4, the event acquisition module 50 associates the output presented by the artificial intelligence agent with the modified output actually adopted by the user. For example, if the artificial intelligence agent presents a response that only refers to a static configuration file, and the user modifies it to a procedure that checks the actual runtime state, the former can be recorded as the rejected output and the latter as the accepted output.

[0055] As shown in Figure 5, the decision review management module 60 receives results, defects, performance, reproducibility, or review results from team members after a predetermined time has elapsed since task execution. Even if an output was initially accepted, if defects or problems are discovered later, the decision review management module 60 records information indicating a reversal of the evaluation.

[0056] As shown in Figure 6, the preference pair generation module 70 refers to the feedback event table 45 and the decision review table 46 to generate pairs of accept and reject outputs. If the evaluation is reversed by the post-review, the preference pair generation module 70 updates the weights, accept / reject flags, or output status of the preference pair.

[0057] As shown in Figure 7, the context search module 80 performs vector similarity search and text matching search using the work area identifier, task body, or request body. The reranking module 90 processes the candidate contexts, increasing the score of contexts that were deemed successful in past decision reviews and decreasing the score of contexts that were deemed unsuccessful or rejected.

[0058] As shown in Figure 8, the communication interface 34 can use loopback HTTP, Unix Domain Socket, standard input / output, MCP, named pipe, WebSocket, HTTP API, HTTPS, gRPC, message queue, or equivalent intra-device or network communication paths. In one embodiment, the information processing device 30 may be configured to restrict connections from external networks.

[0059] In the first embodiment, the information processing device 30 is implemented as a Tauri desktop application. The storage device 33 holds SQLite and vector search indexes, and the communication interface 34 provides an HTTP API or MCP standard input / output coupled to a loopback address.

[0060] In the second embodiment, the information processing device 30 is implemented as an Electron application, a CLI resident process, an OS resident service, a server application, a cloud service, a container, a virtual machine, or a distributed system. The storage device 33 may be DuckDB, PostgreSQL, RocksDB, a file-based store, object storage, a distributed data store, a vector database, a graph database, or a combination thereof.

[0061] In the third embodiment, an artificial intelligence agent on the cloud queries the information processing device 30 via an adapter on the terminal or server. In this case, the user's decision-making data and preference data are stored in a storage device 33 on the terminal, server, cloud, or distributed system, and the information transmitted externally may be summarized, anonymized, or limited to contextual information approved by the user or administrator.

[0062] In the above embodiment, an embedded model on the same device or a managed server can be used as the default embedded provider. An external embedded provider may be used only when the user or administrator has set up API authentication information that they have prepared themselves. The confidential information storage area access means 35 can access the credential management function provided by the operating system, the secret information management function of the server, the cloud secret management service, or a confidential information storage area equivalent thereto.

[0063] This invention can be applied not only to personal use but also to employee terminals within a company, development teams, legal teams, sales teams, collaborative work with external parties, and customer support teams. Companies can improve the response consistency of artificial intelligence agents while managing employee or team member decision data on terminals, servers, the cloud, or storage devices on distributed systems. [Explanation of Symbols]

[0064] 10. The First Artificial Intelligence Agent

[0065] 20. The Second Artificial Intelligence Agent

[0066] 30 Information Processing Devices

[0067] 31 processors

[0068] 32 memory

[0069] 33 Storage device

[0070] 34 Communication Interfaces

[0071] 35. Means for accessing the confidential information storage area

[0072] 40 databases

[0073] 41 Session History Table

[0074] 42 Agent Status Table

[0075] 43 Inter-agent message table

[0076] 44 Search Index Table

[0077] 45 Feedback Event Table

[0078] 46 Decision Review Table

[0079] 47 Preferred Pair Tables

[0080] 50 Event Acquisition Modules

[0081] 60 Decision Review Management Module

[0082] 70 Preference Pair Generation Module

[0083] 80 Contextual Search Module

[0084] 90 Re-ranking Module

[0085] 100 API endpoints

Claims

1. An information processing device that generates a user-specific decision-making dataset corresponding to a user entity from the history of interaction with an artificial intelligence agent or artificial intelligence tool or the history of task execution, and provides it to multiple artificial intelligence agents or artificial intelligence tools, comprising one or more processors, memory, storage device, and communication interface, wherein the processor receives or generates a feedback event that associates the output presented to the user by the artificial intelligence agent or artificial intelligence tool with at least one of the user's modification operation, acceptance or rejection, evaluation input, final acceptance result, or equivalent user response information to the output, transforms the feedback event according to a predetermined data schema, and generates or updates the user-specific decision-making dataset by writing it to the storage device as structured data relating to the judgment or decision of the user belonging to the user entity, and provides contextual information, judgment criterion information, ranking information, or output adjustment information based on the user-specific decision-making dataset via the communication interface in response to requests from subsequent artificial intelligence agents, artificial intelligence tools, or subsequent sessions of artificial intelligence agents.

2. The information processing device according to claim 1, wherein the user entity is at least one of a single user, a group of users consisting of multiple users, a department, a team, a project, a role, a job title, an organization, a tenant, a work area, an account, a terminal, or a combination thereof, and the feedback event includes at least one of an event identifier, a prompt context, a searched context, an artificial intelligence output, user response information, user correction, a corrected output, a status, a reason for rejection or acceptance, a source artificial intelligence agent identifier, a work area identifier, and a time.

3. An information processing device according to claim 1 or 2, wherein the processor records a decision review in the storage device, including the result, lesson learned, final judgment, outcome, confidence level, or re-evaluation after a predetermined time has elapsed for the feedback event; generates preference pairs associating an acceptance output with a rejection output based on the feedback event or the decision review; and updates the weights, acceptance / rejection flags, or learning output state of the preference pairs when the evaluation reverses from the initial evaluation in the decision review.

4. The information processing apparatus according to Claim 3, wherein the processor searches for candidate contexts using vector representations, full-text indexes, or combinations thereof held in the storage device, reconstructs the ranking of candidate contexts based on the feedback events, the decision reviews, the preference pairs, or the decision data set for each user, and the communication interface includes at least one of loopback HTTP, WebSocket, domain socket, named pipe, standard input / output, MCP, gRPC, shared memory, function call, HTTP API, HTTPS, message queue, or equivalent intra-device communication path or network communication path.

5. A program characterized by causing one or more computers to execute each of the processes of the processor in the information processing device described in claim 1.

Images