Information processing system based on anomaly detection and autonomous countermeasure inference by an AI agent

Abstract

To quickly identify the cause and derive solutions for anomalies in processes autonomously executed by AI agents that handle multiple users. [Solution] The present invention is an information processing system comprising: an AI agent that autonomously performs processing in response to multiple users; an anomaly detection unit that monitors the processing and detects anomalies; a cause estimation unit that estimates the cause of the detected anomaly; and a solution method inference unit that infers a solution method for the anomaly based on the estimated cause.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and an information processing program that monitor processes autonomously executed by an AI agent, detect abnormalities in the processes, estimate the causes, and infer solutions.

Background Art

[0002] In recent years, AI agents that autonomously execute tasks based on user instructions using AI are being widely used, and AI agents may automatically execute processes such as access to external business systems, acquisition of information, registration of information, and transmission of notifications.

[0003] For example, a technique has been proposed in which an AI agent decomposes a predetermined task into a plurality of tasks, assigns sub-agents corresponding to each task, evaluates and generates sub-agents based on execution results, and updates the assignment.

[0004] However, when an AI agent autonomously executes a process on a business system, an abnormality including a risk such as information leakage may occur due to an error in the destination or an inconsistency in the set value. On the other hand, in the conventional technology, a framework for detecting an abnormality inherent in the process of the agent itself, estimating the cause, and inferring a solution has not been sufficiently provided.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] When an AI agent autonomously performs processing on an external business system, errors may occur due to processing mistakes or inappropriate settings. Such errors may manifest not only as processing failures but also as risk events such as unintended sharing of information with external parties.

[0007] Therefore, there is a need for technology that enables AI agents to monitor their own processes, detect anomalies, estimate the cause of the anomaly, and infer solutions to the anomaly based on the estimated cause, thereby enabling them to respond to anomalies and prevent their recurrence. [Means for solving the problem]

[0008] The information processing system according to the present invention comprises an AI agent that autonomously performs processing, an anomaly detection unit that monitors the processing performed by the AI ​​agent and detects abnormalities in the processing, a cause estimation unit that estimates the cause of the detected anomaly, and a solution method inference unit that infers a solution method for the anomaly based on the estimated cause.

[0009] Furthermore, in one aspect of the present invention, at least one of the cause estimation unit and the solution method inference unit inputs contextual information to a large-scale language model provided via a network and performs cause estimation or solution method inference based on the output of the large-scale language model.

[0010] Furthermore, in one aspect of the present invention, incident records relating to anomalies are generated according to a predetermined structured template and stored as knowledge, the cause is estimated or a solution is inferred by referring to the stored incident records, and proactive intervention can be performed in response to the detection of similar situations. [Effects of the Invention]

[0011] According to the present invention, an AI agent can detect anomalies in processes it autonomously executes, estimate the cause of the anomaly, and infer a solution. This enables a rapid response when an anomaly occurs, thereby suppressing the interruption of business processes and the expansion of risks.

[0012] Furthermore, according to one aspect of the present invention, it is possible to perform causal estimation and solution method inference based on contextual information using a large-scale language model, and even if the nature of the anomaly is diverse, the cause of the anomaly and the solution method can be derived flexibly.

[0013] Furthermore, according to one aspect of the present invention, it is possible to accumulate incident records, detect similar situations using embedded vectors, take proactive measures, verify proposed modifications, evaluate their effectiveness and update them, thereby contributing to preventing the recurrence of abnormalities and stabilizing operations. [Brief explanation of the drawing]

[0014] [Figure 1] This is a diagram illustrating the configuration of the present invention. [Figure 2] This is a functional block diagram of the present invention. [Figure 3] This is a flowchart of the present invention. [Figure 4] This is a sequence diagram of the present invention. [Figure 5] This is another sequence diagram of the present invention. [Modes for carrying out the invention]

[0015] Embodiments of the present invention will be described below with reference to the drawings. Note that this embodiment is an example of the present invention, and the present invention is not limited to this embodiment.

[0016] As shown in Figure 1, the information processing system 10 is configured by connecting external terminals 12, information processing devices 20, external business systems 30, model provisioning devices 40, etc., to each other via a network 11 so that they can communicate with one another.

[0017] The external terminal 12 is a terminal for users to access the information processing system 10, and is comprised of, for example, a personal computer, a tablet device, a smartphone, etc.

[0018] The external business system 30 is an external information processing system that the information processing apparatus 20 accesses for executing processing, and may include, for example, a scheduling system, a messaging system, a customer management system, a development management system, and the like.

[0019] The information processing apparatus 20 is configured by, for example, server or computer resources on the cloud, and executes the AI agent 21. The AI agent 21 autonomously executes predetermined processing based on user instructions or internal events.

[0020] The processing executed by the AI agent 21 may include access to the external business system 30, acquisition of information, registration, update, notification, sharing, and the like. Further, the processing may be decomposed into a plurality of sub-processes and executed sequentially or in parallel.

[0021] The information processing apparatus 20 includes a processing execution unit 22 that executes the processing by the AI agent 21, an abnormality detection unit 23 that monitors the processing by the processing execution unit 22 and detects an abnormality in the processing, a cause estimation unit 24 that estimates the cause of the abnormality, and a solution method inference unit 25 that infers a solution method for the abnormality based on the cause estimation result.

[0022] The processing execution unit 22 performs processing such as, for example, calling an API of the external business system 30, generating parameters to be transmitted to the external business system 30, and interpreting the response of the external business system 30.

[0023] The abnormality detection unit 23 acquires the execution context regarding the processing executed by the processing execution unit 22, and detects an abnormality based on the execution context. The execution context may include a processing execution log, input parameters used in the processing, destination identification information generated by the processing, transmission or sharing setting information, a processing result, authority information, and the like.

[0024] Anomalies can include not only processing failures, unexpected errors, and inconsistencies in the processed data, but also risk events such as unintended sharing with external parties, incorrect recipients, and abuse of authority. Furthermore, anomalies are not limited to cases where actual harm occurs, but may also include near misses that were detected before they occurred.

[0025] The anomaly detection unit 23 may detect an anomaly by evaluating whether the destination identification information included in the execution context was generated by estimation, whether the destination is outside a predetermined trusted area, whether the settings to expand the scope of transmission or sharing are included, etc.

[0026] The anomaly detection unit 23 may detect anomalies using a rule-based approach, a machine learning model, or a combination of both.

[0027] In one embodiment, the anomaly detection unit 23 can input at least one of the following into a large-scale language model 41 provided via a network and have it evaluate it, thereby detecting anomalies based on the evaluation results.

[0028] The cause estimation unit 24 estimates the cause of the detected anomaly based on the execution context and related information. The cause may include, for example, an incorrect input value, an incorrect destination, inconsistent setting values, excess or insufficient permissions, a change in the specifications of the external business system 30, or a missing processing procedure.

[0029] The cause estimation unit 24 may estimate the reproduction conditions and scope of impact depending on the type of anomaly. For example, if the anomaly relates to external sharing, it may estimate the shared object, the notification target, the confidentiality of the shared content, etc.

[0030] The solution method inference unit 25 infers a solution method for the anomaly based on the cause estimation result. The solution method may include, for example, re-executing the process, modifying input parameters, confirming the destination, changing the settings for sending or sharing, reviewing permissions, adding processing steps, etc.

[0031] The solution method inference unit 25 may present the inferred solution method as advice to the AI ​​agent 21, or it may apply it to the processing execution unit 22 as control information for the AI ​​agent 21 to re-execute the process.

[0032] In one embodiment, the information processing device 20 may further include a knowledge storage unit 26 that stores incident records related to anomalies as knowledge. The incident records include, for example, the content of the event, the cause, the solution, the category, and the severity, and are generated according to a structured template.

[0033] The model providing device 40 is a device that provides a large-scale language model 41 via the network 11. The large-scale language model 41 is used as an inference model by the cause estimation unit 24, the solution method inference unit 25, or the anomaly detection unit 23.

[0034] At least one of the cause estimation unit 24 and the solution method inference unit 25 can input contextual information, such as execution context, details of anomalies, and past incident records, into the large-scale language model 41, and perform cause estimation or solution method inference based on the estimation results output by the large-scale language model 41.

[0035] The large-scale language model 41 may be implemented on the same computing resources as the information processing device 20, or it may be provided via the network 11 as a separate computing resource from the information processing device 20.

[0036] Next, an example of the information processing method according to this embodiment will be described with reference to Figure 3.

[0037] First, the AI ​​agent 21 starts processing in response to a processing request from the external terminal 12 or an internal event (step S101). Next, the processing execution unit 22 executes the processing using the AI ​​agent 21 (step S102).

[0038] Next, the anomaly detection unit 23 monitors the processing performed by the processing execution unit 22 and obtains the execution context (step S103).

[0039] The anomaly detection unit 23 detects an anomaly in the process based on the execution context (step S104). If no anomaly is detected, the process may continue (step S110).

[0040] If an abnormality is detected, the cause estimation unit 24 estimates the cause of the detected abnormality (step S105).

[0041] Next, the solution method inference unit 25 infers a solution method for the anomaly based on the estimated cause (step S106). At least one of the cause estimation and solution method inference may be performed using the large-scale language model 41.

[0042] Next, in an embodiment that includes a knowledge storage unit 26, the solution method reasoning unit 25 or the anomaly detection unit 23 generates an incident record according to a structured template and stores it in the knowledge storage unit 26 (step S107).

[0043] Next, in an embodiment that includes an intervention control unit 28, the intervention control unit 28 performs a pre-intervention in response to the detection of an abnormality (step S108). The pre-intervention may include at least one of the following: issuing a warning, requesting confirmation before execution, changing the execution conditions of the process, or suppressing the execution of the process.

[0044] Changing the execution conditions may include running the process in an execution mode that suppresses external notification or sharing, or running the process in a simulation execution mode. The simulation execution mode is an execution mode that allows the results of the process to be confirmed without actual transmission or reflection to the external business system 30.

[0045] Next, in an embodiment that includes an effectiveness evaluation unit 29, the effectiveness evaluation unit 29 evaluates the results of applying the solution method or the results of prior intervention, and updates the priority of the incident records stored in the knowledge storage unit 26, the judgment criteria of the anomaly detection unit 23, or the input conditions to the large-scale language model 41 based on the evaluation results (step S109).

[0046] Next, we will describe an embodiment that includes proposed modifications in the solution method inference. The solution method inference unit 25 can generate proposed modifications to the program that executes the process or proposed changes to the parameters used in the process as a solution method.

[0047] The solution method inference unit 25 may, for example, generate differential information as a proposed modification, or generate recommended parameter values ​​as a proposed change.

[0048] The information processing device 20 is equipped with a verification environment 50 and can determine whether the proposed modification or change is applicable based on the results of testing the modification or change in the verification environment 50.

[0049] The verification environment 50 is an environment that minimizes the impact on the external business system 30, and can be configured, for example, by performing simulated execution using mocks or stubs, applying to a sandbox environment, or performing reproduction execution using historical data.

[0050] Next, an example of the processing in this embodiment will be described as a sequence with reference to Figures 4 and 5. When a processing request is sent from the external terminal 12 to the information processing device 20, the AI ​​agent 21 starts processing, and the processing execution unit 22 executes processing including a tool call to the external business system 30.

[0051] The anomaly detection unit 23 monitors the process and, upon acquiring an execution context suspected of being abnormal, inputs the contextual information into the large-scale language model 41 for evaluation. Based on the evaluation results, it can detect anomalies.

[0052] When an anomaly is detected, the cause estimation unit 24 and the solution method inference unit 25 derive the cause and solution method using the large-scale language model 41 as necessary, and the intervention control unit 28 may intervene, such as requesting confirmation or changing execution conditions, before continuing processing.

[0053] Next, let's look at some specific use cases. For example, when AI agent 21 performs the process of sending a meeting invitation to a scheduling system, if an email address is used as the recipient identification information, and if that email address is generated through estimation, there is a risk of information leakage due to misdelivery.

[0054] In this case, the anomaly detection unit 23 can detect an anomaly by detecting that the destination identification information included in the execution context is estimated and generated, or that the destination is outside the trusted area, or that the setting includes an expansion of the scope of transmission or sharing.

[0055] The cause estimation unit 24 may estimate that the cause of the abnormality is insufficient determination of the destination identification information, or that the transmission or sharing settings are excessive.

[0056] The solution method inference unit 25 may infer solutions such as requesting confirmation of destination identification information, executing the process in an execution mode that suppresses notification or sharing, switching to a simulation execution mode, or suppressing the execution of the process.

[0057] Furthermore, in an embodiment that includes a knowledge storage unit 26, an incident record relating to the anomaly is generated according to a structured template, and the event details, cause, solution, category, and severity are stored.

[0058] The cause estimation unit 24 or the solution method inference unit 25 can more quickly estimate the cause and infer a solution for similar anomalies by referring to past incident records stored in the knowledge storage unit 26.

[0059] The importance level may be calculated based on, for example, whether or not it is sent externally, destination attributes, authorization attributes, or a confidentiality estimate. The confidentiality estimate may be calculated, for example, by a large-scale language model 41.

[0060] If the importance level is high, the intervention control unit 28 may implement stronger interventions, such as suppressing processing or forcing verification.

[0061] Next, the detection of similar situations in the knowledge storage unit 26 will be described. The knowledge storage unit 26 may convert at least a portion of the incident record into an embedding vector and store it.

[0062] The anomaly detection unit 23 converts the current context related to the current process into an embedding vector, calculates the similarity between the embedding vector and the embedding vector stored in the knowledge storage unit 26, and can detect an anomaly based on past incidents if the similarity is greater than or equal to a predetermined value.

[0063] This allows for preventative detection based on similar situations before an anomaly becomes apparent.

[0064] Furthermore, the intervention control unit 28 can automatically apply an execution mode or simulation execution mode that suppresses external notification or sharing in response to preventative detection based on similar situations.

[0065] The generation of the embedding vector may be performed by the information processing device 20 or by an external model providing device 40.

[0066] The embedding vector is generated to reflect the contextual information contained in the incident log and may include features such as the nature of the destination identifier, transmission settings, permissions, and confidentiality.

[0067] In the configuration using the verification environment 50, the proposed modifications or changes generated by the solution reasoning unit 25 can be tested in the verification environment 50, and the applicability can be determined based on the success or failure of the test. This allows for autonomous improvement while suppressing adverse effects on the external business system 30.

[0068] Multiple AI agents 21 may be provided, and incident records stored in the knowledge storage unit 26 can be shared by multiple AI agents 21. In this case, incident records related to an anomaly detected by one AI agent 21 can be used by another AI agent 21 for anomaly detection, cause estimation, and solution method inference.

[0069] The information processing system 10 may have multiple tenants corresponding to multiple user organizations, and the knowledge storage unit 26 may remove personally identifiable information or organization-specific information from incident records and generate and store generalized pattern information abstracted by a large-scale language model 41.

[0070] Generalized pattern information allows risk patterns and countermeasure patterns to be shared in a way that can be referenced among user organizations, while suppressing the inclusion of personal information and organization-specific information.

[0071] The effectiveness evaluation unit 29 evaluates the results of applying the solution or intervention, and based on the evaluation results, it may update the priority of the incident record, the judgment criteria of the anomaly detection unit 23, or the prompts to be input to the large-scale language model 41.

[0072] This makes it possible to maintain or improve the accuracy of anomaly detection and inference in response to changes in the operating environment, changes in the specifications of external business systems 30, changes in data distribution, etc.

[0073] Furthermore, the processes of anomaly detection, cause estimation, and solution method inference may be performed on a single device or distributed across multiple devices.

[0074] The program of the present invention can enable a computer to function as an information processing system that uses a large-scale language model 41 provided via a network, at least one of the cause estimation unit 24 and the solution method inference unit 25.

[0075] Furthermore, the present invention is not limited to the embodiments described above, and can be modified as appropriate by those skilled in the art, and may include various modifications within the scope of the claims.

[0076] According to the embodiments described above, an information processing system can be provided that helps in responding to anomalies and preventing recurrence by detecting anomalies when an AI agent autonomously executes processing, estimating the cause, and inferring a solution. [Explanation of symbols]

[0077] 10 Information Processing Systems 11 Network 12 External terminals 20 Information Processing Devices 21 AI Agents 22 Processing Execution Unit 23 Anomaly detection unit 24 Cause estimation part 25 Solution reasoning part 26 Knowledge Accumulation Department 27 Vectorization section 28 Intervention Control Unit 29. Effectiveness Evaluation Department 30 External Business Systems 40 Models Provided 41 Large-scale language models 50 Verification Environment

Claims

1. Multiple AI agents, each corresponding to a different user, A log aggregation unit that aggregates behavior logs related to the actions of the aforementioned multiple AI agents, An analysis unit that analyzes the aforementioned behavioral logs to detect patterns that need improvement, A proposal generation unit that generates improvement suggestions for template setting information applied to at least some of the plurality of AI agents, A verification unit that verifies whether the generated improvement proposal violates predetermined safety conditions, An application unit that reflects the verified improvement proposals in the template setting information and applies the updated template setting information to at least some of the multiple AI agents, An evaluation and rollback unit that evaluates the effectiveness of the improvement proposal based on the action log after application, and rolls back the template settings information to its state before the update if the predetermined improvement is not achieved, An information processing system characterized by comprising the following features.

2. An information processing method performed by multiple AI agents, each corresponding to a different user, The process of aggregating behavioral logs related to the actions of the aforementioned multiple AI agents, The process involves analyzing the aforementioned behavioral logs to detect patterns that need improvement, A step of generating improvement suggestions for template setting information applied to at least some of the aforementioned multiple AI agents, A step of verifying whether the generated improvement proposal violates predetermined safety conditions, The process includes reflecting the verified improvement suggestions in the template configuration information and applying the updated template configuration information to at least some of the multiple AI agents, A step of evaluating the effectiveness of the improvement proposal based on the action log after application, and if the predetermined improvement is not achieved, rolling back the template settings information to the state before the update, An information processing method characterized by including

3. A program for causing a computer to execute the information processing method described in claim 2.

4. In the information processing system described in claim 1, The aforementioned template setting information includes an immutable area that is not to be modified, An information processing system characterized in that the verification unit disapproves an improvement proposal if the improvement proposal includes modifications to the invariant domain.

5. In the information processing system described in claim 1, The application unit applies the updated template setting information to a predetermined canary target among the plurality of AI agents in advance. An information processing system characterized in that the evaluation and rollback unit determines, based on the evaluation results concerning the canary target, whether to apply the updated template setting information to the whole system or to roll it back.

6. In the information processing system described in claim 1, The template configuration information includes at least configuration information regarding response policies and configuration information regarding the use of external tools. An information processing system characterized in that the application unit only implements the improvement proposal if it obtains approval from an administrator or user when the improvement proposal falls under a predetermined high-risk change.

7. In the information processing system described in claim 1, The application unit records the update history of the template setting information as version information. An information processing system characterized in that the evaluation and rollback unit performs the rollback based on the version information.

8. In the information processing system described in claim 1, An information processing system characterized in that the application unit controls the reflection of the improvement suggestion based on speed limit information that limits the amount or number of changes to the template setting information per unit time.

9. In the information processing system described in claim 1, The analysis unit detects seasonal or periodic patterns from the behavior log, The proposal generation unit generates improvement proposals that include temporary setting additions corresponding to the seasonal or periodic patterns. An information processing system characterized in that the application unit assigns expiration date information to the temporary setting addition.

10. In the information processing system described in claim 1, The analysis unit calculates statistical information regarding the frequency of use or failure rate of external tools from the activity log. An information processing system characterized in that the proposal generation unit generates improvement proposals, including enabling, disabling, or modifying the calling procedure of an external tool, based on the statistical information.

11. In the information processing system described in claim 1, The analysis unit detects from the behavior log a history of the user modifying the AI ​​agent's response, An information processing system characterized in that the proposal generation unit generates improvement proposals, including changes to the response policy included in the template setting information, when the common modification trends among multiple users satisfy predetermined conditions.

12. In the information processing system described in claim 1, An information processing system further comprising an audit unit that generates and records an audit log including the verification results by the verification unit, the scope of application, and the results of the rollback.

Images