Information processing system, information processing method, and program

Abstract

To provide a mechanism for selecting an appropriate AI module from a plurality of AI modules.SOLUTION: A first inputter that inputs, to a plurality of AI modules each capable of executing an action for an inputted request, a question for determining an AI module to be in charge of the acquired request, an acquirer that acquires answers to the question from the plurality of AI modules, a determiner that determines an AI module to be in charge of the acquired request from among the plurality of AI modules based on the acquired answers, and a second inputter that inputs the acquired request to the determined AI module and causes the execution module to execute the action for the request.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Non-Patent Document 1 is a document disclosing background technology in this technical field. Non-Patent Document 1 states, "ChatHub is an innovative browser extension that allows you to use multiple chatbots on a single platform. This tool supports well-known chatbots such as ChatGPT, Bing, Bard, and Claude, and provides users with a diverse conversation experience." It should be noted that ChatGPT, Bing, Bard, and Claude mentioned in Non-Patent Document 1 are all registered trademarks. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] @kerimeka, “[Generative AI] Multiple AI execution · Introducing ChatHub: Trends in new technologies”, [online], December 23, 2023, Qiita Co., Ltd., [Retrieved July 8, 2024], Internet<URL:https: / / qiita.com / kerimeka / items / 0e99701c840eed6656c0> Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, various AIs have been developed, each with its own strengths and weaknesses in processing. Therefore, there is a demand for technology that enables processing to be performed using the appropriate AI from among multiple AIs. The above-mentioned Non-Patent Document 1 does not mention such technology. The present invention has been made in view of the above circumstances, and provides a mechanism for selecting an appropriate AI module from among a plurality of AI modules. [Means for solving the problem]

[0005] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above-mentioned problem, and one example is an information processing system comprising: an acquisition means for acquiring a request; a first input means for inputting a question to multiple AI modules each capable of performing an action on the input request to determine which AI module will be responsible for the acquired request; an acquisition means for acquiring answers to the question from the multiple AI modules; a determination means for determining which AI module from the multiple AI modules will be responsible for the acquired request based on the acquired answer; and a second input means for inputting the acquired request to the determined AI module and causing it to perform an action on the request. [Effects of the Invention]

[0006] According to the present invention, a mechanism for selecting an appropriate AI module from among a plurality of AI modules is provided. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 shows an example of the configuration of an information processing system 100. As shown in FIG. [Figure 2] FIG. 2 shows an example of the configuration of the business support server 101. [Figure 3] FIG. 3 shows an example of the configuration of the LLM server 102. [Figure 4] FIG. 4 shows an example of the configuration of the user terminal 103. [Figure 5] FIG. 5 illustrates an example of a question answering process 500 . [Figure 6] FIG. 6 shows an example of a question answering process 500 . [Figure 7] FIG. 7 shows an example of the data flow in the question answering process 500. DETAILED DESCRIPTION OF THE INVENTION

[0008] 1. Example Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0009] 1-1.Configuration FIG. 1 shows an example of the configuration of an information processing system 100 according to this embodiment. The information processing system 100 includes a business support server 101, an LLM server 102, and multiple user terminals 103. The devices that make up this system are connected via a wired or wireless network and are capable of sending and receiving information to and from each other.

[0010] Of the devices that make up this system, the business support server 101 is a server that receives queries from users and selects the most suitable LLM module for the received queries. The LLM server 102 includes a plurality of LLM modules, each responsible for a different field, and each LLM module generates and outputs a response to a received query. The plurality of user terminals 103 are terminal devices that transmit queries input by users to the business support server 101 and receive responses to the transmitted queries.

[0011] Each of these devices includes a processor that executes an operating system, applications, programs, etc., a main storage device such as RAM (Random Access Memory), an auxiliary storage device such as an IC card, hard disk drive, SSD (Solid State Drive), flash memory, etc., a communication control unit such as a network card, wireless communication module, mobile communication module, etc., an input device such as a touch panel, keyboard, mouse, voice input, or input based on motion detection captured by a camera unit, and an output device such as a monitor, display, etc. The output device may also be a device or terminal that transmits information to be output to an external monitor, display, printer, device, etc.

[0012] The main memory stores various programs and applications (modules), and the processor executes these programs and applications to realize the various functional elements of the overall system. Each module may be implemented in hardware, such as by integration. Each module may be an independent program or application, or may be implemented as a subprogram or function within a single integrated program or application.

[0013] In this specification, each module is described as the entity (subject) that performs the processing, but in reality, the processing is carried out by a processor that processes various programs, applications, etc. (modules).

[0014] Various databases (DBs) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a set of data so that it can accommodate any data manipulation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. There are no limitations on how the database is implemented; for example, it can be a database management system, spreadsheet software, or a text file such as XML or JSON.

[0015] 1-1-1. Business support server 101 FIG. 2 shows an example of the configuration of the business support server 101. The business support server 101 is configured, for example, by one or more servers arranged on a cloud. The server's main memory device 201 stores programs and applications such as a conductor module 210 and a butler module 211. These programs and applications are executed by the processor 203 to realize the various functional elements of the business support server 101. Each module will be described below.

[0016] The conductor module 210 is an interface for exchanging data with the user. Specifically, this module receives requests from users and passes the received requests to the butler module 211. The requests received from users include answers to questions and requests for actions.

[0017] The answer to the question is, for example, an answer to a question about application procedures. One example is an answer to a question such as "How do I apply for expenses?" On the other hand, an action request is, for example, a request to create a sentence or document. An example is a request such as "Create data for an expense claim."

[0018] The conductor module 210 also inputs a request received from a user to an LLM module 310 selected by a selector LLM module 312 (described later). This causes the LLM module 310 to execute an action for the request. The module then returns the execution result of the action to the user. After that, the module stores the user request and the execution result of the action in the query cache 220 in association with each other.

[0019] The butler module 211 is a module that determines which LLM module 310 is to be in charge of a user request. Specifically, when the module receives a request from the conductor module 210, it checks whether a similar request is stored in the query cache 220. If the module finds a request with a similarity equal to or greater than a predetermined value, the module sends the action result corresponding to the found request to the conductor module 210. The conductor module 210 returns the action result to the user as a response.

[0020] That is, when the butler module 211 acquires another request that has a similarity equal to or greater than a predetermined value compared to the acquired request, it outputs the result of the action taken on the acquired request as a response. This response output does not use the LLM module 310. This reduces the processing load on the system.

[0021] The similarity is, for example, an index indicating to what extent the distributed representations (vectors) of the requests are oriented in the same direction, and more specifically, is cosine similarity.

[0022] If no similar requests are found as a result of the above check, the butler module 211 inputs a question to each LLM module 310. The input question is used to determine which LLM module 310 will be in charge of the acquired request. The butler module 211 obtains answers to the input questions from each LLM module 310. The obtained answers include a confidence level indicating the degree of suitability to handle the obtained request and a reason. The confidence level may be expressed numerically or as a binary value of "YES" or "NO." Meanwhile, the reason is information indicating the basis for calculating the confidence level. The butler module 211 sends the obtained answer to the merger module 311 .

[0023] Next, the auxiliary storage device 202 will be described. The auxiliary storage device 202 includes the above-mentioned query cache 220. The query cache 220 stores user requests and action execution results in association with each other.

[0024] 1-1-2.LLM Server 102 FIG. 3 shows an example of the configuration of the LLM server 102. The LLM server 102 is configured, for example, by one or more servers arranged on a cloud. The server's main memory device 301 stores programs and applications such as a plurality of LLM modules 310, a merger module 311, and a selector LLM module 312. These programs and applications are executed by the processor 303 to realize the various functional elements of the LLM server 102. Each module will be described below.

[0025] Each of the multiple LLM modules 310 is an AI module that can execute an action in response to an input request. The number of LLM modules 310 is arbitrary. Each LLM module 310 is assigned a specific area of ​​responsibility and processes requests related to that area. The areas that each module is responsible for include, for example, "expense claims" and "inquiries."

[0026] In this embodiment, the LLM (i.e., large-scale language model) is a deep learning model that has been trained in advance using a large-scale dataset. In other words, the LLM is a generative AI (or generative AI). Since LLM is a deep learning model that has been trained in advance using a large dataset, it can be used without training data or additional training.

[0027] The merger module 311 is an LLM module that formats the answers of the LLM module 310. Specifically, this module first obtains the answers obtained from each LLM module 310 from the butler module 211. The obtained answers include a confidence level and reason indicating the degree to which the answer is suitable for handling the user's request. The merger module 311 stores each obtained answer in the answer DB 320 in association with the request ID.

[0028] The module also deletes the reason from each acquired answer and generates data (in this embodiment, JSON data as an example) that describes only the confidence level. The module then passes the generated JSON data to the selector LLM module 312. The reason for removing the reason from each response here is to prevent hallucination of the selector LLM module 312.

[0029] The selector LLM module 312 is a module that determines, from among a plurality of LLM modules 310, an LLM module 310 that will be in charge of a user request. Specifically, the module first obtains JSON data from the merger module 311. The confidence level of each LLM module 310 is described in this JSON data. Then, based on the acquired JSON data, the module determines which LLM module 310 will be in charge of the user request from among the multiple LLM modules 310. At that time, the module determines the LLM module 310 with the highest confidence as the responsible LLM. The selector LLM module 312 stores the information of the determined LLM module 310 in the response DB 320 in association with the request ID.

[0030] If the confidence level described in the JSON data satisfies a predetermined condition, the selector LLM module 312 requests additional information to be referenced for determining the LLM module 310. The predetermined condition here is, for example, when the confidence levels of the multiple LLM modules 310 exceed 50. Another example is when the difference between the highest confidence level and the second highest confidence level is equal to or less than a predetermined value. The above value "50" is merely an example and may be set to any value.

[0031] When a predetermined condition is met, the selector LLM module 312 may, for example, ask the user which of the LLM modules 310 to select. Alternatively, the module may ask the user questions to further clarify the user's query. These questions are sent to the user via the butler module 211 and the conductor module 210. When the selector LLM module 312 receives the user's response to the inquiry, it determines the LLM module 310 that will be in charge of the user's request based on the received information. This improves the accuracy of estimating the appropriate LLM module 310.

[0032] Next, the auxiliary storage device 302 will be described. The auxiliary storage device 302 stores the answer DB 320. This answer DB 320 is a database in which answers (including confidence levels and reasons) from each LLM module 310 are associated with information indicating the ultimately determined responsible LLM module 310 for each request. The information in the response DB 320 is used to improve the performance of each LLM module 310. For example, the information is used to improve the expertise of each LLM module 310. The information is also used to improve the accuracy of the confidence levels output by each LLM module 310.

[0033] 1-1-3. User terminal 103 FIG. 4 shows an example of the configuration of the user terminal 103. The user terminal 103 is, for example, a terminal device such as a smartphone, a tablet terminal, a notebook PC, or a desktop PC. The terminal's main memory device 401 stores programs and applications such as a browser module 410. The processor 403 executes these programs and applications to realize the various functional elements of the user terminal 103.

[0034] The browser module 410 is a module for exchanging information with the conductor module 210 .

[0035] 1-2.Operation Next, the question answering process 500 executed in the information processing system 100 will be described with reference to Fig. 5 to Fig. 7. Fig. 5 and Fig. 6 show an example of the question answering process 500, and Fig. 7 shows an example of the data flow in the process.

[0036] 5 to 7, for ease of explanation, only two LLM modules 310A and 310B are shown as examples of multiple LLM modules 310. LLM module 310A is an LLM module responsible for "expense claims," ​​and LLM module 310B is an LLM module responsible for "inquiries."

[0037] First, the conductor module 210 receives a query from a user (step 501), and then passes the received query to the butler module 211 (step 502). When the butler module 211 receives a query, it checks whether a similar query is stored in the query cache 220 (step 503). If the butler module 211 finds a request with a similarity equal to or greater than a predetermined value, the butler module sends the action result corresponding to the found request to the conductor module 210 (not shown). The conductor module 210 returns the action result to the user as a response (not shown).

[0038] On the other hand, if no similar requests are found as a result of the above check, the butler module 211 generates a prompt for each LLM module 310 (step 504). The generated prompt (in other words, a command statement) includes the following information: (1) Query (2) A command to output a confidence level and reason indicating the degree to which the system is suitable for handling the query. (3) Information indicating the area of ​​responsibility (4) One or more example queries related to your field

[0039] Of these pieces of information, information (3) and (4) differ for each LLM module 310. The prompt for LLM module 310A includes information indicating that the module's area of ​​responsibility is "expense claim" and one or more example queries related to "expense claim." On the other hand, the prompt for LLM module 310B includes information indicating that the module's area of ​​responsibility is "inquiry" and one or more example queries related to "inquiry." Based on this information, each LLM module 310 outputs a confidence level and reason indicating the degree to which it is suitable to handle the query.

[0040] Once the butler module 211 generates the prompt, it inputs it to each LLM module 310 (step 505). Each LLM module 310 outputs an answer to the input prompt (step 506), which includes a confidence level and reasoning that indicates the degree to which it is suitable to handle the input query.

[0041] When the butler module 211 receives the answers from each LLM module 310, it then generates a prompt for the merger module 311 (step 507). The generated prompt includes the following information: (1) Responses from each LLM module 310 (including confidence level and reasons) (2) A statement instructing that the response from each LLM module 310 be stored in the response DB 320 in association with the request ID. (3) A command to remove the reasons from each answer and generate JSON data containing only the confidence level for each LLM module 310. (4) A statement instructing the selector LLM module 312 to generate and input a prompt containing the following information: (a) A statement instructing the system to determine one LLM module 310 based on the generated JSON data. (b) A statement instructing the user to select one of the LLM modules 310 when the confidence level of the LLM modules 310 exceeds "50" and to select one of the LLM modules 310 based on the user's response. (c) a statement instructing the Butler module 211 to pass category information indicating the determined LLM module 310; (d) A statement instructing that category information indicating the determined LLM module 310 be stored in the response DB 320 in association with the request ID.

[0042] Once the butler module 211 generates the prompt, it inputs it to the merger module 311 (step 508). Upon receiving the input prompt, the merger module 311 stores the responses from each LLM module 310 in the response DB 320 in association with the request ID (step 509). The module also generates a prompt (step 510) and inputs it to the selector LLM module 312 (step 511). The prompt generated here includes the above information (a) to (d).

[0043] The selector LLM module 312 receives the input prompt and determines one LLM module 310 based on the JSON data (step 512). At that time, the module selects the LLM module 310 with the higher confidence level. The module then passes category information indicating the determined LLM module 310 to the butler module 211 (step 513). The module also stores category information indicating the determined LLM module 310 in the response DB 320 in association with the request ID (step 514).

[0044] If the confidence levels of multiple LLM modules 310 exceed 50, the module asks the user which one to select (not shown). This inquiry is sent to the user via the butler module 211 and the conductor module 210. Then, upon receiving the user's response to the inquiry, the module selects one of the LLM modules 310 (not shown) based on the received information.

[0045] When the butler module 211 acquires the category information from the selector LLM module 312, it passes the acquired information to the conductor module 210 (step 515). The conductor module 210 inputs the query received from the user in step 501 to the LLM module 310 indicated by the acquired category information. If the acquired category information indicates LLM module 310A, the module inputs the query to LLM module 310A (step 516). The module then obtains an answer to the query from LLM module 310A (step 517) and returns it to the user (step 518). This concludes the explanation of the question answering process 500.

[0046] According to the question answering process 500 described above, it is possible to select an LLM module 310 that is appropriate for answering a query from among a plurality of LLM modules 310. Furthermore, by utilizing the cache information stored in the query cache 220, the processing load related to question answering can be reduced. Additionally, hallucination of the selector LLM module 312 can be prevented by formatting the responses from each LLM module 310 (i.e., by removing the reason). Furthermore, by requesting additional information from the user when multiple confidence levels exceed "50," the accuracy of estimating the appropriate LLM module 310 is improved.

[0047] 2. Variations The above embodiment may be modified as follows: The following modifications may be combined with each other.

[0048] 2-1. Answers from LLM Module 310 In the above embodiment, the prompt input from the butler module 211 to the LLM module 310 includes a statement instructing it to output a confidence level and a reason. However, outputting a reason is not necessarily required. The LLM module 310 may be instructed to output only a confidence level, without outputting a reason. Even in this case, the selector LLM module 312 can select the LLM module 310 to which the query is input based on the confidence level.

[0049] 2-2. Shaping means In the above embodiment, the merger module 311 formats the answer of the LLM module 310. However, this formatting is not necessary. The merger module 311 may be omitted, and the selector LLM module 312 may select the LLM module 310 based on the unformatted answer. In this case, the answer still includes a confidence level, so the selector LLM module 312 can still select the LLM module 310.

[0050] 2-3. Instructions to the selector LLM module 312 In the above embodiment, a prompt is input from the merger module 311 to the selector LLM module 312. However, this prompt may also be input by the butler module 211. In this case, the butler module 211 obtains JSON data from the merger module 311, generates a prompt including the above information (a) to (d), and inputs it to the selector LLM module 312. This method also allows the selector LLM module 312 to select an LLM module 310.

[0051] 2-4.AI Module In the above embodiment, the LLM module 310, i.e., the generative AI, is used. However, a conventional AI may be used instead of the generative AI. In this case, the AI ​​module replacing the LLM module 310 may have, for example, a confidence estimation model and an answer estimation model.

[0052] The confidence estimation model is a trained machine learning model that uses a query as an explanatory variable and confidence as a target variable. This model receives a query input from the butler module 211 and outputs a confidence corresponding to the query. The selector LLM module 312 selects an AI module to use based on this confidence.

[0053] On the other hand, the answer estimation model is a trained machine learning model that uses the query as an explanatory variable and the answer as a target variable. When the AI ​​module is selected by the selector LLM module 312, this model receives a query input from the conductor module 210 and outputs an answer corresponding to the query. The conductor module 210 returns the output answer to the user. Question answering processing can be realized by using the AI ​​modules described above.

[0054] 2-5.Functional arrangement In the above embodiment, the question answering process 500 is executed by two servers, the business support server 101 and the LLM server 102. However, this functional arrangement is merely an example. The functions of each server may be arranged as appropriate depending on the environment in which the question answering service is provided. For example, some or all of the functions of the LLM server 102 may be provided in the business support server 101.

[0055] 2-6. System components Each device constituting the information processing system 100 may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or may be a wearable terminal such as glasses, a wristwatch, or clothing. Each device may also be a stationary or portable computer, or a server located on the cloud or a network. Each device may also function as a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, a combination of multiple of these terminals may be used. For example, a combination of one smartphone and one wearable terminal may logically function as a single terminal. Other information processing terminals may also be used.

[0056] 2-7.Other The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0057] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0058] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. The above-described embodiments disclose at least the configurations described in the claims. [Explanation of symbols]

[0059] 100...information processing system, 101...business support server, 102...LLM server, 103...user terminal, 210...conductor module, 211...butler module, 220...query cache, 310...LLM module, 311...merger module, 312...selector LLM module, 320...answer DB

Claims

1. A means for obtaining a request; a first input means for inputting a question to a plurality of AI modules each capable of executing an action for the input request, in order to determine an AI module to be assigned to take charge of the acquired request; an acquisition means for acquiring an answer to the question from the plurality of AI modules; A determination means for determining an AI module to be in charge of the acquired request from among the plurality of AI modules based on the acquired answer; a second input means for inputting the acquired request to the determined AI module and causing the AI ​​module to execute an action for the request; An information processing system comprising:

2. The obtained answer includes a confidence level and a reason indicating the degree to which the obtained answer is suitable for handling the request; Further, a formatting means for deleting a reason from the obtained answer is provided, the determining means determines an AI module to be in charge of the acquired request from among the plurality of AI modules based on the answer from which the reason has been deleted. The information processing system according to claim 1 .

3. a requesting means for requesting and acquiring additional information to be referenced in order to determine an AI module to be in charge of the acquired request when the acquired answer satisfies a predetermined condition; The determination means determines an AI module to be in charge of the acquired request from among the plurality of AI modules based on the acquired answer and the acquired additional information. The information processing system according to claim 1 .

4. The acquiring means further includes an output means for outputting a result of the action as a response when the acquiring means acquires another request having a similarity of a predetermined value or more compared with the acquired request. The information processing system according to claim 1 .

5. Each of the plurality of AI modules is a generative AI module. The information processing system according to claim 1 .

6. 1. A computer-implemented information processing method, comprising: obtaining a request; inputting a question to a plurality of AI modules each capable of performing an action on the input request in order to determine an AI module to be assigned to handle the acquired request; obtaining an answer to the question from the plurality of AI modules; determining an AI module to be in charge of the acquired request from among the plurality of AI modules based on the acquired answer; inputting the acquired request to the determined AI module and causing the AI ​​module to execute an action on the request; An information processing method including:

7. On the computer, obtaining a request; inputting a question to a plurality of AI modules each capable of performing an action on the input request in order to determine an AI module to be assigned to handle the acquired request; obtaining an answer to the question from the plurality of AI modules; determining an AI module to be in charge of the acquired request from among the plurality of AI modules based on the acquired answer; inputting the acquired request to the determined AI module and causing the AI ​​module to execute an action on the request; A program to execute.

Images