Information processing apparatus, information processing method, and program

Abstract

To generate an appropriate prompt for obtaining an answer desired by a user who is not familiar with a generation AI.SOLUTION: An information processing device according to an aspect of the present disclosure includes a processor configured to, when input information is input, determine an answer policy based on the input information, search the input information in a knowledge database based on the answer policy, and generate a prompt based on the input information and a search answer of the knowledge database, and an output device configured to output the prompt to a generated AI.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to so-called Retrieval Augmented Generation (RAG), which combines retrieval and generative AI (Artificial Intelligence). [Background technology]

[0002] Generally, answers to anticipated questions, such as Q&A, are stored in a knowledge database (DB). To obtain the answer a user wants, they must select a question from the knowledge DB that is likely to contain the answer, or perform a text search on the knowledge DB. If the user does not select an appropriate question or uses incorrect terms in the search, the user will not be able to obtain the answer they want. If the user does not obtain the answer they want, they may change the terms and repeat the search, or select a different question. However, since there is no guarantee that the answer they want is stored in the knowledge DB, the user may waste their effort.

[0003] Recently, information processing devices have been proposed that utilize easily usable application programming interfaces (APIs) based on large language models (LLMs).

[0004] LLM is a generative AI model specialized in language processing. When a user inputs a string of characters called a prompt, it generates an answer based on the prompt and outputs it to the user. To obtain the desired answer from LLM, it is necessary to generate an appropriate prompt. Therefore, technology is needed to generate appropriate prompts to obtain the desired answer. Furthermore, because LLM generates answers based on various information available on the internet, it may generate answers based on incorrect information, or it may generate an incorrect answer even if based on correct information.

[0005] A technology has been proposed that extracts keywords from a question, searches a database to obtain candidate documents, and generates prompts to be input into an LLM based on the question and candidate documents (e.g., Patent Documents 1 and 2).

[0006] A technology has been proposed in which a knowledge database is searched for a question entered from a terminal to obtain question-related information, and the question and question-related information are embedded in a prompt and sent to the LLM (for example, Patent Document 3). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 7325152 [Patent Document 2] Patent No. 7313757 [Patent Document 3] Patent No. 7396582 Summary of the Invention [Problem to be solved by the invention]

[0008] Creating Q&As takes time, and maintaining them is also costly, as all related content must be updated when products or prices change.

[0009] Generating prompts to input to the AI ​​generator is not easy. If the input prompt is inappropriate, the AI ​​generator may give an answer that does not meet the user's expectations. It is also difficult for users who are not familiar with the AI ​​generator to generate appropriate prompts to obtain the answer they want. [Means for solving the problem]

[0010] An information processing device in one aspect of the present disclosure includes a processor that, when input information is input, determines a response policy based on the input information, searches a knowledge database for the input information based on the response policy, and generates a prompt based on the input information and the search response from the knowledge database, and an output device that outputs the prompt to be input to a generation AI. [Effects of the Invention]

[0011] According to the present disclosure, an information processing device can generate appropriate prompts to be input to a generation AI. In particular, because the information processing device generates prompts by searching a knowledge DB based on a response policy based on input information, the risk of the generation AI providing incorrect information is reduced when information unrelated to the knowledge DB is input. [Brief explanation of the drawings]

[0012] [Figure 1] Overall system configuration diagram [Figure 2] FIG. 1 shows a configuration of an information processing device. [Figure 3] Diagram showing the structure of the knowledge database [Figure 4] FIG. 1 is a diagram showing a processing flow executed by an information processing device. [Figure 5] FIG. 10 is a diagram showing an example of a prompt generated by an information processing device. [Figure 6] A diagram showing an example of a user response [Figure 7] A diagram showing an example of a response when a user question is unrelated to the information stored in the knowledge database. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate.

[0014] (Embodiment 1) FIG. 1 shows the overall configuration of the system 100.

[0015] The system 100 includes an input / output terminal 110, an information processing device 120, a knowledge database (DB) 130, and an LLM 140.

[0016] The input / output terminal 110 is a device that receives a user question from a user and outputs a user answer to the user question. The user question may be in a form of sentences used daily (natural sentences).

[0017] The information processing device 120 is a device that generates prompts to be input to the LLM based on questions about a specific domain input by the user. When the information processing device 120 receives information input by the user from the input / output terminal 110, it searches the knowledge DB 130 for the information input by the user and obtains the search results.

[0018] The knowledge DB 130 is a storage device such as a hard disk drive (HDD) or a fixed memory (SSD: Solid State Drive).

[0019] The LLM 140 is a generative AI model specialized for language processing. The LLM 140 receives a prompt generated by the information processing device 120 and generates answer content in response to the input prompt. The LLM 140 can use any LLM, such as ChatGPT by OpenAI or tsuzumi by NTT (Nippon Telegraph and Telephone Corporation). The LLM 140 may be configured in the cloud or on a terminal. Because tsuzumi has a small parameter size, when tsuzumi is used, the LLM 140 can be configured on a terminal.

[0020] FIG. 2 shows the configuration of the information processing device 120.

[0021] The information processing device 120 includes a processor 121, an input device 122, and an output device 123. In addition to the devices shown in FIG. 2, the information processing device 120 may include a communication device, a memory, and a storage device. The devices such as the processor 121 and the input device 122 are connected by a bus 124 for communicating information. The bus 124 may be configured using a single bus, or may be configured using different buses between the devices.

[0022] The processor 121 is composed of a computer including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 121 receives a user question from the input / output terminal 110 via the input device 122 and analyzes the received user question. The processor 121 determines an answer policy based on the analyzed user question. The processor 121 determines whether to search the knowledge DB 130 based on the answer policy, and searches the knowledge DB for the user question. The processor 121 generates a prompt to be input to the LLM 140 based on the search answer obtained from the knowledge DB 130 and the answer policy. Furthermore, when information generated by the LLM 140 is input from the LLM 140, the processor 121 generates a user answer based on the search answer input from the knowledge DB 130 and the information generated by the LLM 140, and outputs the user answer to the input / output terminal 110.

[0023] The input device 122 may be an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) through which a user question is input from outside, or may be an interface or communication device through which a user question is input from the input / output terminal 110. The input device 122 may be connected to the knowledge DB 130 and the LLM 140.

[0024] The output device 123 may be an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs the user's answer to the outside, or may be an interface or communication device that transmits the user's answer to the input / output terminal 110. The output device 123 may be connected to the knowledge DB 130 and the LLM 140.

[0025] The input device 122 and the output device 123 may be integrated (for example, a touch panel), or the input device 122 and / or the output device 123 may be a communication device.

[0026] The memory is a computer-readable recording medium and may be configured by, for example, at least one of a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), a RAM (Random Access Memory), etc. The memory may also be called a register, a cache, a main memory (primary storage device), etc. The memory can store executable programs (program codes), software modules, etc. for implementing a wireless communication method according to an embodiment of the present disclosure.

[0027] The storage is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blue-ray® disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy disk, a magnetic strip, etc. The storage may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including at least one of a memory and a storage.

[0028] A communication device is hardware (input / output device) for communicating between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0029] The information processing device 120 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 121 may be implemented using at least one of these pieces of hardware.

[0030] FIG. 3 shows the configuration of the knowledge DB 130.

[0031] The knowledge DB 130 is a database that stores information about specific Internet domains and services (commercial products). For example, the database may contain HTML documents acquired by an administrator from a web server that is the same as the organization, regular text, or PDF (Portable Document Format) documents. A domain is a collection of information about a specific service (commercial product) or a collection of HTML documents belonging to an Internet domain. The knowledge DB may be installed on a terminal, a network, or a cloud. The knowledge DB 130 may store information about multiple domains. Each of the multiple knowledge DBs 130 may store information about its respective domain.

[0032] For example, knowledge DB 130 is a DB that stores official information about irumo, and is managed by NTT Docomo, Inc., which provides irumo. Knowledge DB 130 may also store other information, such as information about "ahamo" and "eximo." For example, knowledge DB 130 may be composed of three DBs: DB 131 (hereinafter referred to as "irumoDB") that stores information about "irumo," DB 132 (hereinafter referred to as "ahamoDB") that stores information about "ahamo," and DB 133 (hereinafter referred to as "eximoDB") that stores information about "eximo." Knowledge DB 130 does not have to include three DBs.

[0033] The knowledge DB 130 stores document content such as web document files in html format and portable document format (pdf) files. The knowledge DB 130 may store information that has been duplicated by a crawler crawling the Internet, or may store information that is suitable for a search by performing pre-processing and / or duplicate deletion on the crawled information to remove character strings unnecessary for the search. The knowledge DB 130 may store an index for vector search and / or an index for keyword search.

[0034] (operation) FIG. 4 shows a processing flow executed by the information processing device 120.

[0035] The operation of the system 100 will be explained based on a specific example, for example, a case where a user wants to know the price of irumo, provided by NTT DoCoMo, Inc., and inputs "Tell me the price of irumo" into the input / output terminal 110.

[0036] The information processing device 120 determines whether a user question, i.e., information entered by a user into the input / output terminal 110, is input from the input / output terminal 110 (step S401). If a user question is not input (step S401, No), the information processing device 120 returns to step S401 and waits for input of a user question.

[0037] When a user question is input (step S401, Yes), the information processing device 120 analyzes the input user question and determines whether the user question is a question about information stored in the knowledge DB 130 (step S402). If the knowledge DB 130 is made up of multiple DBs, the information processing device 120 may determine which DB is the most appropriate DB. For example, the information processing device 120 determines whether the input user question is a question about "irumo," a question about "ahamo," a question about "eximo," or a question unrelated to any of these.

[0038] The information processing device 120 determines an answering policy depending on, for example, whether the input question is specific information stored in the knowledge DB 130. The information processing device 120 analyzes the input user question and determines that it is a question about "irumo." The information processing device 120 determines whether the recognized question is a question related to information stored in the knowledge DB 130. In this example, the input information is determined to be a question related to the irumoDB.

[0039] The information processing device 120 may determine that the user question is related to multiple DBs. For example, if the input user question is "Compare the prices of irumo and ahamo," the information processing device 120 determines that the irumoDB and ahamoDB are related DBs.

[0040] Furthermore, the information processing device 120 determines the answer policy of the answer content to be generated by the LLM 140 based on the result of the determination in step S402 (step S403).

[0041] If the input information is a question related to one of the databases that make up the knowledge database 130, the answering policy is, for example, "Answer in Japanese, and if you don't know, please don't try to answer, just say you don't know."

[0042] There may be cases where a user inputs a question unrelated to the knowledge DB 130. In this case, a question unrelated to the knowledge DB 130 is input to the information processing device 120. For example, information such as "teach me how to make pork miso soup" is input to the information processing device 120. If the input information is a question about information not stored in the knowledge DB 130, the answering policy is, for example, "cannot answer."

[0043] In other words, the system 100 is a system that answers questions related to the knowledge DB 130, but does not answer questions unrelated to the knowledge DB 130.

[0044] The response policy may include information indicating how many search results should be obtained, i.e., the amount of search results. The amount of search results may be determined based on information set in the LLM 140 to which the information processing device 120 queries, or the amount of search results may be determined depending on the DB to be searched.

[0045] The response policy may include information indicating the conditions for obtaining search results.

[0046] The information processing device 120 determines whether to search the knowledge DB 130 based on the answering policy (step S404). The information processing device 120 determines to search the knowledge DB 130 if the answering policy is "answer in Japanese, and if you don't know, please say you don't know without trying to answer," but not to search the knowledge DB 130 if the answering policy is "cannot answer."

[0047] If it is determined in step S403 that the knowledge DB 130 will not be searched (step S404, No), the information processing device 120 does not search the knowledge DB 130 and executes the process of step S408.

[0048] If the information processing device 120 determines in step S403 to search the knowledge DB 130 (step S404, Yes), it searches the knowledge DB 130 for the input "What is the price of irumo?" (step S405). The information processing device 120 can obtain search answers from the knowledge DB 130, such as "This is a price plan that combines a voice plan, ISP service, and data communication in one package." and "For information on the data capacity for the month of change, please see 'If I change my irumo data capacity, when will it take effect?'"

[0049] The information processing device 120 selects information to be searched in accordance with the determination made in step S402.

[0050] For example, when information about the irumo and ahamo domains is related to each other, the information processing device 120 may search both pieces of information. For example, when the information processing device 120 determines that the irumoDB is a DB related to the user's question, the information processing device 120 may search the irumoDB and the ahamoDB, and when the information processing device 120 determines that the eximoDB is a DB related to the user's question, the information processing device 120 may search the eximoDB and the ahamoDB. Even when the information processing device 120 determines that multiple DBs are related to the user's question, the information processing device 120 may determine to search some of the multiple related DBs. For example, the information processing device 120 may determine the DB to search based on a feedback history for answers to past inquiries made by the user. Feedback is provided by pressing a button such as "helpful," and the information processing device 120 may determine the DB to search based on the number of times the button was pressed. The information processing device 120 may determine the DB to search based on the number of times the button was pressed (for example, up to the top three) or the number of times the button was pressed more than a predetermined number. The information processing device 120 may determine that a DB is to be searched if it determines that the user question and the DB are related to each other to a predetermined degree or higher, or may determine that the DB is to be searched if it determines ... related to the user question to a degree higher than a predetermined degree or higher, for example, the top three DBs in order of the degree of relevance to the user question.

[0051] The search may be a vector search that uses the similarity of vectors, a keyword search, a hybrid search that combines vector search and keyword search, or other search methods such as a semantic search that uses a machine learning model that reflects the user's search intent or additional information on the vector being searched.

[0052] The information processing device 120 receives search answers from the knowledge DB 130. The search answers may include relevance information for the user question along with the search results.

[0053] The information processing device 120 generates a prompt to be output to the LLM 140 based on the search answer input from the knowledge DB 130 and the answer policy (step S406). When multiple search answers are input, the information processing device 120 may determine whether to use each search answer to generate a prompt based on feedback. The information processing device 120 may determine whether to use each search answer to generate a prompt based on the number of times the "helpful" button is pressed. The information processing device 120 may determine to use a search answer to generate a prompt if the number of times the button is pressed is equal to or greater than a predetermined number, or may determine to use a search answer to generate a prompt if the button is pressed frequently. The information processing device may generate a prompt based on highly relevant search answers.

[0054] The information processing device 120 outputs the generated prompt to be input to the LLM 140.

[0055] 5 shows an example of a prompt 500 generated by the information processing device 120. The prompt 500 includes a search answer 510 from the knowledge DB, an answer policy 520, and a question 530 input from the input / output terminal 110.

[0056] The information processing device 120 acquires information generated by the LLM 140 (hereinafter referred to as "LLM response") from the LLM 140 (step S407).

[0057] The information processing device 120 outputs an answer to the user question input from the input / output terminal 110, i.e., the user answer (step S408). The information processing device outputs the user answer, for example, as a Web document file in HTML format. If the information processing device 120 determines in step S403 to search the knowledge DB 130, it outputs a Web document file containing the LLM answer acquired in step S407 as the answer, and if it determines in step S404 not to search the knowledge DB 130, it outputs a Web document file containing the "answer policy" determined in step S403 as the answer.

[0058] An example of the user answer is shown in Fig. 6. The LLM answer may be used as the user answer, or the information processing device 120 may generate the user answer based on the LLM answer.

[0059] 6 shows an example answer 600 when it is determined in step S402 that the user question is related to information stored in knowledge DB 130. Example answer 600 includes domain information 610, a question 620, answer information 630, and an input section 640. The dotted lines in example display 600 are added for illustrative purposes only and are not actually displayed.

[0060] The domain information 610 indicates the domain information stored in the knowledge DB 130 and may include products or trademarks (service marks) related to the searched knowledge DB 130. If multiple DBs are searched, multiple pieces of domain information may be displayed.

[0061] The question 620 may include characters and / or a pictogram (icon) 621 indicating that it is a question, and a user question 622 input from the input / output terminal 110.

[0062] The answer information 630 may include characters and / or pictograms (icons) 631 indicating that it is an answer, first information 632, and second information 633.

[0063] Both the first information 632 and the second information 633 may be generated by the LLM 140. The first information 632 is an LLM answer generated by the LLM 140, and the second information 633 may indicate resource information of a document stored in the knowledge DB referenced when generating the first information 632, such as the location of a web server (URL: Uniform Resource Locator). The first information 632 is information generated by the LLM 140, and the second information 633 may be a URL of resource information of a document in which information returned in response to the search in step S404 is stored. The second information 633 may be a predetermined number of URLs returned in order of highest relevance when information indicating relevance is returned together with the URL in step S405. Banner information may be displayed as the second information 633 instead of the URL, and the web page of the URL may be displayed when the user clicks on the banner information.

[0064] The user can input a new question in the input section 640. The input section 640 may include a button for the user to provide feedback. For example, the input section 640 may include a "helpful" button.

[0065] 7 shows an example answer 700 when the user's question is unrelated to the information stored in knowledge DB 130 (step S402). Example answer 700 includes domain information 610, question 720, answer information 730, and input section 640. The dotted lines in example display 700 are added for illustrative purposes only and are not actually displayed.

[0066] Question 720 still includes characters and / or pictogram (icon) 621 indicating that it is a question and question 722 entered from input / output terminal 110, but the content of question 722 entered from input / output terminal 110 differs from the content of question 622 in example answer 600 shown in Figure 6.

[0067] The answer information 730 may include characters and / or pictograms (icons) 631 indicating that it is an answer, and an answer policy 732.

[0068] 6 and 7 differ only in the content of the user question and the content of the answer, but the domain information 610, the characters and / or pictogram (icon) 621 indicating that it is a question, the characters and / or pictogram (icon) 631 indicating that it is an answer, and the input section 640 are the same.

[0069] <Variation 1> The information processing device 120 may include helpfulness information for the search answer in the prompt, which may be information determined based on the number of times the "helpful" button was pressed or the number of times the button was pressed.

[0070] The prompt may include search results sorted in descending order of helpfulness information, or only a predetermined number of search results with the highest helpfulness information may be included in the prompt.

[0071] By inputting a prompt containing usefulness information into LLM140, LLM140 can be instructed to determine how useful each search result is when using multiple search results, thereby increasing the likelihood of generating an answer that is useful to the user.

[0072] <Variation 2> The information processing device 120 may generate a prompt including a URL or banner information. The URL or banner information may be generated based on helpfulness information. The helpfulness information may be determined based on the number of clicks on the URL or banner.

[0073] <Variation 3> The information processing device 120 may determine whether to refer to the most recent search results and LLM answer based on the relevance between the current user question and the previous user question. The information processing device 120 may obtain the previous LLM answer from the history and use the result of comparing the previous LLM answer with the current user question to search the knowledge DB. The comparison result may be a determination of whether the direction of the LLM answer based on vector evaluation was correct. The information processing device 120 may use the determination result to search the knowledge DB 130.

[0074] For example, consider a case where information about irumo is organized into two databases: a database for non-irumo subscribers and a database for irumo subscribers. The information processing device 120 may initially search the database for non-irumo subscribers, and then switch to the database for irumo subscribers if the current user question determines that the user is a subscriber.

[0075] For example, the information processing device 120 may have performed a search on the irumoDB because the previous user question was "Tell me about irumo," but if the current user question is "I would like to know what happens when there is a little more data," it may perform a search on the ahamoDB this time.

[0076] <Variation 4> The usefulness information may be stored in correspondence with each piece of information stored in the knowledge DB, or in correspondence with search results.

[0077] By storing the usefulness information in the knowledge DB, it becomes easier to know which knowledge DB contains information useful to the user.

[0078] A button other than "helpful" may be displayed in the user's answer. For example, "helpful" and "not helpful" buttons may be displayed. Information for inputting whether or not the answer was helpful may be included. For example, toggle buttons for inputting "helpful" or "not helpful" may be displayed as input unit 640 in FIG. 6.

[0079] In the above-described embodiments, the notation of each component may be replaced with other notations such as "...circuitry," "...assembly," "...device," "...unit," or "...module."

[0080] Furthermore, the present disclosure can be realized by software, hardware, or software linked to hardware. Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by a single LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of a single chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be called an IC, system LSI, super LSI, or ultra LSI.

[0081] The integrated circuit method is not limited to LSI, but may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, a field programmable gate array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI, may be used. The present disclosure may be realized as digital processing or analog processing.

[0082] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology can be used to integrate functional blocks. The application of biotechnology, etc. is also a possibility. [Industrial Applicability]

[0083] The present disclosure is useful for an information processing device that responds to a user's input. [Explanation of symbols]

[0084] 100 systems 110 Input / Output Terminal 120 Information processing equipment 121 processors 122 Input Device 123 Output Device 124 Bus 130 Knowledge DB 131 irumoDB 132 ahamoDB 133 eximoDB 140 LLM

Claims

1. a processor that, when input information is input, determines a response policy based on the input information, searches a knowledge database for the input information based on the response policy, and generates a prompt based on the input information and the search response from the knowledge database; an output device that outputs the prompt to be input to the generated AI; An information processing device comprising:

2. The knowledge database is composed of a plurality of databases, the processor determines a database to search based on the input information; The information processing device according to claim 1 .

3. the processor determines the response policy based on whether the input information is information related to the knowledge database. The information processing device according to claim 1 .

4. The processor further outputs output information based on an answer corresponding to the prompt when the answer is input from the generation AI. The information processing device according to claim 1 .

5. The information processing device When input information is input, a response policy is determined based on the input information, Searching a knowledge database for the input information based on the response policy; generating a prompt based on the input information and the knowledge database search response; outputting the prompt to the generating AI; Information processing methods.

6. On the computer, When input information is input, a response policy is determined based on the input information; searching a knowledge database for the input information based on the response policy; generating a prompt based on the input information and the knowledge database search response; causing a generating AI to output the prompt; A program for performing information processing.

Images