Information processing apparatus, information processing method, and program

Abstract

Improve the accuracy of answers in Retrieval Augmented Generation (RAG). 【Solution means】The database 20 includes a plurality of sub-databases 200. The model storage unit 21 stores large language models (LLMs) associated with each of the plurality of sub-databases 200. The acquisition unit 40 acquires, from the user terminal used by the user, a selected LLM that is one of the plurality of LLMs, and an input sentence for generating an answer by the selected LLM. The LLM management unit 41 combines the search for information stored in the sub-database 200 associated with the selected LLM to generate an answer to the input sentence for the selected LLM. Each of the plurality of LLMs has a defined field of input sentences accepted for generating answers, and each of the plurality of sub-databases 200 stores information related to the field defined for the corresponding LLM.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program, and particularly relates to a technique for generating an answer to an input using a Large Language Model (LLM).

Background Art

[0002] In recent years, generative AI (Artificial Intelligence), particularly natural language processing (NLP) technology called LLM, has been rapidly developed, and the practical application of automatically generating an answer sentence for an input sentence from a user has entered the stage. As one of such LLM technologies, Retrieval Augmented Generation (RAG) has also come to be utilized (for example, refer to Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] RAG is a mechanism that enables an LLM to search for information such as regulations and business documents accumulated within a specific organization as external information. Since the LLM can extract external information and generate an answer based on it, there is an advantage that answers can be obtained even for the latest information and information not used for learning.

[0005] On the one hand, in RAG, since the LLM generates answers based on external information, the accuracy and reliability of the answers can be affected by the quality of the external information. The inventors of the present application have come to recognize the possibility of improving the accuracy of answers by appropriately managing the external information that the LLM in RAG refers to when generating answers to input sentences from users.

[0006] The present invention has been made in view of these points, and an object thereof is to provide a technique for improving the accuracy of answers in RAG.

Means for Solving the Problems

[0007] A first aspect of the present invention is an information processing apparatus. This information processing apparatus includes a database having a plurality of sub-databases each storing different information, a model storage unit storing large language models associated with each of the plurality of sub-databases, an acquisition unit that acquires, from a user terminal used by a user, a selected LLM that is one of the plurality of LLMs and an input sentence for generating an answer by the selected LLM, an LLM management unit that combines searches for information stored in the sub-database associated with the selected LLM to generate an answer to the input sentence by the selected LLM, and an output unit that outputs the answer to the user terminal. The LLMs associated with each of the plurality of sub-databases are each defined in terms of the field of the input sentence they accept for generating answers, and the plurality of sub-databases each store information related to the field defined for the associated LLM.

[0008] The acquisition unit may further acquire a search range that defines a search target range among the information stored in the sub-database associated with the selected LLM. The information processing apparatus may further include an extraction unit that refers to the sub-database associated with the selected LLM and extracts extraction information that is information related to the input sentence within the range specified by the search range. The LLM management unit may input the input sentence and the extraction information into the selected LLM, and may acquire an answer generated by the selected LLM.

[0009] The information stored in the sub-database may be tagged. The information processing apparatus may further include a query generation unit that analyzes the input sentence to generate a metadata identification query, and a similarity evaluation unit that evaluates the similarity between the metadata identification query and the tag. The output unit may display, in a manner selectable by the user, information with a tag having a high similarity evaluated by the similarity evaluation unit on the user terminal.

[0010] The information stored in the sub-database may be tagged with a tag indicating an access permission. The information processing apparatus may further include a user management unit that manages by associating a plurality of users with attributes indicating the permissions of each user. The LLM management unit may combine searches for information within the range permitted by the user's permissions to cause the selected LLM to generate an answer.

[0011] The acquisition unit may further receive from the user terminal information and a designation of the sub-database storing the information. The LLM management unit may cause the LLM associated with the sub-database to analyze the received information and identify a tag to be attached to the information. The information processing apparatus may further include a database management unit that attaches the identified tag to the received information and stores it in the designated sub-database.

[0012] The information processing apparatus may further include a learning data storage unit that stores combined data obtained by combining the input sentence acquired by the acquisition unit from the user terminal and the search range, and a learning unit that generates a learning model trained to output a corresponding search range when an input sentence is input based on the combined data.

[0013] When the acquisition unit inputs the input sentence acquired from the user terminal into the learning model, the acquisition unit may acquire the output of the learning model as the search range.

[0014] A second aspect of the present invention is an information processing method. This information processing method includes steps in which a processor acquires, from a user terminal used by a user, a selected LLM that is one of a plurality of LLMs each associated with a different sub-database storing different information, and an input sentence for causing the selected LLM to generate an answer, combines searches for information stored in the sub-database associated with the selected LLM to cause the selected LLM to generate an answer to the input sentence, and outputs the answer to the user terminal. The LLMs each associated with the plurality of sub-databases have a defined field of input sentences that each receives to generate an answer, and the plurality of sub-databases each store information related to the defined field of the associated LLM.

[0015] A third aspect of the present invention is a program. This program causes a computer to obtain, from a user terminal used by a user, a selected LLM, which is one LLM selected from a plurality of LLMs each associated with a different sub-database storing different information, and an input sentence for causing the selected LLM to generate an answer, and to realize a function of generating an answer to the input sentence for the selected LLM by combining a search for information stored in the sub-database associated with the selected LLM, and a function of outputting the answer to the user terminal. The LLMs each associated with the plurality of sub-databases are each defined in terms of the field of input sentences accepted for generating answers, and the plurality of sub-databases each store information related to the field defined for the associated LLM.

[0016] To provide this program or to update a part of the program, a computer-readable recording medium recording this program may be provided, or this program may be transmitted via a communication line.

[0017] Note that any combination of the above components, and those obtained by converting the expression of the present invention among a method, an apparatus, a system, a computer program, a data structure, a recording medium, etc. are also effective as aspects of the present invention.

Advantages of the Invention

[0018] According to the present invention, the accuracy of answers in RAG can be improved.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0020] <Overview of the Embodiment> FIG. 1 is a diagram for explaining the outline of the processing executed by an information processing system S according to an embodiment of the present invention. Hereinafter, the outline of the embodiment will be described with reference to FIG. 1.

[0021] The information processing system S according to the embodiment includes an information processing apparatus 1 and user terminals T, and they are connected in a mode that can communicate with each other via a known communication network N such as the Internet or a LAN (Local Area Network). In FIG. 1, an example in which the information processing system S includes two user terminals T, i.e., a first user terminal T1 and a second user terminal T2, is shown, but the number of user terminals T is not limited to two.

[0022] The information processing apparatus 1 is an apparatus for causing an LLM to answer an input sentence acquired from a user via a user terminal T using the RAG technique. For this purpose, the information processing apparatus 1 can access a database 20 that stores external information. FIG. 1 shows an example in the case where the information processing apparatus 1 itself includes the database 20, but the database 20 may be provided by an external storage device different from the information processing apparatus 1.

[0023] The information processing apparatus 1 changes the LLM that generates an answer according to the field to which the content of the input sentence acquired from the user belongs. In other words, the LLM used by the information processing apparatus 1 has a predetermined specialized field for answering respectively. Here, the field to which the content of the input sentence belongs varies depending on the usage scenario of the information processing apparatus 1. For example, when the information processing apparatus 1 is used for business advice in a company, examples of the field include legal affairs, human resources, sales, development, general affairs, etc. Also, when the information processing apparatus 1 is used for searching for documents in an organization that administers local governments, etc., examples of the field include household registers, disaster prevention, insurance, education, finance, sports promotion, etc.

[0024] Since each LLM used by the information processing apparatus 1 according to the embodiment has a predetermined specialized field for answering respectively, the user of the information processing apparatus 1 can use the LLM as if designating experts in each field and asking questions. In the sense that each LLM can assist in the work in its respective specialized field, the LLM may be described as "Assistant A" in this specification. FIG. 1 shows an example in the case where the information processing apparatus 1 uses at least two LLMs (Assistants A), i.e., a first Assistant A1 and a second Assistant A2. Note that the number of Assistants A used by the information processing apparatus 1 is not limited to 2 and may be more than 2.

[0025] As described above, the information processing apparatus 1 causes the assistant A to generate an answer using the RAG technology. Since each assistant A used by the information processing apparatus 1 according to the embodiment has a predetermined specialized field of answering, a dedicated database that each assistant A refers to for searching external information is associated. In FIG. 1, the database 20 accessible by the information processing apparatus 1 includes a first sub-database 200a and a second sub-database 200b, and the first assistant A1 and the second assistant A2 are respectively associated therewith. Thereby, since the assistant A can exclude information outside the specialized field from the search, it is possible to suppress noise from being mixed into the search results.

[0026] In this way, since the information processing apparatus 1 according to the embodiment causes the assistant A to generate an answer by combining the search for information stored in the dedicated sub-database 200 associated with the assistant A, the assistant A can improve the search accuracy of external information, and thus can improve the accuracy of the answer in RAG.

[0027] <Functional Configuration of the Information Processing Apparatus 1 According to the Embodiment> FIG. 2 is a diagram schematically showing the functional configuration of the information processing apparatus 1 according to the embodiment. The information processing apparatus 1 includes a storage unit 2, a communication unit 3, and a control unit 4. In FIG. 2, the arrows indicate the main data flow, and there may be a data flow not shown in FIG. 2. In FIG. 2, each functional block shows a configuration in units of functions, not in units of hardware (devices). Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be divided and implemented in a plurality of devices. The exchange of data between the functional blocks may be performed via any means such as a data bus, a network, a portable storage medium, or the like.

[0028] The storage unit 2 is a large-capacity storage device such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of a computer that realizes the information processing apparatus 1, a RAM (Random Access Memory) that serves as a working area of the information processing apparatus 1, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores various information such as an OS (Operating System), application programs, and a database 20 referred to when the application programs are executed.

[0029] The communication unit 3 is a communication interface for the information processing apparatus 1 to communicate with an external device, and is realized by a known communication module such as a LAN module or a Wi-Fi (registered trademark) module. Hereinafter, in this specification, when the information processing apparatus 1 communicates with an external device, the description of the communication unit 3 may be omitted on the premise that it is via the communication unit 3.

[0030] The control unit 4 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural network Processing Unit) of the information processing apparatus 1, and functions as an acquisition unit 40, an LLM management unit 41, an output unit 42, an extraction unit 43, a query generation unit 44, a similarity evaluation unit 45, a database management unit 46, a user management unit 47, and a learning unit 48 by executing the programs stored in the storage unit 2.

[0031] Note that FIG. 2 shows an example in which the information processing apparatus 1 is configured by a single device. However, the information processing apparatus 1 may be realized by computing resources such as a plurality of processors and memories, such as in a cloud computing system. In this case, each unit constituting the control unit 4 is realized by at least one of a plurality of different processors executing a program.

[0032] The memory unit 2 includes a database 20, a model memory unit 21, an authority database 22, and a learning data memory unit 23. The database 20 includes a plurality of sub-databases 200 that store different information respectively. The model memory unit 21 stores an assistant A, which is a plurality of LLMs, each associated with one of the plurality of sub-databases 200. Details of the authority database 22 and the learning data memory unit 23 will be described later.

[0033] The acquisition unit 40 acquires, from the user terminal T used by the user, a selected assistant A (selected LLM), which is one of the plurality of assistants A, and an input sentence for generating an answer for the selected assistant.

[0034] Figs. 3(a)-(b) are diagrams for explaining the acquisition of the selected assistant and the input sentence by the acquisition unit 40. Specifically, Fig. 3(a) schematically shows an example of a selection screen of the assistant A displayed on the display unit of the user terminal T. Fig. 3(b) schematically shows an example of an input screen of the input sentence displayed on the display unit of the user terminal T. Although not limited, as an example, the information processing apparatus 1 according to the embodiment functions as a web server and provides services by the assistant A to the user terminal T in the form of a web application. When the user accesses the information processing apparatus 1 via the user terminal T, each screen shown in Figs. 3(a)-(b) is displayed on the web browser executed on the user terminal T.

[0035] In the example of the selection screen shown in Fig. 3(a), an assistant A specialized in sales and an assistant A specialized in legal affairs and intellectual property are displayed. The user operates the user terminal T to select an assistant A. In Fig. 3(a), the user has selected an assistant A specialized in sales. The acquisition unit 40 acquires, as the selected assistant, the assistant A specialized in sales selected by the user from the user terminal T via the communication network N.

[0036] When the user selects any one of the assistants A, the screen displayed on the display unit of the user terminal T transitions to the input screen for the input sentence shown in Fig. 3(b). The input screen shown in Fig. 3(b) shows an example when the user inputs "Tell me the summary of the consumer questionnaire results regarding Product A last year." to the assistant A whose specialized field is sales, which is the selected assistant. When the user operates the user terminal T to send the input sentence, the acquisition unit 40 receives and acquires the input sentence from the user terminal T via the communication network N. Although not shown in the figure, the acquisition unit 40 may acquire the input sentence in such a form that the user selects an input sentence prepared in advance in the form of a template.

[0037] The LLM management unit 41 combines the search for information stored in the sub-database 200 associated with the selected assistant to generate an answer to the input sentence for the selected assistant A. In the example shown in Figs. 3(a)-(b), the user selects the assistant A whose specialized field is sales as the selected assistant. In this case, the sub-database 200 associated with the selected assistant stores only information related to sales, such as product information and sales materials of the company where the information processing apparatus 1 is used. In other words, the sub-database 200 associated with the selected assistant does not store information other than information related to sales, for example, information related to legal affairs or finance. Therefore, the assistant A whose specialized field is sales searches only for information associated with its specialized field as external information and does not extract other information. Therefore, it can be expected that the answer generated by the selected assistant will be an accurate answer that combines information in the specialized field of the selected assistant.

[0038] The output unit 42 outputs the answer generated by the selected assistant by the LLM management unit 41 to the user terminal T. In this way, for each of the plurality of sub-databases 200 included in the information processing apparatus 1 according to the embodiment, the field of the input sentence received by each of the plurality of assistants A associated therewith for generating an answer is determined, and the plurality of sub-databases 200 store information related to the field determined for the associated assistant A. Thereby, the information processing apparatus 1 according to the embodiment can improve the accuracy of the answer in RAG.

[0039] In this way, in the information processing apparatus 1 according to the embodiment, according to the field of the input sentence input by the user, an assistant A that specializes in that field is caused to generate an answer. Further, as shown in FIG. 3(a), when the user designates a selected assistant to generate an answer from among the plurality of assistants A, it can also be expected that the user himself / herself has some foresight information regarding the field of the input sentence. Therefore, if there is a mechanism for feeding back the foresight information possessed by the user himself / herself to the selected assistant, it can be expected to contribute to the improvement of the accuracy of the answer. Therefore, the acquisition unit 40 may further acquire a search range that defines the range to be searched among the information stored in the sub-database 200 associated with the selected assistant.

[0040] FIG. 4 is a diagram for explaining the acquisition of the search range by the acquisition unit 40. The sub-database 200 associated with the selected assistant has a hierarchical structure, which generally has a structure also called a folder or a directory (hereinafter, referred to as a "folder" in this specification). The information stored in the sub-database 200 is classified and recorded in one or more folders in advance, and FIG. 4 shows a state in which the names of the folders are listed.

[0041] Figure 4 shows that the sub-database 200 has 11 types of folders from folder A to folder K. In Figure 4, the folders of folder A, folder D, folder F, folder G, and folder H indicate that they are the folders selected by the user. The acquisition unit 40 acquires the folders selected by the user as the search range that the selection assistant determines for the search target among the information stored in the sub-database 200.

[0042] The extraction unit 43 refers to the sub-database 200 associated with the selection assistant and extracts extraction information, which is information related to the input sentence, within the range specified by the search range acquired by the acquisition unit. The LLM management unit 41 inputs the input sentence and the extraction information into the selection assistant and acquires the answer generated by the selection assistant. In this way, by limiting the extraction source of external information to the search range specified by the user, the extraction of information unintended by the user can be suppressed. Thereby, the information processing apparatus 1 can improve the accuracy of the answer in RAG.

[0043] Here, the information stored in the sub-database 200 according to the embodiment is tagged. For example, the information stored in the sub-database 200 being classified and recorded in advance in one or more folders is synonymous with each piece of information being tagged to indicate that it is stored in a specific folder. In addition to this, the information stored in the sub-database 200 is tagged with one or more tags indicating its respective attributes.

[0044] For example, the information stored in the sub-database 200 is tagged with tags such as date and time information indicating the date and time when the information was stored, information indicating the data type such as whether the information is a document or an image, and information indicating the size of the information. As another example, the information stored in the sub-database 200 storing information related to sales is tagged with tags indicating the content of the information, such as whether it is information related to the company's product information, information related to sales materials, or information related to consumer questionnaire results.

[0045] In addition, since the folders in the sub-database 200 are also a type of information, tags can be attached not only to the units of files stored in each folder but also to the folders themselves. FIG. 5 is a diagram schematically showing an example of a user interface for attaching tags to information, and specifically, it is a diagram showing a user interface for attaching tags to folders.

[0046] In the example shown in FIG. 5, "Name", "Region", and "Genre" are examples of tags attached by the user to each folder. For example, a folder with a name tag of "Folder A" has a region tag of "City X" and a genre tag of "Sports" attached. Also, by selecting "Add", the user can add a new tag. These tags can be selected and set by the user from among a plurality of pre-prepared tags, or can also be set by the user directly inputting them. Also, the user can set two or more tags for one piece of information (a folder in the example of FIG. 5). Note that the user interface shown in FIG. 5 is displayed on the user terminal T by the output unit 42 under the control of the database management unit 46. The database management unit 46 acquires the information input into the user interface displayed on the user terminal T as tags.

[0047] Returning to the description of FIG. 4. FIG. 4 shows an example where the user manually designates the search range, but it is convenient if the folders to be designated can be prioritized using some index based on the tags attached to the information stored in the sub-database 200. Therefore, the information processing apparatus 1 includes a query generation unit 44 and a similarity evaluation unit 45. Specifically, first, the query generation unit 44 analyzes the input sentence acquired by the acquisition unit 40 to generate a metadata identification query. The similarity evaluation unit 45 evaluates the similarity between the metadata identification query generated by the query generation unit 44 and the tags attached to the information.

[0048] FIG. 6 is a diagram for explaining an example of query generation by the query generation unit 44 and similarity evaluation by the similarity evaluation unit 45. FIG. 6 assumes that the information processing apparatus 1 is used in an administrative agency such as a city hall, and shows an example when the acquisition unit 40 acquires a sentence "Please show me some materials regarding the autumn citizen sports festival." as the input sentence I from the user. In this example, the query generation unit 44 generates character strings such as "autumn", "fall", "sports", "exercise", "materials", "poster", etc. as the metadata identification query Q by analyzing the input sentence I. The query generation unit 44 may be realized using known techniques such as morphological analysis and thesaurus, or may generate the metadata identification query Q using an LLM. In the latter case, the query generation unit 44 may instruct the LLM management unit 41 to generate the metadata identification query Q by the assistant A.

[0049] The similarity evaluation unit 45 vectorizes the metadata identification query Q generated by the query generation unit 44 using a known embedding technique in NLP. Similarly, the similarity evaluation unit 45 also vectorizes the tag L attached to the information. FIG. 6 shows an example when the similarity evaluation unit 45 converts the metadata identification query Q and the tag L into vectors V of a high dimension (for example, 1024 dimensions) respectively.

[0050] The similarity evaluation unit 45 evaluates the similarity between the vectorized metadata identification query Q and the vectorized tag L using a method for measuring a known vector similarity such as cosine similarity. The example shown in FIG. 6 shows an example when the similarity evaluation unit 45 determines that the tag L of "marathon competition" is similar to the metadata identification query Q of "sports". Similarly, the similarity evaluation unit 45 also shows that it has determined that the tag L of "Tanabata festival" is not similar to any of the plurality of metadata identification queries Q. Thereby, the query generation unit 44 and the similarity evaluation unit 45 can identify the tag L similar to the metadata identification query Q extracted from the input sentence I among the tags L attached to the information stored in the sub-database 200.

[0051] The output unit 42 displays, in a manner that allows the user to select, the information with tags having a high similarity evaluated by the similarity evaluation unit 45 on the user terminal T. FIG. 7 is a diagram for explaining the acquisition of a search range based on the similarity evaluated by the similarity evaluation unit 45. Specifically, FIG. 7 shows an example in which a plurality of folders included in the sub-database 200 are sorted in descending order of the amount of information with a high similarity evaluated by the similarity evaluation unit 45 and displayed in a manner that allows the user to select. Thereby, the information processing apparatus 1 can make it easier for the user to select a folder that stores a large amount of highly probable information related to the input sentence I input by the user from among the plurality of folders.

[0052] Here, the information stored in the sub-database 200 is not necessarily information that anyone can freely view. For example, documents containing trade secrets are highly confidential, and in some cases, it may be preferable to limit the users who are permitted to view them. Therefore, the information stored in the sub-database 200 is tagged with a tag indicating the permission to access. The user management unit 47 manages by associating a plurality of users with attributes indicating the authority of each user.

[0053] FIG. 8 is a diagram schematically showing the data structure of the authority database 22 referred to by the user management unit 47. The authority database 22 is stored in the storage unit 2 and is managed by the user management unit 47. In the example of the authority database 22 shown in FIG. 8, the access authority for each sub-database 200 is stored in association with a user identifier for uniquely identifying a user. For example, it is shown that the user with the user identifier UID0001 has the access authority levels of 1 for the first sub-database 200a, the second sub-database 200b, and the Xth sub-database 200x. The same applies to other user identifiers. In the example shown in FIG. 8, it is shown that the higher the number of the level indicating the access authority, the more information access is permitted.

[0054] The LLM management unit 41 combines the search for information within the scope permitted by the user's access rights and selects the LLM to generate an answer. As a result, the information processing apparatus 1 can refer to external information within the scope of the access rights permitted to the user and present an answer to the user.

[0055] The above has described how the information processing apparatus 1 refers to the external information stored in the sub-database 200 to generate an answer for Assistant A. Subsequently, the processing executed by the information processing apparatus 1 when the user stores external information in the sub-database 200 will be described.

[0056] When the user attempts to newly store information in the sub-database 200, the user is highly likely to understand the content of that information. If the user understands the content of the information, the user will also have knowledge about the field of the content of that information. In such a case, it is preferable to utilize the user's knowledge in selecting the sub-database 200 in which the information should be stored. Therefore, the acquisition unit 40 receives from the user terminal T the information newly stored by the user and the designation of the sub-database 200 in which the information is to be stored.

[0057] The LLM management unit 41 causes Assistant A associated with the sub-database 200 to analyze the received information and identify the tags to be attached to the information. The database management unit 46 attaches the tags identified by Assistant A to the information received by the acquisition unit 40 and stores it in the sub-database 200 designated by the user. As a result, the information processing apparatus 1 can automatically assign tags to the information newly stored in the sub-database 200 without bothering the user by utilizing the user's knowledge regarding the content of the information.

[0058] Here, depending on the information newly stored by the user, there are cases where it is preferable to determine the period during which the information is valid. For example, when storing information regarding the provisions of a law for which a legal amendment is planned in the sub-database 200 related to legal regulations, the implementation date becomes the start date of the valid period, and if the repeal date is determined, that date becomes the end date of the valid period. Therefore, the database management unit 46 causes the output unit 42 to display a user interface for setting the valid period of the information on the user terminal T.

[0059] FIG. 9 is a diagram schematically showing an example of a user interface for setting the valid period of information. Specifically, FIG. 9 shows an example of a user interface for a user to set the start and end of the valid period of document information regarding a contract document for a business consignment contract with XX Company. By the user operating the user interface and checking the check boxes for setting the start and end of the valid period respectively, the start and end dates can be set.

[0060] When a valid period is set for the information, the extraction unit 43 extracts the extraction information on the condition that, in addition to the range specified in the search range acquired by the acquisition unit 40, the date at the time of extraction is information included in the valid period of the information. Thereby, the information processing apparatus 1 can improve the accuracy of the answer in the RAG.

[0061] Also, the database management unit 46 may delete the information whose valid period has ended from the sub-database 200. Thereby, since the unnecessary information is deleted from the sub-database 200, the capacity of the sub-database 200 can be effectively utilized.

[0062] As described with reference to FIGS. 4 and 7, the user who inputs the input sentence I also manually specifies the search range. That is, the information processing apparatus 1 acquires combination data that is data combining the input sentence I by the user and the search range at that time. By accumulating this combination data in the information processing apparatus 1, it can be used for analysis of statistical information regarding the input sentence I and the ease of selection of the corresponding search range.

[0063] Therefore, the learning data storage unit 23 stores and accumulates the combination data combining the input sentence and the search range acquired by the acquisition unit 40 from the user terminal T. Based on the combination data stored in the learning data storage unit 23, the learning unit 48 generates a learning model that is learned using a known machine learning technique so as to output a corresponding search range when an input sentence is input. The learning unit 48 stores the generated learning model in the model storage unit 21.

[0064] When the learning model is generated by the learning unit 48, the acquisition unit 40 acquires the output of the learning model as the search range when the input sentence acquired from the user terminal T is input to the learning model. As a result, the information processing apparatus 1 can automatically specify the search range or a candidate thereof only by the user inputting the input sentence I.

[0065] <Processing flow of the information processing method executed by the information processing apparatus 1> FIG. 10 is a flowchart for explaining the flow of information processing executed by the information processing apparatus 1 according to the embodiment. The processing in this flowchart starts, for example, when the information processing apparatus 1 is activated.

[0066] The acquisition unit 40 acquires a selected assistant, which is one LLM selected from a plurality of LLMs, from the user terminal T used by the user (S2). Further, the acquisition unit 40 also acquires the input sentence I for which the selected assistant is to generate an answer from the user terminal T (S4).

[0067] The LLM management unit 41 combines the search for information stored in the sub-database 200 associated with the selected assistant and instructs the selected assistant to generate an answer to the input sentence I (S6). The output unit 42 outputs the answer generated by the selected assistant to the user terminal T (S8). When the output unit 42 outputs the answer to the user terminal T, the processing in this flowchart ends.

[0068] <Effects achieved by the information processing apparatus 1 according to the embodiment> As described above, according to the information processing apparatus 1 according to the embodiment, the accuracy of the answer in RAG can be improved.

[0069] As described above, the present invention has been described using embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist. For example, all or part of the device can be configured by functionally or physically dispersing and integrating it in any unit. Also, new embodiments resulting from any combination of multiple embodiments are included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination have the effects of the original embodiments combined. Such modification examples will be described below.

[0070] <First modification example> Above, the case where the user manually specifies the search range, the case where the search range is determined based on the similarity evaluated by the similarity evaluation unit 45, and the case where the learning model outputs the search range have been described. The method for obtaining the search range is not limited to these, and other methods may be used. For example, the LLM associated with each sub-database 200 may identify the search range by having a conversation with the user to narrow down the search range.

[0071] <Second modification example> In the above description, the case where one LLM is associated with one sub-database 200 has been mainly described. However, two or more LLMs may be associated with one sub-database 200. For example, in order for an LLM specializing in labor matters to generate an answer, it may be necessary to access legal information. In preparation for such a case, in addition to the LLM specializing in legal matters, an LLM specializing in labor matters may be associated with the sub-database 200 storing legal information. Thereby, it can be expected that the LLM specializing in labor matters will generate a more accurate answer.

[0072] <Third Modification Example> In the above description, the case where the similarity evaluation unit 45 evaluates the similarity between the vectorized metadata identification query Q and the vectorized tag L using a method for measuring a known vector similarity such as cosine similarity has been mainly described. Instead of this, the similarity evaluation unit 45 may identify the similarity between the metadata identification query Q and the tag L by evaluating whether they exactly match or partially match. In this case, the similarity evaluation unit 45 may evaluate that the metadata identification query Q and the tag L are similar when they exactly match or partially match, and dissimilar otherwise.

Explanation of Reference Numerals

[0073] 1 ··· Information Processing Apparatus 2 ··· Storage Unit 20 ··· Database 200 ··· Sub-Database 21 ··· Model Storage Unit 22 ··· Authority Database 23 ··· Learning Data Storage Unit 3 ··· Communication Unit 4 ··· Control Unit 40 ··· Acquisition Unit 41 ··· Management Unit 42 ··· Output Unit 43 ··· Extraction Unit 44 ··· Query Generation Unit 45 ··· Similarity Evaluation Unit 46 ··· Database Management Unit 47 ··· User Management Department 48 ··· Learning Department N ··· Communication Network S ··· Information Processing System T ··· User Terminal

Claims

1. A database including a plurality of sub-databases each storing different information, a model storage unit storing a plurality of large language models (LLMs) each associated with one of the plurality of sub-databases, an acquisition unit that acquires, from a user terminal used by a user, a selected LLM that is one of the plurality of LLMs and an input sentence that is input by the user and causes the selected LLM to generate an answer, an LLM management unit that generates an answer to the input sentence for the selected LLM by combining searches for information stored in the sub-database associated with the selected LLM, an output unit that outputs the answer to the user terminal, and a user management unit that manages by associating a plurality of users with attributes indicating the authority of each user, wherein the plurality of LLMs each associated with one of the plurality of sub-databases have a defined field of input sentences that each receives to generate an answer, wherein the plurality of sub-databases each store information related to the field defined for the associated LLM and tagged with a tag indicating the authority to determine access permission, wherein one of the LLMs specialized in the field is associated with one of the sub-databases storing information related to the field, and wherein the LLM management unit generates an answer for the selected LLM by combining searches for information within the range permitted to be accessed by the authority of the user, an information processing apparatus.

2. The acquisition unit further acquires a search range that defines a search target range among the information stored in the sub-database associated with the selected LLM, and the information processing apparatus further includes an extraction unit that refers to the sub-database associated with the selected LLM and extracts extraction information that is information related to the input sentence within the range specified by the search range, wherein the LLM management unit inputs the input sentence and the extraction information to the selected LLM and acquires the answer generated by the selected LLM, The information processing apparatus according to claim 1.

3. The information stored in the sub-database is tagged, and the information processing apparatus further includes a query generation unit that analyzes the input sentence to generate a metadata identification query, and a similarity evaluation unit that evaluates the similarity between the metadata identification query and the tag. The output unit displays, in a manner that allows the user to select, information with tags having a high similarity evaluated by the similarity evaluation unit on the user terminal. The information processing apparatus according to claim 1 or 2.

4. The acquisition unit further receives, from the user terminal, information and a designation of a sub-database that stores the information. The LLM management unit causes the LLM associated with the sub-database to analyze the received information and identify tags to be attached to the information. The information processing apparatus further includes a database management unit that attaches the identified tags to the received information and stores the information in the designated sub-database. The information processing apparatus according to claim 3.

5. The information processing apparatus includes a learning data storage unit that stores combined data obtained by combining an input sentence acquired by the acquisition unit from the user terminal and a search range, and a learning unit that generates a learning model trained to output a corresponding search range when an input sentence is input, based on the combined data. The information processing apparatus according to claim 2.

6. The acquisition unit acquires, as a search range, the output of the learning model when the input sentence acquired from the user terminal is input to the learning model. The information processing apparatus according to claim 5.

7. When an expiration date is set for the information, the extraction unit extracts the extraction information on the condition that, in addition to the range specified by the search range acquired by the acquisition unit, the information whose extraction date is included in the expiration date of the information. The information processing apparatus according to claim 2.

8. A processor performs steps of: acquiring, from a user terminal used by a user, a selected LLM that is one of a plurality of LLMs each associated with a different sub-database storing different information, and an input sentence input by the user that causes the selected LLM to generate an answer; combining searches for information stored in the sub-database associated with the selected LLM to cause the selected LLM to generate an answer to the input sentence; and outputting the answer to the user terminal. For each of the plurality of LLMs associated with each of the plurality of sub-databases, the field of the input sentence received for generating an answer is defined. The plurality of sub-databases each store information related to the field defined by the associated LLM, and the information is tagged with a tag indicating the authority to determine the accessibility. One of the sub-databases storing information related to the field is associated with one of the LLMs specialized in the field. In the step of generating the answer, the selected LLM is caused to generate an answer by combining searches for information within the range permitted to be accessed by the authority associated with the user. Information processing method.

9. On a computer, a function of obtaining, from a user terminal used by a user, a selected LLM that is one of a plurality of LLMs each associated with a plurality of sub-databases storing different information, and an input sentence for causing the selected LLM to generate an answer, the input sentence being the input sentence input by the user; a function of causing the selected LLM to generate an answer to the input sentence by combining searches for information stored in the sub-database associated with the selected LLM; a function of outputting the answer to the user terminal; The plurality of LLMs each associated with the plurality of sub-databases are each defined with the field of the input sentence that they accept to generate an answer. The plurality of sub-databases each store information related to the field defined by the associated LLM, and the information is tagged with a tag indicating the authority to determine the accessibility. One of the sub-databases storing information related to the field is associated with one of the LLMs specialized in the field. The function of generating the answer causes the selected LLM to generate an answer by combining searches for information within the range permitted to be accessed by the authority associated with the user. Program.

Images