Information processing apparatus, information processing method, and information processing program

Abstract

To eliminate anxiety about security in an answering system.SOLUTION: An information processing apparatus comprises: a reception unit 21 which receives a query which a user uses an input device 11 to input; a search unit 23 which makes a search related to one or more documents stored in a document database 31 on the basis of the received query and acquires one or more parts highly relevant to the query; a transmission unit 25 which transmits the acquired parts to a server device via a communication device 13; a reception unit 26 which receives an answer sentence generated by the server device using the transmitted parts, from the server device via the communication device 13; and a display unit 27 which displays the received answer sentence on a display device 12 to present an answer to the query to the user.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Semantic search of documents using LLMs (Large Language Models) such as ChatGPT is known, and for example, chatbots are being developed that use LLMs to answer questions about products. The response system consists of a user device, such as a smartphone that runs an application incorporating a chatbot, and a server device that provides the LLM. However, because LLM cannot generate answers using data that it has not learned, the chatbot had the problem of being unable to answer questions about product specifications that it did not know, for example. To solve such problems, the product specification document to be answered is given to the LLM as context. For example, in response to a question (query) such as "My air conditioner is making strange noises, what should I do?", the server searches the database based on the query, and generates text such as "This air conditioner is model A and has specification B. If C, please do D," which is then concatenated and queried to the LLM. In this way, the LLM can be supplemented and the LLM can be made to answer questions about unlearned knowledge. LlamaIndex (https: / / www.llamaindex.ai) is known as an open source answering system that complements LLM in this way. LlamaIndex generates context that is highly relevant to the query from knowledge sources such as PDF files, and sends a prompt consisting of the query and context to the LLM (ChatGPT), enabling questions and answers to be asked about unique data. Although not an LLM, a configuration that complements processing by a server device that provides the functionality of a machine learning model as a cloud service has been well known in the past (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 255803 [Non-patent literature]

[0004] [Non-Patent Document 1] How LlamaIndex v0.7 works and how to customize it [online], July 12, 2023, [searched September 15, 2023], Internet<URL:https: / / note.com / npaka / n / n27a36f784fb3> Summary of the Invention [Problem to be solved by the invention]

[0005] The mechanism disclosed in Non-Patent Document 1 makes it possible to create an answering system that uses unique data that has not yet been learned by the LLM. However, the data needs to be sent to a cloud (external server) to be given to the LLM. Therefore, if the data to be sent is classified information or data that should be kept secret, there are security concerns. One aspect of the present invention is to eliminate security concerns in a reply system. [Means for solving the problem]

[0006] As one aspect, the present invention provides an information processing device that obtains an answer to a query by processing a document and a query using a large-scale language model provided by a server device, the information processing device comprising: a reception unit that receives a query input by a user using an input device; a search unit that performs a search of one or more documents stored in a document database that is private to the server device based on the received query, and obtains one or more pieces of content that are highly relevant to the query as search results; a transmission unit that transmits the search results to the server device via a communication device; and a receiving unit that receives an answer sentence generated by the server device using the transmitted search results from the server device via the communication device. Effect of the Invention

[0007] According to one aspect of the present invention, it is possible to eliminate security concerns in response systems. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of a reply system. [Diagram 2] FIG. 2 is a diagram showing a schematic configuration of a user device. [Diagram 3] FIG. 2 is a diagram illustrating a schematic configuration of an administrator device. [Figure 4] FIG. 2 is a diagram illustrating data managed by a user device. [Diagram 5] FIG. 2 is a diagram illustrating a procedure for a query and an answer on a user device. [Figure 6] 13 is a flowchart illustrating a response process performed by a user device. [Figure 7] FIG. 11 is a diagram (part 1) for explaining the confirmation process executed by the user device. [Figure 8] FIG. 2 is a diagram (part 2) for explaining the confirmation process executed by the user device. [Figure 9] 10 is a flowchart illustrating a confirmation process executed by an administrator device. [Figure 10] FIG. 1 is a block diagram illustrating an embodiment of a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a schematic diagram illustrating the configuration of a response system according to this embodiment. The reply system 1 includes a user device (information processing device) 10 used by a user, and a server device 60 that provides a cloud service. The user device 10 is connected to a local network NW, such as an in-house network, and an administrator device 40 used by a network administrator or a department manager is connected to the in-house network. Here, the local network NW will be described as the in-house network NW. The user device 10 is connected to the Internet via an internal company network NW, and the server device 60 is open to the Internet. The user device 10 and the server device 60 can communicate with each other via the in-house network NW and the Internet. The user device 10 and the administrator device 40 are, for example, a PC (Personal Computer) or a smartphone. The user device 10 includes a document database (DB) 31 and a vector database (DB) 32. As described later, the document DB 31 stores one or more documents as knowledge sources, and the vector DB 32 stores the documents in the document DB 31 in a vectorized form.

[0010] The document DB 31 and the vector DB 32 may be stored in, for example, a file server or a database server connected to the in-house network NW. Note that embedding, which will be described later, is a feature vector obtained by converting text, and it is difficult to restore the original text from the vector. Therefore, since there are few security concerns, the vector data DB 32 may be provided in the server device 60 or other external storage. In this embodiment, the user device 10 is described as having a vector DB 32 in its internal storage.

[0011] The server device 60 provides a large scale language model (LLM) as a cloud service. The server device 60 generates an answer in natural language using LLM to the question received from the user by the user device 10, and transmits the answer to the user device 10. The answer generated by the server device 60 is presented to the user of the user device 10. The user device 10 may be a terminal device dedicated to inputting questions and displaying answers to the answer system 1. Alternatively, the user device 10 may execute an arbitrary application, receive questions from a user about how to use the application, and display the answers.

[0012] LLM cannot generate answers using data that it has not learned. However, by providing LLM with the document that is the source of the answer as context, LLM can provide answers about matters that it has not learned. For example, even if an LLM has not yet learned the specifications of a certain product, it can still answer questions about it by being given the product specification document.

[0013] As described below, the response system 1 of this embodiment is capable of presenting an answer to a user even when the content for which an answer is sought is related to a confidential document and the confidential document cannot be learned by the LLM, an external service. When the user device 10 receives a question (query) from a user, it performs embedding (described later) on the query and calculates a feature vector of the query. Then, the user device 10 compares the feature vector of the query with the feature vectors of the documents stored in the vector DB 32, and acquires the IDs of documents that are highly relevant (have a high degree of similarity) to the calculated query. The user device 10 acquires a portion of the document (a chunk, which will be described later) corresponding to the acquired ID from the document DB 31. Then, the user device 10 transmits to the server device 60 a prompt including the acquired document and a query. The server device 60 generates a response sentence based on the transmitted prompt and transmits it to the user device 10 . The user device 10 displays the transmitted answer on a display device of the user device 10, thereby presenting the answer to the user. Alternatively, the user device 10 may output the answer by reading it out from a speaker provided in the user device 10.

[0014] FIG. 2 is a diagram showing a schematic configuration of a user device according to this embodiment. The user device 10 includes an input device 11, a display device 12, a communication device 13, a processing unit 20, and a storage unit 30. The input device 11 is a keyboard or a touch panel, and is used by the user to input a question to the user device 10, etc. The display device 12 is a liquid crystal display device or an organic EL display device, and displays a display screen 100 described below, which includes an input field for a question, search results, a confirmation dialog, and an answer field for displaying an answer received by the user device 10 from the server device 60. The communication device 13 connects the user device 10 to a network. The processing unit 20 executes each process in the user device 10 . The storage unit 30 stores various information and data for the processing unit 20 to perform processing. For example, the storage unit 30 stores a document DB 31 and a vector DB 32. The storage unit 30 can also store a confirmation-free information database (DB) 33 that registers chunks, documents, and folders that do not require confirmation of whether they can be transmitted.

[0015] The processing unit 20 includes a receiving unit 21 , a converting unit 22 , a searching unit 23 , a confirming unit 24 , a transmitting unit 25 , a receiving unit 26 , and a display unit 27 . The receiving unit 21 performs a process of receiving a query (question sentence) input by a user using the input device 11. The conversion unit 22 performs a process of embedding the query received by the reception unit 21 to convert it into a feature vector (vectorization). The conversion unit 22 also performs a process of embedding a document stored in the document DB 31 to convert it into a feature vector, and stores the converted feature vector in the vector DB 32. The search unit 23 performs a process of searching the vector DB 32 for documents (chunks) that are highly relevant to the query, using the query vectorized by the conversion unit 22 . The confirmation unit 24 performs a process of confirming whether or not the documents (chunks) of the search results of the search unit 23 can be transmitted to the server device 60. The confirmation unit 24 may also perform a process of transmitting the search results to the administrator device 40 via the communication device 13, and receiving from the administrator device 40 the search results that have been determined by the administrator device 40 to be transmittable. The transmission unit 25 performs a process of transmitting the search results and the query that have been determined to be transmittable to the server device 60 via the communication device 13 . The receiving unit 26 performs a process of receiving, from the server device 60 via the communication device 13, a reply message generated by the server device 60 using the transmitted search result. The display unit 27 performs a process of displaying the above-mentioned display screen 100 on the display device 12 .

[0016] FIG. 3 is a diagram showing a schematic configuration of the administrator device according to this embodiment. The administrator device 40 includes an input device 41 , a display device 42 , a communication device 43 , a processing unit 50 , and a storage unit 30 . The input device 41 is a keyboard or a touch panel, and is used by the user to input a question to the administrator device 40, etc. The display device 42 is a liquid crystal display device or an organic EL display device, and displays a display screen 100 including the search results received from the user device 10 and a confirmation dialogue. The communication device 43 connects the administrator device 40 to the network. The processing unit 50 executes each process in the administrator device 40 .

[0017] The processing unit 50 includes a reception unit 51, a receiving unit 52, a confirmation unit 53, a transmission unit 54, and a display unit 55. The reception unit 51 performs a process of receiving an input operation to the administrator device 40 performed by the administrator using the input device 41 . The receiving unit 52 performs a process of receiving documents (chunks) of the search results from the user device 10 via the communication device 13 . The confirmation unit 53 performs a process of confirming whether or not the document (chunk) of the search result received from the user device 10 can be transmitted to the cloud (server device 60). The transmission unit 54 performs processing to transmit the search results and the query that have been determined to be transmittable to the user device 10 via the communication device 13 . The display unit 55 performs a process of displaying the above-mentioned display screen 100 on the display device 42 .

[0018] This embodiment will now be described in detail. The user device 10 transmits a document to complement the LLM for unlearned content. However, as an example, only a part of a document that is the source of the answer sentence is transmitted to the server device 60, rather than the entire document. One reason for transmitting only a part of the document to the server device 60 is that the number of tokens that the LLM can handle at one time has an upper limit, such as 8092, and all of the text contained in the document cannot be provided to the LLM. A token is a word that is generated by dividing text using morphological analysis. For example, the text "The weather is good today" is divided into the tokens "Today," "is," "Weather," and "Good." In many natural language processing models, including LLM, text is divided into units called tokens before being input to a machine learning model.

[0019] Another reason for transmitting a part of the document to the server device 60 is the problem of information leakage caused by supplying the entire document. By inputting the entire document to the LLM, the LLM may be able to provide a more detailed answer, but there is a concern that the LLM may learn the entire document, including parts that are not directly used to answer the query. Even if learning is not performed, there is a certain amount of risk associated with the network path to the server device 60 and the act of transmitting data to an external server. Therefore, in the answering system 1 of this embodiment, the document is divided into chunks each consisting of, for example, 1024 tokens, and the chunk that is closest to the query content is queried to the LLM. If there are chunks similar to the query content across multiple documents, the chunks with the most similar content in each document can be collected and queried to the LLM. This avoids the problem of the number of tokens that can be entered into the LLM, and also prevents confidential information from leaking outside the company. For example, when searching 100GB of in-house data, only the chunk of about 8KB closest to the query is sent to the LLM. This dramatically reduces the risk of information leakage and makes the system more secure.

[0020] However, even if an entire document is sent to the server device 60, it is much more secure than storing the entire document DB 31 in the server device 60, and it is possible to have the LLM generate an answer using the entire document. Therefore, although the problem of the number of tokens that can be input to the LLM still exists, it is acceptable to send an entire document to the server device 60, taking into account the risks and benefits.

[0021] In this embodiment, for example, embedding is used to search for chunks or documents that are closest to the query. Embedding itself is a known technique that expresses the meaning of sentences or words as real number vectors in order to handle them on a computer. Embedding, which expresses the meaning of a sentence with a real vector, is sometimes called "sentence embedding". Sentence embedding is used to calculate the semantic similarity between sentences. By calculating the similarity between vectors obtained by sentence embedding, the semantic similarity between sentences can be measured, and document search that takes into account the meaning of the sentence can be performed using this similarity. Sentence embedding and similarity calculation themselves are well-known technologies, so a detailed explanation will be omitted. As an example, the semantic similarity between sentences can be measured numerically by calculating the cosine similarity from the inner product of the feature vector of a query and the feature vector of a document (chunk). "Sentence embedding" is performed by applying a known AI model to a document, either chunk by chunk or the entire document, to convert the document into vector data (feature vectors) that clearly indicate its meaning.

[0022] FIG. 4 is a diagram illustrating data held by the user device. As described above, the user device 10 performs embedding for each chunk of all confidential documents stored in the document DB 31, and calculates feature vectors. The user device 10 includes a vector DB 32, and stores the calculated feature vectors in the vector DB 32. The user device 10 stores confidential documents in the document DB 31. When it is assumed that a query is made to the LLM of the external server device 60 from the in-house network NW using the user device 10, the confidential documents are, for example, in-house documents. The document DB 31 may not be provided in the user device 10 itself, but may be stored in, for example, a file server or a database server connected to the in-house network NW, or in an external storage that is not open to the server device 60. Here, the user device 10 will be described as having the document DB 31 in its internal storage.

[0023] 4, documents each assigned an ID are stored in the document DB 31. The documents themselves may be PDF documents or Word documents, and text extracted from them is stored in the document DB 31. Since the number of tokens that the LLM can handle at one time has an upper limit, such as 8092, the document DB 31 divides and stores documents into chunks, each including 1024 tokens, as described above.

[0024] In the document DB 31, each document is assigned an ID (1, 2, 3, 4, 5...), and each chunk into which the document is divided is also assigned an ID (1-1, 1-2, 1-3..., 2-1, 2-2, 2-3..., 4-1, 4-2, 4-3...). The same is true for the vectors that are transformed from chunks: in the vector ID32, an ID is assigned that corresponds to the ID of the original chunk and the document that contained it. Furthermore, documents in the document DB 31 can be divided into folders, so that, for example, confidential documents and documents with low confidentiality can be managed separately. While there are security issues as explained above, we can get around the token limit issue by querying the LLM for only the chunks that are closest to the query. If the LLM specifications are not taken into consideration, multiple chunks that are determined to be transmittable may be transmitted. For example, as shown in FIG. 4, the user device 10 compares the feature vector of the query with the vectors stored in the vector DB 32, and searches for vectors with high relevance, for example, vectors with IDs of 1-3, 2-2, and 3-1, and chunks with corresponding IDs are searched for in the document DB 31. The chunks with IDs 1-3, 2-2, and 3-1 are sent to the server device 60 along with the query and are used to generate the answer. When transmitting the entire document to the server device 60, embedding is performed on each document in the document DB 31 to generate a feature vector, which is then compared with the feature vector of the query. As a result, the entire document that is most similar to the query is transmitted to the server device 60 as a search result and is used to generate an answer.

[0025] Incidentally, even when only a portion of a document is transmitted to the server device 60 as in the present embodiment, the transmitted portion may contain confidential data (such as confidential data in the case of a company). When the entire document is transmitted, there is an even higher risk of confidential data being included. Therefore, the user device 10 prevents the transmission of confidential data to the server device 60 by a confirmation process or prohibition processing.

[0026] The configuration of this embodiment for preventing confidential data from being transmitted to the server device 60 will be described below. For example, the contents of a chunk that is considered to be highly relevant to the searched query, the file name of a document that includes the chunk, and even a confirmation dialog may be displayed on the display screen of the user device 10. Then, the chunk may be transmitted to the LLM by the server device 60 only when the user of the user device performs an operation to permit transmission. Instead of asking the user of user device 10 to confirm whether transmission is possible, the administrator of the system or organization to which the user belongs may be asked to confirm whether transmission is possible. In this case, user device 10 transfers the contents of the searched chunk and the file name of the document containing the chunk to administrator device 40 present on the company network.

[0027] When the user operates to permit transmission in response to the confirmation dialogue displayed on the administrator device 40, the administrator device 40 transmits information indicating that transmission is permitted to the user device 10. When the user operates to refuse transmission in response to the confirmation dialogue displayed on the administrator device 40, the administrator device 40 transmits to the user device 10 information indicating that transmission is not permitted. The user device 10 may transmit chunks of search results to the server device 60 only when transmission permission information is received from the administrator device 40 . In this way, confidential data can be prevented from being unconditionally transmitted to the outside (outside the company) without confirmation, making for a more secure response system.

[0028] Also, when confirmation is obtained from both the user and the administrator, the server device 60 may transmit the chunk to the LLM. In this case, even if an operation to permit transmission is performed on the user device 10, the contents of the searched chunk and the file name of the document containing the chunk are transferred to the administrator device 40. As a result, if an operation to prohibit transmission is performed on the administrator device 40, the user device 10 does not transmit the chunk to the server device 60, and if an operation to permit transmission is performed on the administrator device 40, the user device 10 does not transmit the chunk to the server device 60. In this way, even if a user of the user device 10 attempts to transmit confidential data to the outside at his / her own discretion, the leakage of confidential data can be prevented at the discretion of the administrator.

[0029] However, it is troublesome for both the user and the administrator to confirm whether transmission is possible every time a query is made. Therefore, it may be possible to not perform the operation of confirming whether transmission is possible for chunks that have already been permitted for transmission or related chunks. That is, for a chunk in the document DB 31 that has previously been permitted to be transmitted to the server device 60 by a user or a user and an administrator, confirmation by the user or the administrator may not be required thereafter. For this purpose, the user device 10 may store the ID of the chunk in the confirmation-free information DB 33 of the storage unit 30. Furthermore, for chunks included in the same document (file) as a chunk previously permitted to be transmitted to the server device 60 in the document DB 31, confirmation does not need to be requested before transmission even if the chunks are not directly permitted to be transmitted. For this reason, the user device 10 can store the ID of the document in the confirmation-free information DB 33 of the storage unit 30. Furthermore, for a chunk of a document stored in the same folder as a document containing a chunk that has previously been permitted to be transmitted to the server device 60 in the document DB 31, confirmation may not be required before transmission even if the chunk is not directly permitted to be transmitted. For this purpose, the user device 10 may store the name of the folder in the confirmation-free information DB 33 of the storage unit 30.

[0030] It is not necessary to require confirmation before sending documents stored in a specific folder on the document DB 31. Although it may be considered undesirable from a security standpoint to omit confirmation by the user or administrator, by thoroughly implementing an operation in which confidential data is appropriately stored in a specific folder, it is possible to omit cumbersome confirmation and improve the usability of the reply system.

[0031] When a chunk determined to be highly relevant as a result of the search is included in a confidential document, the user device 10 may not transmit the chunk to the server device 60. When embedding a document stored in the document DB 31, if it is found that a prohibited word such as "confidential", "secret", or "Confidential" is written in a specific position, such as a header, the user device 10 can register the document in the document DB 31 or the vector DB 32 as a document that cannot be transmitted. That is, for a document in which prohibited words have been confirmed, the user device 10 can register the ID of the document in the confirmation-unnecessary information DB 33. For documents stored in the confirmation-free information DB 33, the user device 10 can unconditionally not transmit the documents to the server device 60 without confirmation from the user or the administrator. In the first place, it is also possible not to embed documents containing prohibited words and to exclude them from the search using the feature vector of the query. In this way, confidential data can be prevented from being transmitted to the server device 60 regardless of the decision of the user or administrator. In addition, the file attributes may be checked, and documents to which no specific authority, such as company-wide viewing, has been granted may also be stored in the confirmation-free information DB 33. It should be noted that chunks contained in documents that are not stored in the confirmation-unnecessary information DB 33 may be sent to the server device 60 unconditionally, or confirmation may be sought from the user or administrator of the user device 10.

[0032] FIG. 5 is a diagram illustrating a procedure for a query and a response on a user device. FIG. 5(a) shows a display screen 100 that is displayed on the display device 12 of the user device 10 when a user inputs a question (query). Display screen 100 in Fig. 5(a) displays a question input field 101 and a send button 102. The user inputs a question in natural language into question input field 101 using input device 11, and selects send button 102. This operation causes user device 10 to compare the feature vector of the question sentence input into question input field 101 with the feature vectors stored in vector DB 32, as described above, and searches for chunks (search results) corresponding to vectors with high relevance. When an entire document is to be transmitted, the search result is the document.

[0033] FIG. 5(b-1) shows the display screen 100 displaying the transmission confirmation screen. In FIG. 5(b-1), the user device 10 displays a confirmation field 103 and a confirmation dialog 104 on the display screen 100. In a confirmation box 103, the contents of the search results are displayed for the user to confirm. The confirmation dialog 104 displays the text "Do you want to send?" along with a "Yes" button and a "No" button. If the user checks the search results displayed in the confirmation field 103 and judges that there is no problem in sending them to the server device 60, the user selects the "Yes" button in the confirmation dialog 104, and the user device 10 determines that the displayed search results can be sent and sends them to the server device 60. When the user judges that the search results should not be sent to the server device 60, and selects the "No" button in the confirmation dialog 104, the user device 10 determines that the search results displayed cannot be sent and does not send them to the server device 60.

[0034] When there are multiple search results, the search results may be displayed one by one in the confirmation field 103, and the send / no send may be determined in the confirmation dialog 104. As shown in Fig. 5(b-1), multiple search results may all be displayed in the confirmation field 103 with check boxes 107, and the "Yes" button in the confirmation dialog 104 may be selected with a check mark placed on a search result that is determined to be OK to send to the server device 60. If there are no search results that can be sent, the user should select "No" in the confirmation dialog box 104 without checking any of the search results.

[0035] Alternatively, the decision as to whether or not to transmit the search results to server device 60 may be made by an administrator, rather than by the user of user device 10 . FIG. 5(b-2) shows the display screen 100 displaying a transmission confirmation screen for the administrator. The display in Figure 5(b-2) is similar to that in Figure 5(b-1), except that the check box 107 is not displayed in the confirmation field 103, and the confirmation dialog 104 displays the text "Do you want to confirm with the administrator?" along with a "Yes" button and a "No" button. When the user selects the "Yes" button, the displayed search results are sent to the administrator device 40, and a confirmation screen similar to that of FIG. 5(b-1) is displayed on the display device of the administrator device 40, and the administrator uses the confirmation screen to decide whether or not to transmit. The confirmation result is sent to the user device 10, and the user device 10 decides whether or not to transmit based on the confirmation result. The user device 10 sends the search results that the administrator device 40 has determined to be transmittable to the server device 60.

[0036] FIG. 5(c-1) shows the display screen 100 on which the answer sentence is displayed. In FIG. 5(c-1), the user device 10 shows a display screen 100 displaying a reply field 105 for the administrator. If the search results are sent to the server device 60 as a result of confirming whether or not to send using the confirmation dialog, a reply text is generated in the server device 60. In this case, the user device 10 displays the reply text received from the server device 60 in the reply field 105. At that time, it is possible to display a list of the search results used by the server device 60 to generate the reply text, i.e., the search results sent by the user device 10 to the server device 60. The list includes the paths in the document DB 31 of the documents used in the reply, and the chunks specifically used in the reply. folderA / kansu-manual.pdf[3] means the third chunk of kansu-manual.pdf (document ID 1) in folder A. folderA / program-tebiki.pdf[2] means the second chunk of program-tebiki.pdf (document ID 2) in folder A. Also, folderB / function-method.pdf[1] means the first chunk of function-method.pdf (document ID 3) in folder B.

[0037] 5(c-2) shows the display screen 100 on which the answer cannot be displayed. If there is no sendable search result as a result of checking whether the answer can be sent using the confirmation dialog, the server device 60 cannot generate an answer. Therefore, the user device 10 cannot display any answer in the answer field 105, and displays a message such as "Answer cannot be displayed."

[0038] FIG. 6 is a flowchart illustrating the answering process performed by the user device. In step S101, the user device 10 (reception unit 21) determines whether or not a query (question sentence) has been input by the user using a keyboard, a touch panel, or the like. When it is determined that a query has been input (Yes in step S101), the user device 10 (acceptor 21) accepts the query in step S102. In step S103, the user device 10 (conversion unit 22) calculates the embedding of the received query, and converts it into a feature vector indicating the meaning of the query. In step S104, the user device 10 (search unit 23) compares the calculated feature vector of the query with the feature vectors of the chunks stored in the vector DB 32, and searches for chunks that are highly related to the query. Then, the user device 10 (search unit 23) acquires the corresponding chunks in the document DB 31 as search results based on the IDs assigned to the searched feature vectors.

[0039] In step S105, the user device 10 (confirmation unit 24) confirms whether the chunk of the search result acquired by the search unit 23 can be transmitted to the cloud (server device 60). In step S106, the user device 10 (transmitting unit 25) checks whether or not there is a chunk that can be transmitted as a result of the check. If there are chunks that can be transmitted (Yes in step S106), the user device 10 (transmitter 25) transmits the chunks that can be transmitted to the cloud (server device 60) in step S107, and ends this reply process. If there is no chunk that can be transmitted (No in step S106), the user device 10 (display unit 27) displays on the display device 12 a message indicating that the answer cannot be displayed in step S108, and ends the answering process.

[0040] If it is determined that no query has been input (No in step S101), the user device 10 (receiving unit 26) determines whether a reply message has been received from the server device 60 in step S111. If it is determined that the reply message has not been received (No in step S111), the reply process ends. When it is determined that the answer sentence has been received (Yes in step S111), the user device 10 (display unit 27) displays the answer sentence on the display device 12 in step S112, and ends the current answering process. Alternatively, the user device 10 may read out the answer sentence aloud instead of displaying it on the display device 12.

[0041] FIG. 7 is a diagram (part 1) for explaining the confirmation process executed by the user device. FIG. 7A is a flowchart showing a first example of the confirmation process. In step S201, the confirmation unit 24 displays on the display device 12 the contents of the search results other than those stored in the confirmation-free information DB 33 and a confirmation dialog. In step S202, the confirmation unit 24 determines that the search results for which the operation to transmit has been performed in the confirmation dialog can be transmitted. In step S203, the confirmation unit 24 determines that the search results, for which the operation to not transmit has been performed in the confirmation dialogue, cannot be transmitted. In step S204, the confirmation unit 24 stores the search results that are determined to be transmittable in the confirmation-free information DB 33, and ends the confirmation process.

[0042] FIG. 7B is a flowchart showing a second example of the confirmation process. The confirmation unit 24 transmits the search results other than those stored in the confirmation-unnecessary information DB 33 to the administrator device 40 in step S211. The confirmation unit 24 receives the result of the confirmation made by the administrator device 40 in step S212. In step S213, the confirmation unit 24 determines that the search results for which the administrator device 40 has performed an operation to transmit the search results can be transmitted. In step S214, the confirmation unit 24 determines that the search results for which the administrator device 40 has performed an operation to not transmit the search results are not transmittable. In step S215, the confirmation unit 24 stores the search results that are determined to be transmittable in the confirmation-free information DB 33, and ends the confirmation process.

[0043] FIG. 7C is a flowchart showing a third example of the confirmation process. In step S221, the confirmation unit 24 displays on the display device 12 the contents of the search results other than those stored in the confirmation-free information DB 33 and a confirmation dialog. In step S222, the confirmation unit 24 determines that the search results for which the operation to transmit has been performed in the confirmation dialog can be transmitted. In step S223, the confirmation unit 24 determines that the search results for which the operation to not transmit has been performed in the confirmation dialogue cannot be transmitted. In step S224, the confirmation unit 24 transmits the search results that are determined to be transmittable to the administrator device 40. The confirmation unit 24 receives the confirmation result from the administrator device 40 in step S225.

[0044] In step S226, the confirmation unit 24 determines that the search results for which the administrator device 40 has performed an operation to transmit the search results can be transmitted. In step S227, the confirmation unit 24 determines that the search results for which the administrator device 40 has performed an operation to not transmit the search results are not transmittable. In step S228, the confirmation unit 24 stores the search results that are determined to be transmittable in the confirmation-free information DB 33, and ends the confirmation process. Even if the search result (chunk) itself is not stored in the confirmation-free information DB 33, if it is included in a document or folder stored in the confirmation-free information DB 33, confirmation of whether or not it can be sent may not be performed. In addition, due to changes in the company's security policy, it may be necessary to determine that content that was previously determined to be transmittable should now be determined to be untransmittable. Therefore, search results that have once been determined to be transmittable may not be excluded from confirmation, and transmission may be confirmed each time. In this case, it is not necessary to perform the processes of registering search results that have been determined to be transmittable in the confirmation-free information DB 33 in steps S204, S215, and S228.

[0045] FIG. 8 is a diagram (part 2) illustrating the confirmation process executed by the user device. FIG. 8A is a flowchart showing a fourth example of the confirmation process. In step S301, the confirmation unit 24 determines whether the search result is a chunk contained in a document in a specific folder of the document DB 31. If it is determined that the search result is a chunk included in a document in a specific folder of the document DB 31 (Yes in step S301), then in step S302, the confirmation unit 24 determines that the search result is unsendable. If it is determined that the search results do not include any chunks contained in documents in a specific folder of the document DB 31 (No in step S301), the confirmation unit 24 performs, in step S303, any of the confirmation processes described in Fig. 7. Alternatively, the process of step S303 may not be performed, and it may be determined that search results not in a specific folder can be transmitted unconditionally. In step S304, the confirmation unit 24 determines whether there is a pending determination result. If it is determined that there is no pending determination result (No in step S304), the confirmation process ends. If it is determined that there is a pending determination result (Yes in step S304), the confirmation unit 24 returns to step S301 and executes the confirmation process for another search result.

[0046] FIG. 8B is a flowchart showing a fifth example of the confirmation process. In step S311, the confirmation unit 24 determines whether the search result includes a prohibited word. Prohibited words include the above-mentioned "confidential", "secret", and "confidential". If it is determined that the search result includes a prohibited word (Yes in step S311), in step S312, the confirmation unit 24 determines that the search result is unsendable. If it is determined that the search results do not include prohibited words (No in step S311), the confirmation unit 24 performs any of the confirmation processes described in Fig. 7 in step S313. Alternatively, the process in step S313 may not be performed, and it may be determined that the search results that do not include prohibited words are unconditionally transmittable. In step S314, the confirmation unit 24 determines whether there is a pending determination result as to whether transmission is possible. If it is determined that there is no pending determination result (No in step S314), the confirmation process ends. If it is determined that there is a pending determination result (Yes in step S314), the process returns to step S311 and confirmation process is performed on another search result.

[0047] FIG. 9 is a flowchart illustrating the confirmation process executed by the administrator device. The administrator device 40 (receiving unit 52) ​​determines whether or not a search result has been received from the user device 10 in step S401. If it is determined that the search result has not been received (No in step S401), the administrator device 40 ends the process. When it is determined that the search results have been received (Yes in step S401), the administrator device 40 (confirmation unit 53) displays the contents of the search results and a confirmation dialog on the display device 42 in step S402. In step S403, the confirmation unit 53 determines that the search results for which the operation to transmit has been performed in the confirmation dialog can be transmitted. In step S404, the confirmation unit 53 determines that the search results, for which the operation to not transmit has been performed in the confirmation dialog, cannot be transmitted. In step S405, the transmitting unit 54 transmits the confirmation result to the user device 10, and the process ends.

[0048] [Other Examples] In the answering system 1 of the present embodiment described above, in order to determine the documents to be sent to the LLM, embedding is performed on the documents in the document DB 31 to generate feature vectors. The user device 10 that asks the question then embeds the input query to generate feature vectors, and searches for documents that are highly relevant to the query by comparing the feature vectors of the documents with the feature vectors of the query. Alternatively, the user device 10 may perform a search on documents in the document DB without vectorizing the query, and determine highly relevant documents. As an example, a search may be performed using keywords included in the query, and chunks with a large number of hits may be determined to be highly relevant. This method of calculating similarity using the frequency of occurrence of words in the query and documents is known under the name of TF-IDF (Term Frequency-Inverse Document Frequency), etc. Even in this case, the user device 10 determines chunks to be used in the reply from the document DB 31 that is not open to the server device 60, and transmits them to the server device 60 after going through the confirmation processes in Figures 7 and 8. Therefore, it is possible to realize a reply system that uses LLM while suppressing the risk of confidential data leaking.

[0049] FIG. 10 is a block diagram illustrating an embodiment of a computer system. The configuration of the computer device 300 will be described with reference to FIG. 10, a computer device 300 includes a control circuit 301, a storage device 302, a reading / writing device 303, a recording medium 304, a communication interface 305, an input / output interface 306, an input device 307, and a display device 308. The communication interface 305 is connected to a network 309. The components are connected to each other via a bus 310. The user device 10 and the administrator device 40 can be configured by appropriately selecting some or all of the components described in the computer device 300. The control circuit 301 controls the entire computer device 300. The control circuit 301 is a processor such as a central processing unit (CPU), a multi-core CPU, a field programmable gate array (FPGA), or a programmable logic device (PLD). The control circuit 301 functions as, for example, the reception unit 21, the conversion unit 22, the search unit 23, the confirmation unit 24, the transmission unit 25, the reception unit 26, and the display unit 27 in FIG. The control circuit 301 functions as, for example, the acceptance unit 51, the reception unit 52, the confirmation unit 53, the transmission unit 54, and the display unit 55 in FIG.

[0050] The storage device 302 stores various data. The storage device 302 is, for example, a memory such as a Read Only Memory (ROM) and a Random Access Memory (RAM), a Hard Disk (HD), and a SolID State Drive (SSD). The storage device 302 functions as, for example, the storage unit 30 in FIG. 2. The ROM also stores programs such as a boot program. The RAM is used as a work area for the control circuit 301. The HD stores programs such as the OS, application programs, and firmware, as well as various data. When performing various processes, the user device 10 reads out the performance control program stored in the storage device 302 into the RAM. By executing the control program read out into the RAM by the control circuit 301, the user device 10 executes processes including one or more of a reception process, a conversion process, a search process, a confirmation process, a transmission process, a reception process, and a display process. When performing various processes, the administrator device 40 reads out the performance control program stored in the storage device 302 into the RAM. By executing the control program read out into the RAM in the control circuit 301, the user device 10 executes processes including one or more of a reception process, a reception process, a confirmation process, a transmission process, and a display process. Each of the above-mentioned programs may be stored in a storage device of a server on the network 309 as long as the control circuit 301 can access the programs via the communication interface 305 .

[0051] The reading / writing device 303 is controlled by the control circuit 301 and reads / writes data from / to a removable recording medium 304. The reading / writing device 303 is, for example, a FDD (Floppy Disk Drive), a CDD (Compact Disc Drive), a DVDD (Digital Versatile Disk Drive), a BDD (Blu-ray (registered trademark) Disk Drive), a USB (Universal Serial Bus), a SATA, or the like. Various data are stored in the recording medium 304. The recording medium 304 may store, for example, a processing program for an information processing device. The control program of the user device 10 includes one or more of a reception process, a conversion process, a search process, a confirmation process, a transmission process, a reception process, and a display process. The control program of the administrator device 40 includes one or more of a reception process, a reception process, a confirmation process, a transmission process, and a display process. The recording medium 304 is connected to the bus 310 via the reading / writing device 303, and the control circuit 301 controls the reading / writing device 303 to read / write data. Furthermore, the recording medium 304 is a non-transitory recording medium such as, for example, an SD memory card, a floppy disk (FD), a compact disc (CD), a digital versatile disk (DVD), a Blu-ray (registered trademark) disk (BD), or a flash memory.

[0052] The communication interface 305 connects the computer device 300 and other devices via the network 309 so that they can communicate with each other. The communication interface 305 may include an interface having a wireless LAN function and an interface having a short-distance wireless communication function. The wireless LAN interface may support, for example, Wi-Fi (registered trademark) as a wireless LAN standard. The short-distance wireless interface may support, for example, Bluetooth (registered trademark) as a short-distance wireless communication standard. LAN is an abbreviation for Local Area Network. The communication interface 305 functions, for example, as the communication device 13 in FIG. 2 and as the communication device 43 in FIG. 3.

[0053] The input / output interface 306 is connected to, for example, an input device 307, and when a signal indicating various information is input from the connected input device 307, the input / output interface 306 outputs the input signal via a bus 310 to the control circuit 301. Furthermore, when a signal indicating various information output from the control circuit 301 is input via the bus 310, the input / output interface 306 outputs the signal to various devices connected thereto. The input device 307 is, for example, a keyboard, a mouse, a touch panel, etc. The input device 307 functions, for example, as the input device 11 in FIG. 2 and as the input device 41 in FIG. The display device 308 displays various types of information. For example, the display device 308 displays an image drawn by the user device 10. The display device 308 may display information for receiving input on a touch panel. For example, the display device 308 functions as the display device 12 in FIG. 2 and as the display device 42 in FIG. 3. The input / output interface 306, the input device 307, and the display device 308 may function as a GUI (Graphical User Interface), which allows the computer device 300 to receive intuitive operations using a touch panel, a mouse, or the like. The network 309 may be, for example, a LAN, wireless communication, or the Internet, and communicatively connects the computer device 300 to other devices.

[0054] This embodiment is not limited to the embodiment described above, and various configurations or embodiments can be adopted without departing from the gist of this embodiment. [Explanation of symbols]

[0055] 1 Answer System, 10 User Device, 11 Input Device, 12 Display Device, 13 Communication Device, 20 Processing Unit, 21 Reception Unit, 22 Conversion Unit, 23 Search Unit, 24 Confirmation Unit, 25 Transmission Unit, 26 Reception Unit, 27 Display Unit, 30 Storage Unit, 31 Document Database, 32 Vector Database, 33 Confirmation-unnecessary Information, 40 Administrator Device, 41 Input Device, 42 Display Device, 43 Communication Device, 50 Processing Unit, 51 Reception Unit, 52 Reception Unit, 53 Confirmation Unit, 54 Transmission Unit, 55 Display Unit, 60 Server Device, 100 Display Screen, 101 Question Input Field, 102 Send Button, 103 Confirmation Field, 104 Confirmation Dialog, 105 Answer Field

Claims

1. An information processing device that processes a document and a query using a machine learning model provided by a server device to obtain an answer to the query, a reception unit that receives a query input by a user using an input device; a search unit that searches one or more documents stored in a document database that is not open to the server device based on the received query, and obtains one or more contents that are highly related to the query as a search result; a transmission unit that transmits the search result to the server device via a communication device; a receiving unit that receives a response sentence generated by the server device using the transmitted search result from the server device via a communication device; An information processing device comprising:

2. 2. The information processing device according to claim 1, A conversion unit converts the received query into a feature vector, A document stored in the document database is converted into one or more feature vectors and stored in a vector database; the search unit compares the feature vectors stored in the vector database with the feature vectors of the query to search for feature vectors that are highly relevant to the query, and obtains from the document database the contents of documents corresponding to the feature vectors as the search results to be transmitted to the server device; 23. An information processing apparatus comprising:

3. 3. The information processing device according to claim 1, the search unit acquires only the content most relevant to the query as the search result to be transmitted to the server device; 23. An information processing apparatus comprising:

4. 3. The information processing device according to claim 1, a confirmation unit that confirms whether the search result can be transmitted to the server device; the transmitting unit transmits the search results confirmed as transmittable by the confirming unit to the server device.

23. An information processing apparatus comprising:

5. 5. The information processing device according to claim 4, the confirmation unit displays the contents of the search result and a confirmation screen on a display device, and enables the search result to be transmitted when an operation is performed to enable the search result to be transmitted.

23. An information processing apparatus comprising:

6. 5. The information processing device according to claim 4, The confirmation unit transmits the contents of the search results to another device, when receiving information from the other device that the search result is confirmed to be transmittable, the search result is transmittable.

23. An information processing apparatus comprising:

7. 5. The information processing device according to claim 4, the confirmation unit displays the contents of the search result and a confirmation screen on a display device, and transmits the search result to another device when an operation indicating that the search result can be transmitted is performed; when receiving information from the other device that the search result is confirmed to be transmittable, the search result is transmittable.

23. An information processing apparatus comprising:

8. 5. The information processing device according to claim 4, the confirmation unit allows the content that has once been made transmittable to be transmitted without confirmation again; 23. An information processing apparatus comprising:

9. 5. The information processing device according to claim 4, the confirmation unit allows contents included in the same document as contents that have been once allowed to be transmitted to be transmitted without confirmation; 23. An information processing apparatus comprising:

10. 5. The information processing device according to claim 4, the confirmation unit allows contents included in a document included in the same folder as a document including contents that have been confirmed as transmittable to be transmitted without confirmation; 23. An information processing apparatus comprising:

11. 5. The information processing device according to claim 4, the confirmation unit allows the contents of a document included in a specific folder in the document database to be transmitted without confirmation; 23. An information processing apparatus comprising:

12. 5. The information processing device according to claim 4, The information processing device, wherein the confirmation unit confirms that transmission is not permitted when the search result includes a specific character string.

13. 1. An information processing method for causing a processor to process a document and a query using a machine learning model provided by a server device, thereby obtaining an answer to the query, the method comprising: The processor, Accepting a query input by a user using an input device; performing a search for one or more documents stored in a non-public document database on the server device based on the received query, and obtaining one or more contents highly related to the query as a search result; transmitting the search results to the server device via a communication device; receiving a response sentence generated by the server device using the transmitted search result from the server device via a communication device; 23. An information processing method comprising:

14. An information processing program that causes a processor to perform an information processing method for obtaining an answer to a query by processing a document and a query using a machine learning model provided by a server device, the information processing method comprising: The processor, Accepting a query input by a user using an input device; performing a search for one or more documents stored in a non-public document database on the server device based on the received query, and obtaining one or more contents highly related to the query as a search result; transmitting the search results to the server device via a communication device; receiving a response sentence generated by the server device using the transmitted search result from the server device via a communication device; 23. An information processing program comprising:

Images