Information processing system, information processing method, and program

Abstract

It is an object of the present invention to disambiguate questions.SOLUTION: According to an aspect of the present invention, there is provided an information processing apparatus including a question acquisition unit configured to acquire a question input by a user, a search unit configured to search for search target information using the question acquired by the question acquisition unit, and an output unit configured to output a question for resolving ambiguity of the question when the question is evaluated to be ambiguous based on the information searched for by the search unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Increasingly, companies and public institutions are introducing full-text search systems, including enterprise search, to find the information they need from the vast amount of digital documents they have stored. Furthermore, the introduction of question-answering systems, which allow users to search for and answer questions through dialogue, has made it easier for them to find the information they need.

[0003] Furthermore, in recent years, the emergence of generative AI based on large-scale language models such as ChatGPT / GPT has made it possible to provide more natural and accurate question-answering than ever before. While generative AI has a high ability to respond to open-domain questions, it cannot accurately answer domain-specific questions that are not included in the training data, such as questions about specific documents within a company.

[0004] Non-Patent Document 1 reports on a mechanism called RAG (Retrieval-Augmented Generation) that links a generative AI with a search system to enable answers to unlearned matters. RAG uses a search system that registers domain knowledge to acquire domain knowledge that is closely related to the question, and by including the domain knowledge in the instructions (prompts) given to the generative AI, it enables the generative AI to answer unlearned matters.

[0005] Furthermore, Patent Document 1 discloses a method of determining the ambiguity of search terms in a question sentence, and if it is determined that the search terms are ambiguous, generating a question to resolve the ambiguity and asking the question to the user to narrow down the search conditions. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang, Retrieval-Augmented Generation for Large Language Models: A Survey, arXiv:2312.10997, 27 Mar 2024(https: / / arxiv.org / abs / 2312.10997) [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-11079 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0008] RAG works in conjunction with a search system to obtain the knowledge necessary for the answer and generate an answer by including that knowledge in the prompt. However, there are limits to the tokens and data size of prompts. Also, if too much knowledge is included in the prompt, the accuracy of the answer will decrease, so it is necessary to obtain appropriate knowledge for the question and create the prompt. However, if the question is ambiguous and a large amount of knowledge is found in the search, it becomes difficult to select the appropriate knowledge.

[0009] According to the method of Patent Document 1, a method is disclosed in which necessary information is acquired by narrowing down appropriate knowledge through dialogue.

[0010] However, Patent Document 1 assumes that the meaning of a search term may be ambiguous due to the ambiguity of the search term, and has the problem that it cannot take into account ambiguity due to differences in context.

[0011] For example, consider a case where a question is asked in a question-answering system for multiple systems that require logins related to internal business operations, such as "Please tell me how to log in." In the context of the question, it is ambiguous as to which system the "login" refers to, but the meaning of the word "login" is clear, so the method of Patent Document 1 cannot resolve the ambiguity and cannot lead to an appropriate answer.

[0012] Therefore, an object of the present invention is to eliminate the ambiguity of questions. [Means for solving the problem]

[0013] The present invention is characterized by comprising a question acquisition means for acquiring a question input by a user, a search means for searching for search target information using the question acquired by the question acquisition means, and an output means for outputting a question to resolve the ambiguity of the question if the question is evaluated to be ambiguous based on the information searched by the search means. [Effects of the Invention]

[0014] According to the present invention, it is possible to resolve ambiguity in questions. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a question answering device according to an embodiment of the present invention. [Figure 2] 1 is a block diagram showing an example of the hardware configuration of a question answering device and a user terminal according to an embodiment of the present invention. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a question answering device according to an embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating an example of a domain knowledge construction process according to an embodiment of the present invention. [Figure 5] FIG. 2 is a diagram illustrating an example of a configuration of a domain knowledge storage area in the embodiment of the present invention. [Figure 6]FIG. 4 is a diagram illustrating an example of a domain management table in the embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating an example of a question answering process according to an embodiment of the present invention. [Figure 8] 10 is an example of an equation for calculating a search vector used in a neighborhood search according to an embodiment of the present invention. [Figure 9] 10 is an example of a formula for calculating the ambiguity of a question sentence in an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of a dialogue screen for question and answering in the embodiment of the present invention. [Figure 11] FIG. 10 is a diagram showing an example of a question sentence vector generated from a question sentence in the embodiment of the present invention. [Figure 12] FIG. 10 is a diagram illustrating an example of a neighborhood division unit acquired as a result of a neighborhood search of a question text vector according to an embodiment of the present invention. [Figure 13] FIG. 10 is a diagram illustrating an example of a process for generating a refinement query using an LLM in an embodiment of the present invention. [Figure 14] FIG. 10 is a diagram illustrating an example of a refinement search vector according to an embodiment of the present invention. [Figure 15] FIG. 10 is a diagram illustrating an example of a neighborhood division unit acquired as a result of a neighborhood search of a refined search vector according to an embodiment of the present invention. [Figure 16] FIG. 1 is a diagram illustrating an example of a process for generating an answer to a question using an LLM in an embodiment of the present invention. [Figure 17] FIG. 10 is a diagram illustrating an example of a process for generating an answer to a question using an LLM in the second embodiment of the present invention. [Figure 18] FIG. 1 is a diagram showing a conceptual diagram of the processing of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0017] 1 is a diagram showing an example of a system configuration of a question answering apparatus according to an embodiment of the present invention. A question answering apparatus 100 is configured to be connected to a user terminal 110 via a network 120.

[0018] The question answering device 100 presents an answer to a question received from the user terminal 110.

[0019] The user terminal 110 sends a question entered by the user to the question answering apparatus 100 and displays an answer returned by the question answering apparatus 100 .

[0020] FIG. 2 is a block diagram showing an example of the hardware configuration of the question answering apparatus 100 and the user terminal 110 according to the embodiment of the present invention.

[0021] As shown in FIG. 2, the information processing device includes a CPU (Central Processing Unit) 201, a RAM (Random Access Memory) 202, a ROM (Read Only Memory) 203, an input controller 205, a video controller 206, a memory controller 207, and a communication I / F controller 208 connected via a system bus 204.

[0022] The CPU 201 comprehensively controls each device and controller connected to the system bus 204 .

[0023] The RAM 202 functions as a main memory, a work area, etc. for the CPU 201. The CPU 201 loads programs and the like required for executing processing from the ROM 203 or the external memory 211 into the RAM 202, and executes the loaded programs to realize various operations.

[0024] ROM 203 or external memory 211 stores the BIOS (Basic Input / Output System) and OS (Operating System), which are control programs executed by CPU 201, computer-readable and executable programs for realizing this information processing method, and various necessary data (including data tables).

[0025] The input controller 205 controls input from an input device such as a keyboard 209 or a pointing device such as a mouse (not shown). If the input device is a touch panel, the user can issue various instructions by pressing (touching with a finger or the like) icons, cursors, or buttons displayed on the touch panel.

[0026] The touch panel may also be a touch panel capable of detecting positions touched by multiple fingers, such as a multi-touch screen.

[0027] The video controller 206 controls the display on an external output device such as a display 210. The display also includes the display of a notebook computer integrated into the main body. Note that the external output device is not limited to a display, and may be, for example, a projector. In addition, for devices capable of receiving the above-mentioned touch operation, an input device is also provided.

[0028] The video controller 206 can control a video memory (VRAM) for display control, and can use part of the RAM 202 as a video memory area, or can provide a separate dedicated video memory.

[0029] The memory controller 207 controls access to the external memory 211. The external memory may be an external storage device (hard disk) that stores a boot program, various applications, font data, user files, edited files, and various data, a flexible disk (FD), or a CompactFlash (registered trademark) memory connected to a PCMCIA card slot via an adapter.

[0030] The communication I / F controller 208 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, communication using TCP / IP, telephone lines such as ISDN, and 3G lines for mobile phones are possible.

[0031] The CPU 201 enables display on the display 210 by, for example, executing a process of expanding (rasterizing) an outline font into a display information area in the RAM 202. The CPU 201 also enables user instructions using a mouse cursor (not shown) or the like on the display 210.

[0032] 3 is a diagram showing an example of the functional configuration of the question answering apparatus 100 and the user terminal 110. The functions of each functional unit will be explained in the description of the flowcharts of FIGS.

[0033] (Summary) First, an overview of the present invention will be described using FIG. 18 . In this embodiment, the present invention will be described assuming a scenario in which, for example, an employee on a portal site for employees asks a chatbot a question about information (documents) on the site and receives an answer. In this embodiment, an example will be described in which a large language model (LLM) is used as the answer generation processing unit 307, but this is not limited thereto, and a generative AI, a machine learning model, or other method capable of generating an answer may also be used. Furthermore, in this embodiment, the answer generation processing unit 307 is configured to be inside the question and answering processing device 100, but it may also be located externally, and answers may be obtained via an API, for example.

[0034] The questions that users ask chatbots may be vague, and in such cases, the domain knowledge search may fail to retrieve documents suitable for generating an answer. When a user's question is vague, asking additional questions to narrow the topic can clarify the information the user wants to know, enabling an appropriate question and response.

[0035] In the present invention, domain knowledge refers to information that serves as the basis for answers to questions from users. For example, if the present invention is applied as a search system (including a dialogue (chat) style question-answering system) for a company's employee website, the information posted on the employee website (such as information on web pages and posted document files) is the domain knowledge. Domain knowledge is registered and managed by, for example, registering site information obtained by searching for a prefix of a URL and updating it periodically, or by having an administrator register data in advance. In the present invention, domain knowledge is divided into division units in advance and vectorized for each division unit (details will be described later with reference to FIG. 4). Domain knowledge is an example of search target information.

[0036] Therefore, in this invention, the following processing is performed. First, assume the situation where a user asks a chatbot a question for the first time (top of Figure 18). First, (1) obtain the question entered by the user to the chatbot. Then, obtain a question vector by vectorizing the user's question. (2) Combine the question vector and the context vector to obtain a search vector. The context vector is a vectorized version of the conversation flow before the question, and its initial value is 0 for the first question. (3) Using the search vector obtained in (2), search domain knowledge to obtain a specified number of nearby division units similar to the search vector. To determine whether the content of the obtained division units is biased toward a specific content, the ambiguity of the nearby division units obtained by the search is calculated (details will be described later with reference to Figures 7 and 9). (4) If the ambiguity is greater than a predetermined condition (i.e., the content of the division units varies greatly and is determined to be ambiguous), generate a prompt including the question and the nearby division units obtained by the search, and generate a refinement question to narrow down and specify the question content for the user. (5) The prompt generated in (4) is input into the LLM to obtain the generated refinement question, which is then presented to the user along with the options in (6).

[0037] Next, we will explain the processing after presenting a narrowing question to the user. (7) The user selects an option that best suits their intention from the narrowing question presented in (6). (8) Domain knowledge is searched using a narrowing search vector, which is a combination of the context vector and the question text vector. Here, the context vector may be a vector of the division units of the option selected by the user in the narrowing question. (9) Using the narrowing search vector obtained in (8), domain knowledge is searched to obtain a specified number of nearby division units similar to the search vector. The ambiguity of the obtained nearby division units is again calculated. (10) If the ambiguity is smaller than the specified condition (i.e., the content of the division units has little variation and the question is judged to be clear), a prompt is generated that includes the question text and the nearby division units obtained by the search, and (11) an answer to the user's question is generated. (12) The answer to the user's question is presented, and the processing ends.

[0038] By performing the above-described processing, it becomes possible to generate an answer after clarifying the user's question, thereby realizing appropriate question and answering.

[0039] Next, the process performed by the domain knowledge construction processing unit 301 in the embodiment of the present invention will be described with reference to the flowchart shown in FIG.

[0040] (First embodiment)

[0041] (Domain knowledge construction process) In step S401, the domain knowledge construction processing unit 301 acquires a list of documents that are the source of domain knowledge.

[0042] In step S 402 , the domain knowledge construction processing unit 301 initializes the domain knowledge storage area 302 .

[0043] In step S403, the domain knowledge construction processing unit 301 starts repeating the process up to step S410 for each document in the document list.

[0044] In step S404, the domain knowledge construction processing unit 301 classifies the document into categories. In the embodiment of the present invention, for ease of explanation, one document is considered to belong to one category, and the document name is used as the label of the category.

[0045] The categories may be configured by a predetermined classifier, or may be configured to be classified based on the results of clustering the document list.

[0046] In step S405, the domain knowledge construction processing unit 301 divides the document to be processed into processing units. In the embodiment of the present invention, the document is divided into sections.

[0047] The division unit may be configured to divide based on the document structure such as document section or page, or based on the number of characters. When dividing based on the number of characters, the division may be determined taking into account the number of characters that can be included in the prompt. The document may also be configured to be a single division unit without being divided.

[0048] In step S406, the domain knowledge construction processing unit 301 starts the repetitive process up to step S409 for each of the division units obtained in step S405.

[0049] In step S407, the domain knowledge construction processing unit 301 uses the vectorization processing unit 303 to vectorize the division unit to be processed.

[0050] There are no particular restrictions on the vectors to be generated as long as they express the meaning of the segmentation unit. They can be constructed from vectors based on the tf·idf of the document collection (vectors that take into account the frequency of occurrence and importance of words within documents), or from semantic vectors obtained using a generation AI (such as distributed representations based on word meanings).

[0051] In step S408, the domain knowledge construction processing unit 301 associates the vector for each division unit acquired in step S407 and the category acquired in step S404 with the division unit to be processed and stores them in the domain knowledge management table 501 of the domain knowledge storage area 302.

[0052] In step S409, if there are still division units to be processed, the domain knowledge construction processing unit 301 repeats the process from step S406. If there are no more division units to be processed, the process proceeds to step S410.

[0053] In step S410, if there are still documents to be processed, the domain knowledge construction processing unit 301 repeats the process from step S403, and if there are no more documents to be processed, the process ends.

[0054] (Example of domain knowledge construction process) Next, as a concrete example of the flowchart in Figure 4, we will use Figure 6 to explain the case where the domain knowledge construction process is performed on three documents: "Asset Management System," "Travel Expense Reimbursement Application Manual," and "Time Attendance Management System Specifications."

[0055] In step S401, the domain knowledge construction processing unit 301 acquires three documents, "Asset Management System," "Travel Expense Reimbursement Application Manual," and "Time and Attendance Management System Specifications," which are the source of domain knowledge.

[0056] In step S402, the domain knowledge construction processing unit 301 empties the domain knowledge management table 501 in the domain knowledge storage area 302.

[0057] In step S403, the domain knowledge construction processing unit 301 starts the repeated process up to step S410 for the document "asset management system."

[0058] In step S404, the domain knowledge construction processing unit 301 classifies the category of the document "asset management system" as the document name "asset management system."

[0059] In step S405, the domain knowledge construction processing unit 301 divides the document to be processed, "Asset Management System," into processing units 601 (title: "Login Method") and 602 (title: "Form Printing").

[0060] In step S406, the domain knowledge construction processing unit 301 starts the repeated process up to step S409 for the division unit 601 divided in step S405.

[0061] In step S407, the domain knowledge construction processing unit 301 uses the vectorization processing unit 303 to vectorize the division target to be processed, thereby obtaining a vector 603.

[0062] In step S408, the domain knowledge construction processing unit 301 associates the vector 603 acquired in step S407, the category 604 ("asset management system") acquired in step S404, and the division unit 601 to be processed, and stores them in the domain knowledge management table 501 of the domain knowledge storage area 302.

[0063] In step S409, the domain knowledge construction processing unit 301 repeats the process from step S406 because there are still division units 602 to be processed.

[0064] Similar processing is repeated thereafter until the domain knowledge management table 501 reaches the state shown in FIG.

[0065] (Question-answering processing) Next, the process performed by the question and answer processing unit 305 in the embodiment of the present invention will be described with reference to the flowchart shown in FIG.

[0066] In step S701, the question and answer processing unit 305 acquires a question sentence and a context vector from the dialogue management processing unit 304.

[0067] In step S702, the question and answering processor 305 requests the vector processor 303 to vectorize the question acquired in step S701 and acquire a question vector.

[0068] In step S703, the question and answering processor 305 combines the question sentence vector obtained in step S702 with the context vector obtained in step S701 to obtain a search vector. If there is a conversation before the question sentence obtained in step S701, the context vector is a vector of the flow of that conversation (including the question, narrowing-down questions, and their answers), and the method for generating the context vector is not particularly limited. If no questions have been asked before the question sentence, the context vector is 0. The context vector may be a combination of the immediately preceding question sentence vector and answer sentence vector, or may be a combination of the division unit vectors used to generate the answer. It may also be a vector of a nearby division unit selected by the narrowing-down question.

[0069] In an embodiment of the present invention, vectors are synthesized using Equation 1 shown in FIG. 8, but other methods of vector synthesis may also be used. As described above, the search vector is obtained by synthesizing the question vector obtained in step S702 and the context vector obtained in step S701. The upper part of Equation 1 applies when the magnitude of the context vector is greater than 0, in which case the search vector is the synthesis of the context vector and the question vector. The lower part of Equation 1 applies when the magnitude of the context vector is 0, i.e., the first question. In this case, the search vector is equal to the question vector.

[0070] In step S704, the question and answering processor 305 acquires a specified number of division units that are closest in cosine distance to the search vector acquired in step S703 from the division units stored in the domain knowledge management table 501. The cosine distance is an index that indicates the similarity between vectors, and it can be determined that the closer the value is to 0, the higher the similarity between the vectors.

[0071] In the embodiment of the present invention, the similarity of the contents of documents is calculated using the cosine distance, but the method for calculating the vector distance is not particularly limited, and other methods such as Euclidean distance may also be used.

[0072] In step S705, the question and answering processor 305 calculates the ambiguity for the division units near the question sentence vector acquired in step S704 using Equation 2 shown in FIG. 9. In this embodiment, the ambiguity refers to an index indicating the degree of similarity (similarity) between the searched information (acquired division units). Specifically, the ambiguity is calculated by dividing the sum of the distances between the neighboring division unit vectors and the center of gravity acquired in step S704 by the number of neighboring division units. According to this calculation method, the smaller the distance between the neighboring division unit vectors and the center of gravity, the smaller the variation between the vectors, and therefore the smaller the ambiguity value, and it can be determined that the search has been sufficiently narrowed down to a specific topic. Conversely, the greater the distance between the neighboring division unit vectors and the center of gravity, the greater the variation between the vectors, and therefore the larger the ambiguity value, and the greater the variation in the searched topics, and it can be determined that the search has not been sufficiently narrowed down to a specific topic.

[0073] The method for calculating the ambiguity is not limited to the above-mentioned method, and is not particularly limited as long as it indicates whether the content of the acquired division unit is biased toward a specific subject. It may be configured to calculate the ambiguity from the vector of the division unit, or it may be configured to calculate the ambiguity from the feature amount of the text of the division unit.

[0074] In step S706, if the ambiguity is equal to or less than the specified value, the question and answering processor 305 moves the process to step S707, whereas if the ambiguity is greater than the specified value, the process moves to step S710.

[0075] In step S707, the question and answer processing unit 305 requests the answer generation processing unit 307 to generate an answer from the divided text and question sentence acquired in step S704. Details will be described with reference to FIG.

[0076] In an embodiment of the present invention, the answer generation processing unit 307 is configured to use an LLM, but there is no particular limitation, and a generation AI, machine learning model, or other method capable of generating answers may be used.

[0077] In step S708, the question and answering processor 305 requests the vectorization processor 303 to generate a new context vector from the question sentence and the answer sentence generated in step S707.

[0078] In the present embodiment, the vector is generated from the question sentence and the answer sentence, but it may be generated by including the division unit acquired in step S704.

[0079] In step S709, the question and answering processor 305 returns the answer and the new context vector to the dialogue management processor 304.

[0080] In step S710, the question and answer processing unit 305 requests the answer generation processing unit 307 to generate a refined question sentence from the division unit and the question sentence in step S704. Details will be explained with reference to FIG.

[0081] In an embodiment of the present invention, the generation of narrowing questions is configured to use LLM, but there is no particular limitation, and generation AI, machine learning models, and other methods capable of generating narrowing questions may be used.

[0082] In step S711, the question and answer processing unit 305 returns the narrowing question sentence to the dialogue management processing unit 304.

[0083] (Question-answering processing example) Next, a specific example of the flowchart of FIG. 7 will be described with reference to FIGS.

[0084] The dialogue management processing unit 304 displays a dialogue screen 1001 shown in FIG.

[0085] When a user inputs a question into the input field 1002 on the interactive screen 1001, the dialogue management processing unit 304 displays the question 1003 ("I can't log in.") in the dialogue area and requests the question answering processing unit 305 to generate an answer from the question 1003 and the initial context vector (size = 0) using the processing of the flowchart in Figure 7.

[0086] In step S701, the question and answering processor 305 acquires the question sentence 1003 and the initial context vector from the dialogue management processor 304.

[0087] In step S702, the question and answering processor 305 requests the vector processor 303 to vectorize the question 1003 acquired in step S701, and acquires the question vector 1101 shown in FIG.

[0088] In step S703, the question and answering processor 305 calculates a search vector by applying (Equation 1) to the question sentence vector 1101 acquired in step S702 and the context vector acquired in step S701.

[0089] Since the context vector is in the initial state (size=0), the search vector has the same value as the question sentence vector 1101 when the user asks the first question.

[0090] In step S704, the question and answering processor 305 acquires, from the division units stored in the domain knowledge management table 501, a neighboring division unit 1201 (in this case, the prescribed number is 3) that has a close cosine distance to the vector shown in FIG. 12 for the search vector 1101 acquired in step S703.

[0091] In step S705, the question and answering processor 305 calculates 0.1402 as the ambiguity for the neighboring division unit 1201 acquired in step S704 using (Equation 2) shown in FIG.

[0092] In step S706, the question and answering processor 305 determines that the ambiguity 0.1402 is greater than the specified value (in this specific example, the specified value is 0.1), and therefore moves the process to step S710.

[0093] In step S710, the question and answering processor 305 requests the answer generation processor 307 to create a prompt 1301 shown in FIG. 13 from the neighboring division units 1201 and the question 1003, and generates a refined question 1303 using the LLM.

[0094] In step S711, the question and answering processor 305 returns the refinement question sentence 1303 and the neighboring division units 1201 to the dialogue management processor 304.

[0095] The dialogue management processing unit 304 displays the received narrowing-down question on the dialogue screen 1001. In a specific example of an embodiment of the present invention, options 1005 corresponding to nearby division units 1201 are further displayed to allow easy selection.

[0096] Here, it is assumed that the user selects link 1006 ("Travel Expense Reimbursement Application Manual") from option 1005.

[0097] The dialogue management processing unit 304 sets the division unit vector 1202 corresponding to the link 1006 selected by the user as a context vector, and requests the question answering processing unit 305 to generate an answer from the question sentence 1003 and the newly determined context vector 1202 using the processing of the flowchart in Figure 7.

[0098] In step S701, the question and answering processor 305 acquires the question sentence 1003 and the context vector 1202 from the dialogue management processor 304.

[0099] In step S702, the question and answering processor 305 requests the vector processor 303 to vectorize the question 1003 acquired in step S701, and acquires the question vector 1101 shown in FIG.

[0100] In step S703, the question and answering processor 305 calculates a search vector 1401 by applying (Equation 1) to the question sentence vector 1101 acquired in step S702 and the newly determined context vector 1202 acquired in step S701.

[0101] In step S704, the question and answering processor 305 acquires, from the division units stored in the domain knowledge management table 501, a division unit 1501 (in this case, the prescribed number is 3) that has the closest cosine distance of the vector shown in FIG. 15 to the search vector 1401 acquired in step S703.

[0102] In step S705, the question and answering processor 305 calculates 0.083 as the ambiguity for the neighboring division unit 1501 acquired in step S704 using (Equation 2) shown in FIG.

[0103] In step S706, the question and answering processor 305 determines that the ambiguity of 0.083 is equal to or less than the specified value of 0.1, and moves the process to step S707.

[0104] In step S707, the question and answering processor 305 requests the answer generation processor 307 to generate an answer 1602 from the text 1502 of the division unit 1501 acquired in step S704 and the question sentence 1003. An image of answer generation is shown in FIG.

[0105] In step S708, the question and answering processor 305 requests the vectorization processor 303 to generate a new context vector from the question sentence and the answer sentence generated in step S707.

[0106] Details of the new context vector will be omitted, but the method of generating it is not particularly limited.

[0107] In step S709, the question and answer processing unit 305 returns the answer 1602 and the new context vector to the dialogue management processing unit 304.

[0108] The dialogue management processing unit 304 displays the answer sentence in the acquired answer 1602 on the dialogue screen 1001 .

[0109] FIG. 10 is a diagram showing an example of a dialogue screen for question answering in an embodiment of the present invention. The dialogue screen 1001 is a screen displayed on the web browser 310 on the user terminal 110. A user inputs a question to the chatbot in an input field 1002. The question 1003 is a question displayed in the dialogue area and is displayed by inputting the question in the input field 1002. The narrowing question 1004 is a sentence that prompts the user to select an option to narrow down the intent of the question and is acquired in step S711. The option 1005 is an option that the user selects from the options 1005 to select an intent appropriate to the question and may be the nearby division unit 1201 searched in step S704. The link 1006 is an option selected by the user from the option 1005. The answer 1007 is an answer presented to the user's question and is created in step S707. In addition to the answer, the domain knowledge acquired in S704, which served as the basis for generating the answer, is displayed as basis information 1008. 10, the basis information 1008 is a link, and the content of the basis information can be displayed by clicking or other instructions. The link may display the content of the domain knowledge management table 501, or may be configured to refer to external information. There may also be a function that allows the user to evaluate the answer generated for the question.

[0110] FIG. 13 is a diagram showing an example of a process for generating a narrowing question using an LLM in an embodiment of the present invention. Prompt 1301 is a prompt for generating a narrowing question and is created in step S710. Question 1003 is a question entered by the user and acquired in step S701. 1302 is the category name of the nearby division unit searched for in step S704 and is added to the prompt as an option for the user to select. Narrowing question 1303 is created in step S710 by inputting prompt 1301 into the LLM, and can narrow down the topic the user is asking about.

[0111] FIG. 14 is a diagram showing an example of a refinement search vector in an embodiment of the present invention. In this embodiment, the method of expressing the vector is not particularly limited. For example, a vector based on tf·idf in the question sentence or context (a vector that takes into account the frequency of word appearance and importance) may be configured. In this case, the vector is configured using the tf·idf values ​​corresponding to the words and phrases associated with the numbers (0 to 7) in the upper row. While FIG. 14 shows a seven-dimensional vector, this may be changed depending on the question sentence or context. The refinement search vector 1401 is a vector used when searching after presenting a refinement question to the user. The search vector is created in step S703 and is combined by applying the question sentence vector and context vector to (Equation 1). The context vector used to combine the refinement search vector may be the division unit vector 1202 corresponding to the link 1006 selected by the user.

[0112] 16 is a diagram showing an example of a process for generating an answer to a question using an LLM in an embodiment of the present invention. Prompt 1601 is a prompt for generating an answer to a user's question, and is generated in step S707. 1502 is the text of division unit 1501, and is added to the prompt along with the user's question as basis information for generating the answer. Answer 1602 is generated in step S707 by inputting prompt 1601 into the LLM, and is the answer to the user's question.

[0113] (Second embodiment) The second embodiment of the present invention differs from the first embodiment in the instructions required of the answer generation processing unit 307 in the process of step S707.

[0114] FIG. 17 is a diagram showing an example of a process for generating an answer to a question using LLM in the second embodiment of the present invention. In the first embodiment, an answer 1602 is generated from a prompt including the text 1502 of the acquired division unit 1501 and the question 1003. On the other hand, in the second embodiment, as shown in FIG. 17, in the instruction 1601, a precondition 1701 related to the question is added based on the most frequently occurring category of the neighboring division unit 1501 (e.g., "Travel Expense Reimbursement Application Manual"). By including in the prompt an instruction to answer regarding the most frequently occurring category among the categories of documents obtained as search results, it becomes possible to provide an answer based on documents in the most frequent category, thereby improving the accuracy of the answer. Furthermore, the information 1502 to be provided may be configured to be limited to only information from the most frequently occurring category.

[0115] As described above, by determining the ambiguity of the search results based on the user's question and, if ambiguous, narrowing the topic to questions that clarify the user's question, it becomes possible to search for documents that are appropriate to the user's question and generate an appropriate answer.

[0116] Although the embodiments have been described above, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, a recording medium, etc. Specifically, the present invention may be applied to a system made up of multiple devices, or may be applied to an apparatus made up of a single device.

[0117] Furthermore, the program of the present invention is a program that enables a computer to execute the processing method of the flowchart shown in Fig. 3, and the storage medium of the present invention stores a program that enables a computer to execute the processing method of Fig. 3. Note that the program of the present invention may be a program for each processing method of each device in Fig. 3.

[0118] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0119] In this case, the program itself read from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0120] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon disks.

[0121] Furthermore, it goes without saying that not only are the functions of the above-mentioned embodiments realized by the computer executing a program it has read, but also cases are included in which an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0122] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to the computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0123] Furthermore, the present invention may be applied to a system consisting of multiple devices, or to a device consisting of a single device. It goes without saying that the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or device. In this case, the system or device can enjoy the effects of the present invention by reading a recording medium containing a program for achieving the present invention into the system or device.

[0124] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0125] 100 Question Answering Device 110 User terminal 120 Network

Claims

1. a question acquisition means for acquiring a question input by a user; a search means for searching for search target information using the question acquired by the question acquisition means; an output means for outputting a question to disambiguate the question when the question is evaluated as being ambiguous based on the information retrieved by the retrieval means; An information processing device comprising:

2. an answer acquisition means for acquiring an answer to the question output from the output means; 2. The information processing apparatus according to claim 1, wherein the output means outputs an answer to the question that is generated based on the question acquired by the question acquisition means and the answer acquired by the answer acquisition means.

3. 3. The information processing device according to claim 2, wherein, when the question acquired by the question acquisition means is evaluated as not ambiguous, the output means outputs an answer to the question without outputting a question to resolve the ambiguity.

4. 4. The information processing system according to claim 3, wherein the search means further performs a search using an answer acquired by an answer acquisition means.

5. 5. The information processing apparatus according to claim 4, wherein the question for disambiguating the question output by said output means includes options based on the information retrieved by said retrieval means.

6. 6. The information processing apparatus according to claim 5, wherein the answer acquisition means accepts a selection of the option from the user as an answer to a question including the option.

7. The information processing device according to claim 6, characterized in that, based on the information searched using the question acquired by the question acquisition means, the question acquired by the question acquisition means is evaluated as ambiguous when the similarity between the searched information satisfies a predetermined condition.

8. 8. The information processing device according to claim 7, wherein the similarity between the retrieved information is an index defined by an average value of the distance between the center of gravity when the retrieved information is vectorized and each vector.

9. 9. The information processing apparatus according to claim 8, wherein, if a question or an answer exists before the question acquired by the question acquisition means, the search means performs a search using at least one of them.

10. 10. The information processing apparatus according to claim 9, wherein the output means outputs, in an identifiable manner, information used to generate a response from among the information retrieved by the retrieval means.

11. 11. The information processing apparatus according to claim 10, wherein the answer output by said output means is created using a large-scale language model.

12. 12. The information processing apparatus according to claim 11, wherein the output means outputs an answer created by inputting the question acquired by the question acquisition means and the information searched by the search means into a large-scale language model.

13. 13. The information processing apparatus according to claim 12, wherein the question output by said output means is created using a large-scale language model.

14. 14. The information processing apparatus according to claim 13, wherein the output means outputs a question created by inputting the question acquired by the question acquisition means and the information searched by the search means into a large-scale language model.

15. 15. The information processing apparatus according to claim 14, wherein the information searched for by said search means is information specified by a URL that satisfies a predetermined condition.

16. a question acquisition step in which a question acquisition means of the information processing device acquires a question input by a user; a search step in which a search means of the information processing device searches for search target information using the question acquired in the question acquisition step; an output step in which, when the question is evaluated as being ambiguous based on the information searched in the search step, an output means of the information processing device outputs a question for disambiguating the question; 10. A method for controlling an information processing device, comprising:

17. A program for causing at least one computer to function as each of the means of the information processing device according to any one of claims 1 to 15.

Images