Question and Answer System

Abstract

To provide a question and answer system that can provide a direct answer to a question that a user wants to know. [Solution] The question and answer system 1 includes a memory unit 41 that stores QA data, a reception unit 42 that receives input of a user search query, an estimation unit 43 that infers the information desired by the user from the user search query, creates an estimated search query, and estimates the format of an answer by leaving blank any content that needs to be fact-checked for the estimated search query, and creates an answer format sentence, an extraction unit 44 that extracts QA data based on at least the estimated search query and creates an extracted dataset, a classification unit 45 that classifies the extracted dataset into either an answer candidate dataset that is likely to contain at least an answer to the estimated search query or a related dataset other than the answer candidate dataset, a result creation unit 46 that creates a search result sentence for the estimated search query based on at least a part of the answer candidate dataset, and a presentation unit 47 that presents the search result sentence.

Description

[Technical Field]

[0001] The present invention relates to a question and answer system that stores Q&A data including question sentences and answer sentences in advance and presents search result sentences that match the user's search query. [Background technology]

[0002] Organizations of a certain size, such as corporations and public institutions, have help desk departments that handle inquiries from internal employees and staff, as well as external customers and residents. Help desks often accumulate frequently asked questions (FAQ) data that compiles past inquiries and answers. Hereinafter, data including questions and answers, such as FAQ data, will be referred to as "QA data." In recent years, generative AI (artificial intelligence) has also been used to generate QA data from internal documents of organizations.

[0003] The applicants have already developed the question and answer system described in Patent Document 1 in order to present appropriate answers from Q&A data to new questions. The question and answer system of Patent Document 1 can accurately present answers in the order that is thought to be the content desired by the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6770283 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when the multiple answers presented are long, some users find it tedious to check the multiple answers, and such users desire a direct answer to their question.

[0006] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a question and answer system that can present a direct answer to a question that a user wants to know. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the present invention provides a question and answer system comprising: a storage unit that stores Q&A data including pairs of question sentences and answer sentences; a reception unit that receives input of a user search sentence from a user; an estimation unit that estimates information desired by the user from the user search sentence and creates an estimated search sentence, and also estimates the format of an answer in which content that needs to be fact-checked for the estimated search sentence is left blank and creates an answer-format sentence; an extraction unit that extracts the Q&A data based on at least the estimated search sentence and creates an extracted dataset; a classification unit that classifies the extracted dataset into either an answer candidate dataset that is likely to contain an answer to the estimated search sentence or a related dataset other than the answer candidate dataset; a result creation unit that creates search result sentences for the estimated search sentence based on at least a part of the answer candidate dataset; and a presentation unit that presents the search result sentences to the user.

[0008] The estimation unit may send a first prompt to the large-scale language model instructing it to estimate information desired by the user from the user search statement, receive a first response sentence from the large-scale language model in response to the first prompt, create the estimated search statement from the first response sentence, and send a second prompt to the large-scale language model instructing it to estimate an answer format in which content that needs to be fact-checked for the estimated search statement is left blank, receive a second response sentence from the large-scale language model in response to the second prompt, and create the answer-format sentence from the second response sentence.

[0009] In addition, the result creation unit may send a third prompt to the large-scale language model instructing the large-scale language model to provide an answer to the estimated search statement using at least a part of the answer candidate dataset as reference information, receive a third response sentence from the large-scale language model in response to the third prompt, and create the search result sentence from the third response sentence.

[0010] Further, the storage unit may store a knowledge graph of the QA data, the extraction unit may send a fourth prompt to the large-scale language model instructing it to extract keywords from the estimated search sentence, receive a fourth response sentence from the large-scale language model in response to the fourth prompt, search the QA data using first search criteria based on extraction keywords included in the fourth response sentence, and create a first extracted dataset from the search results, and the classification unit may send a fifth prompt to the large-scale language model instructing it to extract keywords from the answer-form sentence, receive a fifth response sentence from the large-scale language model in response to the fifth prompt, and classify the first extracted dataset into the answer candidate dataset and the related dataset based on the classification keywords included in the fifth response sentence and the knowledge graph.

[0011] In addition, the extraction unit may search the QA data using second search conditions based on the extraction keywords included in the fourth response sentence, and the search results may be a second extracted dataset, and the result creation unit may include some of the QA data included in the answer candidate dataset, the related dataset, and the second extracted dataset in the reference information related to the third prompt. [Effects of the Invention]

[0012] The present invention provides a question and answer system that can provide a direct answer to a question that a user wants to know. [Brief explanation of the drawings]

[0013] [Figure 1] Overview of the question-answering system [Figure 2] Block diagram showing the functions of the question-answering system [Figure 3] An example of QA data [Figure 4] An example of a knowledge graph [Figure 5] Flowchart showing the processing flow of the question and answer system [Figure 6] A diagram explaining the user search query reception process by the reception unit. [Figure 7] FIG. 10 is a diagram illustrating the process of creating an estimated search query by the estimation unit. [Figure 8] FIG. 10 is a diagram illustrating a process for determining extraction keywords performed by an extraction unit. [Figure 9] A flowchart showing the flow of an extraction data set creation process performed by an extraction unit. [Figure 10] A diagram explaining the answer-format sentence creation process by the estimation unit. [Figure 11] FIG. 10 is a diagram illustrating a classification keyword determination process performed by a classification unit. [Figure 12] A flowchart showing the flow of a first extracted data set classification process performed by a classification unit. [Figure 13] FIG. 10 is a diagram illustrating a detection result creation process performed by a result creation unit. [Figure 14] A diagram showing an example of a search result sentence DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a diagram showing an overview of a question and answering system. As shown in Fig. 1, the question and answering system 1 is composed of a server 2, a user terminal 3 connected to the server 2 via a network 5, and a first LLM (Large Language Models) server 4a and a second LLM server 4b connected to the server 2 via the network 5. The network 5 is, for example, the Internet.

[0015] The server 2 is a general-purpose information processing device, such as a physical server located in a data center or a virtual server provided by a cloud service provider. The server 2 has a processor 21, a memory 22, a storage device 23, an output device 24, an input device 25, and a communication device 26, which are connected via a bus 27. The processor 21 is a CPU (Central Processing Unit) or the like, and reads a computer program stored in advance in the storage device 23 or the like into the memory 22 and sequentially executes a plurality of instructions. The memory 22 is a volatile memory such as a semiconductor memory, and is a storage device to and from which the processor 21 can directly read and write data.

[0016] The storage device 23 is a non-transitory storage medium readable by an information processing device, such as a hard disk drive, solid state drive, or USB (Universal Serial Bus) memory, and stores computer programs and data. The output device 24 is a liquid crystal display, an organic EL display, a speaker, or the like, and outputs data according to instructions from the processor 21. The input device 25 is a mouse, keyboard, touch panel, microphone, or the like, and receives instructions from a user and inputs data. The communication device 26 is a wireless LAN (Local Area Network) device, a wired LAN cable, or the like, and transmits and receives data to and from other information processing devices wirelessly or via a wire.

[0017] Like the server 2, the user terminal 3 is a general-purpose information processing device, such as a notebook PC (short for "personal computer"), desktop PC, smartphone, or tablet terminal. The user terminal 3 has a processor 31, memory 32, storage device 33, output device 34, input device 35, and communication device 36, which are connected via a bus 37. The processor 31 to bus 37 of the user terminal 3 are similar to the processor 21 to bus 27 of the server 2, and therefore detailed description thereof will be omitted.

[0018] The first LLM server 4a and the second LLM server 4b are servers equipped with large-scale language models used in the field of natural language processing. Large-scale language models are language models constructed by learning large amounts of data using deep learning. The first LLM server 4a and the second LLM server 4b are generative AI (artificial intelligence) servers that generate response sentences in response to instruction sentences (hereinafter referred to as "prompts"), such as ChatGPT (OpenAI), Gemini (Google), Claude (Anthropique), and Copilot (Microsoft). Hereinafter, the first LLM server 4a is referred to as a generative AI server capable of executing complex tasks with high accuracy, and the second LLM server 4b is referred to as a generative AI server with high processing speed and low usage fees.

[0019] 2 is a block diagram showing the functions of the question and answering system 1. The server 2 has the functions of the storage unit 41, estimation unit 43, extraction unit 44, classification unit 45, and result creation unit 46 of the question and answering system 1. The user terminal 3 has the functions of the reception unit 42 and presentation unit 47 of the question and answering system 1.

[0020] The storage unit 41 stores Q&A data including pairs of question sentences and answer sentences. The storage unit 41 also stores a knowledge graph of the Q&A data. A knowledge graph is a graph structure that represents "entities" such as people, places, and things, and the relationships between them, and has a one-to-one correspondence with the Q&A data.

[0021] The reception unit 42 receives input of a user search query from a user. The reception unit 42 may receive the user search query by text input or by voice input. In the case of voice input, the reception unit 42 converts the voice data into text data (character data) using a known voice recognition technology.

[0022] The estimation unit 43 estimates the information desired by the user from the user search statement and creates an estimated search statement. The estimation unit 43 also estimates the format of an answer to the estimated search statement, leaving blank any items that require fact-checking, and creates an answer-style sentence. The answer-style sentence is a sentence in the format of an answer to the estimated search statement, leaving blank any items that require fact-checking, such as the amount, name, and regulations.

[0023] The extraction unit 44 refers to the storage unit 41, extracts Q&A data based on at least the estimated search query, and creates an extracted dataset. A dataset means a collection of one or more data of the same type. The extracted dataset is a collection of one or more extracted Q&A data.

[0024] The classification unit 45 refers to the memory unit 41 as necessary and classifies the extracted datasets created by the extraction unit 44 into at least either answer candidate datasets that are likely to contain an answer to the estimated search query, or related datasets other than the answer candidate datasets.

[0025] The result creation unit 46 references the storage unit 41 as necessary and creates search result sentences for the estimated search query based on at least a part of the answer candidate data set. The search result sentences contain answers to the questions the user wants to know.

[0026] The presentation unit 47 presents the search result sentences created by the result creation unit 46 to the user. The presentation unit 47 may output the search result sentences as text or as audio. In the case of audio output, the presentation unit 47 converts text data (character data) into audio data using a known voice synthesis technique.

[0027] In the above description, the server 2 includes the memory unit 41, the estimation unit 43, the extraction unit 44, the classification unit 45, and the result creation unit 46, and the user terminal 3 includes the reception unit 42 and the presentation unit 47. However, this is not limited to this. For example, the server 2 may include only the memory unit 41, and the user terminal 3 may include the reception unit 42, the estimation unit 43, the extraction unit 44, the classification unit 45, the result creation unit 46, and the presentation unit 47. Furthermore, the question and answering system 1 may be a standalone system in which the functions of the server 2 and the user terminal 3 are implemented in a single unit. Furthermore, the question and answering system 1 may include the large-scale language model in the server 2, and not include the first LLM server 4a and the second LLM server 4b.

[0028] Fig. 3 is a diagram showing an example of QA data. The QA data 50 is data on frequently asked questions (FAQ) in a given organization or field, or data created from given information using generative AI technology. As shown in Fig. 3, the QA data 50 includes the following items: an identification number 51, a title 52, a question 53, an answer 54, and tag information 55.

[0029] The identification number 51 is a number that uniquely identifies the QA data 50. The title 52 is text data that indicates an overview of the QA data 50. The question 53 is text data that indicates the content of the question. The answer 54 is text data that indicates the content of the answer to the question 53. The tag information 55 is information that serves as a mark for organizing the QA data 50, and is a type of metadata or attribute information.

[0030] The question and answer pairs included in the QA data 50 are not limited to a one-to-one relationship, but may also have a many-to-many relationship. The QA data 50 with the identification number 51 of "1" includes three sentences in the question 53 and five sentences in the answer 54. The tag information 55 includes keywords related to the title 52, the question 53, or the answer 54.

[0031] FIG. 4 is a diagram showing an example of a knowledge graph. The knowledge graph 56 is created based on QA data 50. There is a one-to-one relationship between the QA data 50 and the knowledge graph 56. The server 2 extracts entities (substances) and relationships from the QA data 50. The entities are people, places, things, events, ideas, etc., and mainly correspond to subject and object words contained in the QA data 50. The relationships are connections between entities and mainly correspond to predicate words contained in the QA data 50. The server 2 treats the extracted entities as nodes (points) and connects the nodes (points) with edges (lines) based on the extracted relationships in accordance with a predefined schema (rules, framework), thereby constructing the graph structure shown in FIG. 4, which is the knowledge graph 56. The knowledge graph 56 in FIG. 4 has 35 nodes.

[0032] 5 is a flowchart showing the processing flow of the question and answer system. As shown in Fig. 5, the accepting unit 42 of the user terminal 3 accepts an input of a user search query from a user (step S1), and transmits the user search query to the server 2 (step S2).

[0033] 6 is a diagram illustrating the user search query reception process performed by the reception unit. The reception unit 42 of the user terminal 3 displays a user search query reception screen 60 on the output device 24, such as a liquid crystal display. The user search query reception screen 60 has a text box 61 for inputting a user search query and a question button 62 for instructing the start of a search. When the user uses the input device 25 to input a user search query in the text box 61 and presses the question button 62, the reception unit 42 of the user terminal 3 transmits the text data entered in the text box 61 to the server 2 as a user search query 63.

[0034] In the example of Figure 6, text data of "How to get a Shinkansen ticket" is entered in a text box 61. This text data is sent to the server 2 as a user search statement 63. The user search statement 63 may be one or more words, as in the example of Figure 6, or may be a sentence. The user search statement 63 in Figure 6 is "How to get a Shinkansen ticket."

[0035] Next, when the estimation unit 43 of the server 2 receives the user search statement 63 from the user terminal 3 (step S3), it estimates the information desired by the user and creates an estimated search statement (step S4).

[0036] 7 is a diagram illustrating the process of creating an estimated search statement by the estimation unit. The estimation unit 43 of the server 2 sends a prompt 71 to the first LLM server 4a (large-scale language model) to instruct it to estimate information desired by the user from the user search statement 63. The estimation unit 43 of the server 2 then receives a response statement 72 from the first LLM server 4a (large-scale language model) in response to the prompt 71, and creates an estimated search statement 73 from the response statement 72.

[0037] Prompt 71 in Figure 7 instructs the system to predict up to three things the user wants to know from user search statement 63 (= search statement entered by the user). The estimated search statements 73 in Figure 7 are, first, "How to purchase a Shinkansen ticket," second, "Procedure for reserving a Shinkansen ticket," and third, "Things to note when purchasing a Shinkansen ticket."

[0038] Next, the extraction unit 44 of the server 2 determines extraction keywords (=keywords for extracting the QA data 50) from the estimated search query 73 created in step S4 (step S5).

[0039] 8 is a diagram illustrating the extraction keyword determination process performed by the extraction unit. The extraction unit 44 of the server 2 sends a prompt 74 to the second LLM server 4b (large-scale language model) to instruct it to extract keywords from an estimated search statement 73. The extraction unit 44 of the server 2 then receives a response statement 75 from the second LLM server 4b (large-scale language model) in response to the prompt 74, and determines extraction keywords 76 from the response statement 75. The extraction keywords 76 are included in the response statement 75.

[0040] The prompt 74 in Figure 8 instructs the user to consider what keywords to use to gather information from the estimated search statement 73 (= what the user wants to know, estimated from the search statement entered by the user). The extraction keywords 76 in Figure 8 are the first sentence "How to purchase Shinkansen tickets," the second sentence "Procedure for reserving Shinkansen tickets," and the third sentence "Points to note when purchasing Shinkansen tickets."

[0041] Next, the extraction unit 44 of the server 2 extracts the QA data 50 from the storage unit 41 based on the extraction keywords determined in step S5, and creates an extracted data set (step S6).

[0042] 9 is a flowchart showing the flow of the extraction data set creation process by the extraction unit. The extraction unit 44 of the server 2 extracts the first two keywords from the user search statement 63 (step S21). In the user search statement 63 of FIG. 6, the first two keywords are "Shinkansen" (bullet train) and "ticket."

[0043] Next, the extraction unit 44 searches the tag information 55 of the QA data 50 stored in the storage unit 41 using the following search condition A, and creates the search results as data set A (step S22). <Search criteria A> Meet one of the following two conditions: (A1) The tag information has cos similarity with both of the two keywords extracted from the user search sentence, and is equal to or greater than a threshold value. (A2) The tag information includes both of the two keywords extracted from the user search sentence.

[0044] Cosine similarity is a measure of the similarity between two vectors, and is the cosine value of the angle between the two vectors. Cosine similarity is calculated by dividing the inner product of two vectors by the magnitudes of the two vectors, and is a value normalized to the range of -1 to 1. The extraction unit 44 converts two keywords and tag information 55 extracted from a user search query into distributed representations (vectors) using a known document embedding model (e.g., SBERT, etc.), and calculates cosine similarity. The same applies to the calculation of cosine similarity below.

[0045] Next, the extraction unit 44 searches the tag information 55 of the QA data 50 stored in the storage unit 41 using the following search condition B, and creates the search results as data set B (step S23). <Search condition B> Meet one of the following two conditions: (B1) The tag information has a cos similarity with only one of the two keywords extracted from the user search sentence that is equal to or greater than a threshold value. (B2) The tag information includes only one of two keywords extracted from the user search sentence.

[0046] Next, the extraction unit 44 selects the first two keywords from the extraction keywords 76 (step S24). If the extraction keywords 76 include three sentences as in Fig. 8, the first two keywords are extracted for each sentence. In the case of the extraction keywords 76 in Fig. 8, the first two keywords for all three sentences are "Shinkansen" (bullet train) and "ticket."

[0047] Next, the extraction unit 44 searches the tag information 55 of the QA data 50 stored in the storage unit 41 using the following search criteria C, and creates the search results as dataset C (step S25). If the extraction keywords 76 include multiple sentences, the extraction unit 44 repeats the search as many times as the number of sentences included in the extraction keywords 76, combines all the search results, and creates the dataset C. <Search condition C> Meet one of the following two conditions: (C1) The tag information has cos similarity with both of the two keywords selected from the extraction keywords that is equal to or greater than a threshold value. (C2) The tag information includes both of two keywords selected from the extraction keywords.

[0048] Next, the extraction unit 44 searches the tag information 55 of the QA data 50 stored in the storage unit 41 using the following search criteria D, and creates the search results as a dataset D (step S26). If the extraction keywords 76 include multiple sentences, the extraction unit 44 repeats the search as many times as the number of sentences included in the extraction keywords 76, combines all the search results, and creates the dataset D. <Search condition D> Meet one of the following two conditions: (D1) The tag information has a cos similarity with only one of two keywords selected from the extraction keywords that is equal to or greater than a threshold value. (D2) The tag information includes only one of two keywords selected from the extraction keywords.

[0049] Next, the extraction unit 44 combines the data sets A and B into one to create a first extracted data set (step S27), and also combines the data sets C and D into one to create a second extracted data set (step S27).

[0050] The first extracted dataset is extracted using "both of the two keywords" of search condition A and search condition C, i.e., an AND condition. On the other hand, the second extracted dataset is extracted using "only one of the two keywords" of search condition C and search condition D, i.e., an OR condition. Therefore, it is considered that the first extracted dataset is more likely to include Q&A data 50 having information that the user wants to know than the second extracted dataset.

[0051] In the above description, the extraction unit 44 extracted the QA data 50 based on both the user search sentence 63 and the estimated search sentence 73 (extraction keywords 76), but the QA data 50 may be extracted based on only the estimated search sentence 73.

[0052] Returning to the explanation of Fig. 5, the estimation unit 43 of the server 2 next estimates the format of a response that leaves blank the contents that need to be confirmed for the estimated search statement 73, and creates a response-format sentence (step S7).

[0053] 10 is a diagram illustrating the process of creating an answer-format sentence by the estimation unit. The estimation unit 43 of the server 2 sends a prompt 81 to the first LLM server 4a (large-scale language model) instructing it to estimate the format of an answer by leaving blank any content that needs to be fact-checked for the estimated search statement 73. The extraction unit 44 of the server 2 then receives a response sentence 82 from the first LLM server 4a (large-scale language model) in response to the prompt 81, and creates an answer-format sentence 83 from the response sentence 82.

[0054] Prompt 81 in Figure 10 instructs the user to predict up to three possible answer formats for estimated search statement 73, with the contents requiring fact-checking left blank. More specifically, prompt 81 in Figure 10 instructs the user to substitute "[---]", i.e., leave the contents requiring fact-checking blank, among the answer formats for estimated search statement 73. Answer format sentences 83 in Figure 10 are: first, "Shinkansen tickets can be purchased at [---]," second, "Shinkansen ticket reservations can be made from [---]," and third, "Shinkansen tickets can be purchased from [---] days in advance."

[0055] Next, the classification unit 45 of the server 2 determines classification keywords from the reply-format sentence 83 created in step S7 (step S8).

[0056] 11 is a diagram illustrating the classification keyword determination process performed by the classification unit. The classification unit 45 of the server 2 sends a prompt 84 to the second LLM server 4b (large-scale language model) to instruct it to extract keywords from a response-form sentence 83. The classification unit 45 of the server 2 then receives a response sentence 85 from the second LLM server 4b (large-scale language model) in response to the prompt 84, and determines classification keywords 86 from the response sentence 85. The classification keywords 86 are included in the response sentence 85.

[0057] The prompt 84 in Figure 11 instructs the user to think about what keywords to use to gather information from the answer-style sentence 83 (= expressions expected to be included in the FAQ). The classification keywords 86 in Figure 11 are "Shinkansen ticket purchase" for the first sentence, "Shinkansen ticket reservation" for the second sentence, and "Shinkansen ticket available for purchase" for the third sentence.

[0058] Next, the classification unit 45 of the server 2 classifies the first extracted dataset into answer candidate datasets and related datasets based on the classification keywords 86 and the knowledge graph 56 determined in step S8 (step S9).

[0059] 12 is a flowchart showing the flow of the first extracted dataset classification process by the classification unit. The classification unit 45 of the server 2 sets the variable of the classification keyword 86 to k (k=1, 2, . . . , m) and assigns 1 to k (step S31). The variable k identifies each sentence included in the classification keyword 86. In the case of the classification keyword 86 in FIG. 11, m=3.

[0060] Next, the classification unit 45 classifies the Q&A data 50 included in the first extracted data set into one of the following two types: answer candidates and related information (step S32). <Answer candidates> Both of the following conditions are met: (E1) All keywords [i] (i = 1, 2, . . . , n) included in the classification keyword [k] have a cos similarity with any node in the knowledge graph that corresponds one-to-one to the QA data that is equal to or exceeds a threshold. (E2) Similarity score s is ranked 1st to 3rd. <Related Information> Q&A data other than answer candidates

[0061] In (E1), "all keywords [i] (i = 1, 2, ..., n) included in classification keyword [k]" means, in the case of the first sentence of classification keyword 86 in Figure 11 (= classification keyword [1]), the three keywords (n = 3) are keyword [1] = "Shinkansen", keyword [2] = "ticket", and keyword [3] = "purchase".

[0062] In (E1), "any node in the knowledge graph 56 that corresponds one-to-one to the QA data 50" refers to the 35 sentences attached to each node (point), such as "Private," "In use," and "Business use," in the case of the knowledge graph 56 in Figure 4.

[0063] For example, in the case of the first sentence of the classification keyword 86 in Figure 11, the classification unit 45 calculates the cosine similarity between each of the keywords [i], "Shinkansen", "ticket", and "purchase", and all nodes in the knowledge graph 56 that correspond one-to-one to the QA data 50. Then, the classification unit 45 stores the maximum value of the cosine similarity for each keyword [i] in the memory 22 or the storage device 23. If the maximum value of the cosine similarity for all keywords [i] is equal to or greater than a threshold, the classification unit 45 determines that the QA data 50 satisfies the condition (E1).

[0064] The similarity score s in (E2) is defined as follows: If the number of keywords is n (n is a natural number greater than or equal to 2), the keyword variable is i (i = 1, 2, . . . , n), the distance of the shortest path between keyword [i] and keyword [i+1] (the distance between adjacent nodes is 1) is p(i), the maximum cosine similarity of each keyword [i] is c(i), and the set of c(i) is C, then the similarity score s is defined by the following formula.

[0065]

number

[0066] For example, in the case of the first sentence of classification keyword 86 in Fig. 11, the distance p(1) of the shortest path between keyword [1] = "Shinkansen" and keyword [2] = "ticket" is p(1) = 2, with reference to Fig. 4. Also, the distance p(2) of the shortest path between keyword [2] = "ticket" and keyword [3] = "purchase" is p(2) = 1, with reference to Fig. 4.

[0067] Next, the classification unit 45 assigns k+1 to the variable k of the classification keyword 86 (step S34), and checks whether the value of the variable k after the assignment is equal to or less than m (= the number of sentences of the classification keyword 86) (step S34). If the value of the variable k is equal to or less than m (Yes in step S34), the classification unit 45 repeats from step S32, and if the value of the variable k is greater than m (No in step S34), the classification unit 45 proceeds to step S35.

[0068] Next, the classification unit 45 combines all the QA data 50 classified as answer candidates to create an answer candidate data set (step S35). The classification unit 45 also combines all the QA data 50 classified as related information to create a related information data set (step S36).

[0069] Returning to the description of Fig. 5, the result creating unit 46 then sorts the QA data 50 included in the answer candidate dataset, the related information dataset, and the second extracted dataset for each dataset (step S10).

[0070] The result creation unit 46 sorts the QA data 50 for each dataset in descending order of values ​​based on the cosine similarity between the user search query 63, the estimated search query 73, and the answer-format sentence 83 and the QA data 50 included in the answer candidate dataset, the related information dataset, and the second extracted dataset. The result creation unit 46 calculates the cosine similarity between each of the user search query 63, the estimated search query 73, and the answer-format sentence 83 for one QA data 50. Therefore, three cosine similarities are calculated for one QA data 50. For example, the result creation unit 46 considers the simple sum, weighted sum, product, etc. of the three cosine similarities as a value based on the cosine similarity, and sorts the QA data 50 in descending order of this value.

[0071] Furthermore, the result creation unit 46 may sort the QA data 50 for each dataset in descending order of a value based on the cosine similarity between any one or two of the user search statements 63, the estimated search statements 73, and the answer-format sentences 83 and the QA data 50 included in the answer candidate dataset, the related information dataset, and the second extracted dataset. That is, the result creation unit 46 sorts the QA data 50 included in the answer candidate dataset, the related information dataset, and the second extracted dataset based on at least one of the user search statements 63, the estimated search statements 73, and the answer-format sentences 83.

[0072] Next, the result creation unit 46 of the server 2 creates a search result sentence for the estimated search sentence 73 based on the answer candidate dataset, the related information dataset, and the second extracted dataset (step S11).

[0073] 13 is a diagram illustrating the detection result creation process performed by the result creation unit. The result creation unit 46 of the server 2 sends a prompt 87 to the first LLM server 4a (large-scale language model) instructing it to provide an answer to the estimated search statement 73 using at least a part of the answer candidate dataset as reference information. The result creation unit 46 of the server 2 then receives a response statement 88 from the first LLM server 4a (large-scale language model) in response to the prompt 87, and creates a search result statement from the response statement 88.

[0074] The prompt 87 in Figure 13 instructs the system to provide an appropriate answer based on the estimated search statement 73 (= what the user wants to know) and reference information (= the top three Q&A data 50 from the answer candidate dataset, related information dataset, and second extracted dataset). The response statement 88 in Figure 13 begins with "The method for purchasing Shinkansen tickets can be summarized as follows" and includes answer content divided into five items. The five items are "1. Please purchase using a corporate card," "2. The following purchase options are available:," "3. Points to note:," "4. When paying:," and "5. If reservation changes occur frequently:." These five items are thought to be what the user wants to know.

[0075] FIG. 14 is a diagram showing an example of a search result sentence. The result creation unit 46 of the server 2 creates a search result sentence 89 including the content of a response sentence 88 and the alignment results of the answer candidate dataset, the related information dataset, and the second extracted dataset. In FIG. 14, range 91 represents the content of the response sentence 88, range 92 represents the alignment results of the answer candidate dataset, range 93 represents the alignment results of the related information dataset, and range 94 represents the alignment results of the second extracted dataset. When the user clicks [···] in range 92, range 93, or range 94 using the input device 25, the title 52, question sentence 53, and answer sentence 54 of the Q&A data 50 are displayed.

[0076] Next, the result creation unit 46 of the server 2 transmits the search result sentence 89 created in step S11 to the user terminal 3 (step S12). Upon receiving the search result sentence 89 from the server 2 (step S13), the user terminal 3 presents the search result sentence 89 to the user (step S14).

[0077] As described above, the question and answering system 1 includes a storage unit 41 that stores Q&A data including pairs of question and answer sentences, a receiving unit 42 that receives a user search query from a user, an estimation unit 43 that estimates the information the user desires from the user search query and creates an estimated search query, and also estimates the format of an answer that leaves blank fields for content that needs to be fact-checked for the estimated search query and creates an answer-format sentence, an extraction unit 44 that extracts Q&A data based on at least the estimated search query and creates an extracted dataset, a classification unit 45 that classifies the extracted dataset into either an answer candidate dataset that is likely to contain at least an answer to the estimated search query or a related dataset other than the answer candidate dataset, a result creation unit 46 that creates search result sentences for the estimated search query based on at least a portion of the answer candidate dataset, and a presentation unit 47 that presents the search result sentences to the user. This allows the question and answering system 1 to present search result sentences that include direct answers to the user's questions.

[0078] The estimation unit 43 sends a first prompt to the large-scale language model instructing it to estimate information desired by the user from the user search query, receives a first response sentence from the large-scale language model in response to the first prompt, and creates an estimated search query from the first response sentence. The estimation unit 43 also sends a second prompt to the large-scale language model instructing it to estimate an answer format for the estimated search query, leaving fields for items that require fact-checking blank, receives a second response sentence from the large-scale language model in response to the second prompt, and creates the answer-format sentence from the second response sentence. This makes it possible to estimate what the user wants to know and the format of the answer to that item. In particular, because the answer format is estimated to leave fields for items that require fact-checking blank, it is possible to avoid the inclusion of incorrect information.

[0079] The result creation unit 46 also sends a third prompt to the large-scale language model instructing it to provide an answer to the estimated search query using at least a part of the answer candidate dataset as reference information, receives a third response sentence from the large-scale language model in response to the third prompt, and creates a search result sentence from the third response sentence. This allows the search result sentence to be created based on Q&A data that is likely to include an answer to the estimated result of what the user wants to know.

[0080] The storage unit 41 also stores a knowledge graph of the QA data. The extraction unit 44 sends a fourth prompt to the large-scale language model instructing it to extract keywords from the estimated search sentence, receives a fourth response sentence from the large-scale language model in response to the fourth prompt, searches the QA data using first search criteria based on the extraction keywords included in the fourth response sentence, and creates a first extracted dataset from the search results. The classification unit 45 sends a fifth prompt to the large-scale language model instructing it to extract keywords from answer-style sentences, receives a fifth response sentence from the large-scale language model in response to the fifth prompt, and classifies the first extracted dataset into an answer candidate dataset and a related dataset based on the classification keywords and the knowledge graph included in the fifth response sentence. This enables accurate classification of the answer candidate dataset and the related dataset.

[0081] The extraction unit 44 also searches for Q&A data using second search criteria based on the extraction keywords included in the fourth response sentence, and creates a second extracted dataset from the search results. The result creation unit 46 includes some of the Q&A data included in the answer candidate dataset, the related dataset, and the second extracted dataset in the reference information for the third prompt. This makes it possible to create search result sentences that include alternative answers, even if the answer candidate dataset alone cannot create search result sentences that include sufficient answers.

[0082] While the preferred embodiments of the question and answer system and the like according to the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications and alterations within the scope of the technical ideas disclosed herein, and it is understood that such modifications and alterations also fall within the technical scope of the present invention. [Explanation of symbols]

[0083] 1. Question and answer system 2...Server 3...User terminal 4a………First LLM server (large-scale language model) 4b………Second LLM server (large-scale language model) 5. Network 41……Storage section 42...Reception 43……Estimation part 44……Extraction part 45……Classification section 46...Results Creation Section 47……Presentation part

Claims

1. a storage unit that stores Q&A data including a set of question sentences and answer sentences; a reception unit that receives an input of a user search statement from a user; an estimation unit that estimates information desired by the user from the user search statement, creates an estimated search statement, estimates a format of an answer that leaves blank any content that needs to be confirmed for the estimated search statement, and creates an answer format statement; an extraction unit that extracts the QA data based on at least the estimated search query and creates an extracted dataset; a classification unit that classifies the extracted datasets into at least either answer candidate datasets that are likely to contain an answer to the estimated search query or related datasets other than the answer candidate datasets; a result generation unit that generates a search result sentence for the estimated search query based on at least a part of the answer candidate dataset; a presentation unit that presents the search result sentences to the user; A question and answer system comprising:

2. The estimation unit transmits a first prompt to the large-scale language model instructing it to estimate information desired by the user from the user search statement, receives a first response sentence from the large-scale language model in response to the first prompt, creates the estimated search statement from the first response sentence, and transmits a second prompt to the large-scale language model instructing it to estimate an answer format in which content that needs to be fact-checked for the estimated search statement is left blank, receives a second response sentence from the large-scale language model in response to the second prompt, and creates the answer-format sentence from the second response sentence.

2. The question and answer system according to claim 1.

3. the result creation unit sends a third prompt to the large-scale language model instructing the large-scale language model to provide an answer to the estimated search sentence using at least a part of the answer candidate dataset as reference information, receives a third response sentence from the large-scale language model in response to the third prompt, and creates the search result sentence from the third response sentence.

3. The question and answer system according to claim 2.

4. the storage unit stores a knowledge graph of the QA data; the extraction unit sends a fourth prompt to the large-scale language model instructing it to extract keywords from the estimated search sentence, receives a fourth response sentence from the large-scale language model in response to the fourth prompt, searches the QA data using first search conditions based on the extraction keywords included in the fourth response sentence, and creates a first extraction data set from the search results; The classification unit transmits a fifth prompt to the large-scale language model, the fifth prompt instructing the large-scale language model to extract keywords from the answer-form sentence, receives a fifth response sentence from the large-scale language model in response to the fifth prompt, and classifies the first extracted dataset into the answer candidate dataset and the related dataset based on classification keywords included in the fifth response sentence and the knowledge graph.

4. The question and answer system according to claim 3.

5. the extraction unit searches the Q&A data using a second search condition based on the extraction keyword included in the fourth response sentence, and sets the search result as a second extracted data set; The result creation unit includes a part of the Q&A data included in the answer candidate dataset, the related dataset, and the second extracted dataset in the reference information related to the third prompt.

5. The question and answer system according to claim 4.

Images