Question answering device and question answering method

Abstract

This enables highly accurate answers to questions. [Solution] The question answering device 100 includes a question answering unit 113 that refers to answer support information (answer support information database 140) in which questions and answers to those questions are stored in association, and outputs an answer corresponding to a question in the answer support information that has a similarity to the question issued by the user of a first predetermined similarity or higher, as an answer to the question issued by the user. If there is no question in the answer support information that has a similarity to the question issued by the user of a first predetermined similarity or higher, the question answering unit 113 may send a prompt including the question issued by the user to the language model server 210, obtain and output an answer to that question, and add the question issued by the user and the outputted answer in association to the answer support information.

Description

Technical Field

[0001] The present invention relates to a question answering apparatus and a question answering method using a language model.

Background Art

[0002] In recent years, in customer support services, it has become common to introduce chatbots that automatically answer customer questions. Questions from customers and answers to those questions are accumulated and summarized as FAQs (Frequently Asked Questions) and used to improve the responsiveness of chatbots.

[0003] As a technology for answering questions from customers, there is an information processing apparatus described in Patent Document 1. This information processing apparatus includes extraction means for extracting questions and answers from a document. The information processing apparatus also includes search means for performing a search on the document and the extracted questions and answers using a search query received from a user. Further, the information processing apparatus includes display control means for displaying the document searched by the search means and for displaying the questions and answers searched by the search means in association with the document that is the source of extraction of the questions and answers.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The information processing apparatus described in Patent Document 1 extracts questions and answers (FAQs) by extracting sentences in the form of question sentences or answer sentences from a document, and outputs those with relatively high search scores among the extracted FAQs. For this reason, if the score is high among the extracted FAQs, an answer may be output even if it deviates from the search query (question). This invention was made in view of the above background, and aims to provide a question answering device and a question answering method that enable highly accurate answers to questions. [Means for solving the problem]

[0006] To solve the above-mentioned problems, the question answering device according to the present invention includes a question answering unit that refers to answer support information, in which questions and answers to those questions are stored in association with each other, and outputs an answer corresponding to a question in the answer support information that has a similarity to the question posed by the user of a first predetermined similarity or higher, as the answer to the question posed by the user. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a question-answering device and a question-answering method that enable highly accurate answers to questions. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments. [Brief explanation of the drawing]

[0008] [Figure 1] This is a functional block diagram of the question answering device according to this embodiment. [Figure 2] This is a data configuration diagram of the answer support information database according to this embodiment. [Figure 3] This is a data configuration diagram of the user evaluation cache according to this embodiment. [Figure 4] This figure shows the administrator screen according to this embodiment. [Figure 5] This is a flowchart of the startup process according to this embodiment. [Figure 6] This figure shows the question answer generation prompt according to this embodiment. [Figure 7] This is a flowchart of the question answering process according to this embodiment. [Figure 8] This figure shows the user screen according to this embodiment. [Figure 9] This is a flowchart of the response acquisition process according to this embodiment. [Figure 10] This figure shows the response generation prompt according to this embodiment. [Figure 11] This is a flowchart of the user evaluation process according to this embodiment. [Figure 12] This is a flowchart of the response support information update process according to this embodiment. [Figure 13] This figure shows the response regeneration prompt according to this embodiment. [Figure 14] This figure shows the additional prompts for answering questions according to this embodiment. [Figure 15] This is a hardware configuration diagram showing an example of a computer that implements the functions of the question answering device according to the above embodiment. [Modes for carrying out the invention]

[0009] ≪Overview of the Question Answering Device≫ The following describes the outline of a question answering device in an embodiment for carrying out the present invention. The question answering device stores the results of extracting questions and answers related to the content from a document, along with the source document, as answer support information (see the answer support information database 140 in Figure 2 below). When a question is received from a user, if there is a question in the answer support information that has a similarity to the question at or above a first predetermined similarity level, the answer to the question is returned to the user. If there is no question in the answer support information with a similarity of at or above the first predetermined similarity level, an answer is generated using a language model server based on the document and returned to the user. The first predetermined similarity level is a very high degree of similarity that can be considered to match the content of the question.

[0010] The question answering device may record the user's evaluation value for the answer in the answer support information. In this case, the question answering device may return the answer to the user if the question in the answer support information has a similarity score of at least a first predetermined similarity score and the evaluation value is at least a predetermined evaluation value.

[0011] When there is no question with a similarity equal to or higher than a first predetermined similarity and an evaluation value equal to or higher than a predetermined evaluation value, the question answering device may generate an answer using a language model server based on the source document of the question with a similarity equal to or higher than a second predetermined similarity and an evaluation value equal to or higher than the predetermined evaluation value. When there is no such question, the question answering device may generate an answer using a language model server based on a document similar to the question issued by the user. Note that the second predetermined similarity is lower than the first predetermined similarity.

[0012] According to such a question answering device, the user can obtain a highly accurate answer to the question. By newly generating an answer according to the similarity of the question and the evaluation value of the answer, it can be expected that the accuracy of the answer will be improved.

[0013] ≪Configuration of Question Answering Device≫ FIG. 1 is a functional block diagram of a question answering device 100 according to the present embodiment. The question answering device 100 is a computer and includes a control unit 110, a storage unit 120, and a communication unit 180. The communication unit 180 includes a communication device and can transmit and receive data to and from a language model server 210, an administrator terminal 230, and a user terminal 240.

[0014] The question answering device 100 transmits a prompt, which is a question / instruction described in text (language information), to the language model server 210. The language model server 210 generates an answer to the prompt using a language model (for example, a large language model) and returns it to the question answering device 100.

[0015] The administrator terminal 230 is a terminal used by an administrator of the question answering device 100. The administrator uses an administrator screen 510 (see FIG. 4 described later) to instruct the setting of the question answering device 100 and the update of answer support information.

[0016] The user terminal 240 is a terminal used by a user (questioner) of the question answering device 100. The user uses a user screen 520 (see FIG. 8 described later) to ask a question to the question answering device 100. The user also uses the user screen 520 to evaluate the answer to the question.

[0017] ≪Question answering device: storage section≫ The memory unit 120 is composed of memory devices such as ROM (Read Only Memory), RAM (Random Access Memory), and SSD (Solid State Drive). The memory unit 120 stores a document database 130, a response support information database 140, a user evaluation cache 150, configuration information 121, a vectorized model 122, and a program 128. The program 128 contains a description of the processing to be executed by the functional unit provided in the control unit 110, which will be described later. The various contents of the memory unit 120 may be stored in an external storage device such as a cloud server and read as needed.

[0018] ≪Memory Section: Document Database≫ The document database 130 contains documents that serve as the basis for answers to user questions. In other words, the content of the questions answered by the question answering device 100 is the content of the documents in the document database 130.

[0019] ≪Memory Section: Answer Support Information Database≫ Figure 2 is a data structure diagram of the response support information database 140 according to this embodiment. The response support information database 140 is, for example, tabular data, where one row (record) represents a question and an answer. More specifically, a record includes columns (attributes) for identification information 141, generation method 142, question 143, answer 144, document 145, and evaluation 146.

[0020] Identification information 141 (labeled "ID" in Figure 2) is the identification information for the record. This identification information can also be considered the identification information for the query.

[0021] The generation method 142 is a record generation method and is either "A" or "B". "A" indicates that the record was generated by the answer support information generation unit 112 or the answer support information update unit 114, which will be described later. Records of "A" are records that existed in the answer support information database 140 before the user's question was received. "B" indicates that the record was generated by the question answer unit 113, which will be described later. Records of "B" are records that include the answer generated after the user's question was received.

[0022] The relationship between Question 143, Answer 144, and Document 145 is that Question 143 and Answer 144 relate to the content of the document shown in Document 145 (Document Name / Identification Information). Rating 146 is the user's evaluation of the accuracy of response 144. Rating 146 is on a 5-point scale, for example, from 1 to 5 (see Figure 8 below), with higher numbers indicating a better evaluation. "Unrated" indicates that there are no questions from users and the response has not been evaluated. The evaluation is also referred to as the user evaluation.

[0023] As explained above, the response support information (see response support information database 140) is associated with and stored the user's evaluation value (see evaluation 146) for the response 144. The answer support information includes document 145, which is associated with question 143, and is stored in the associated memory.

[0024] ≪Memory Unit: User Evaluation Cache≫ Figure 3 is a data configuration diagram of the user evaluation cache 150 according to this embodiment. The user evaluation cache 150 stores a predetermined number of recent user evaluations. The predetermined number is the size of the user evaluation cache 150 set by the administrator (see area 513 in Figure 4, described later).

[0025] ≪Memory Section: Configuration Information≫ Returning to Figure 1, let's continue the explanation of the memory unit 120. The configuration information 121 stores the settings related to the question answering device 100 that have been set by the administrator. The configuration information 121 includes the size of the user evaluation cache 150 (see Figure 3), as well as setting values ​​related to the similarity of questions and user evaluation, which will be referenced in the answer acquisition process (see Figure 9) described later.

[0026] ≪Memory Unit: Vectorization Model≫ Vectorization model 122 is a machine learning model used to convert text (linguistic information) into (numerical) vectors. The input (explanatory variable) of vectorization model 122 is text, and the output (target variable) is a vector. Texts with similar content are converted into similar vectors. The similarity of the vectors is calculated as cosine similarity. In the following description of the embodiment, the similarity of texts such as questions and documents is the cosine similarity of the vectors after they have been converted using vectorization model 122.

[0027] ≪Question Answering Device: Control Unit≫ The control unit 110 includes a CPU (Central Processing Unit) and comprises an administrator operation reception unit 111, an answer support information generation unit 112, a question answering unit 113, and an answer support information update unit 114. The control unit 110 may also include a GPU (Graphics Processing Unit), an NPU (Neural (network) Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), etc.

[0028] ≪Control Unit: Administrator Operation Reception Unit≫ The administrator operation reception unit 111 receives settings and operations for the question answering device 100 via the administrator screen 510 (see Figure 4), which will be described later. Figure 4 is a diagram showing the administrator screen 510 according to this embodiment. The administrator screen 510 is displayed on the administrator terminal 230 used by the administrator.

[0029] Area 511 of the administrator screen 510 displays the average value of the most recent user ratings in the user rating cache 150 (see Figure 3). In area 512, the answer support information reference level (second predetermined similarity), which is referenced in the answer acquisition process described later (see Figure 9), is set. The answer support information reference level is a threshold related to the similarity between the question posed by the user and question 143 in the answer support information database 140 (see Figure 2). If there is no question in the answer support information database 140 whose similarity to the question posed by the user is equal to or greater than the similarity indicated by the answer support information reference level, a new answer is generated (see step S37 in Figure 9 described later). The similarity indicated by the answer support information reference level is, for example, 0.9 for high, 0.8 for medium, and 0.7 for low. The answer support information reference level (second predetermined similarity) is a lower (smaller) value than the first predetermined similarity.

[0030] In area 513, the size of the user evaluation cache 150 (see Figure 3) is set. When the "Start System" button 514 is pressed, the administrator operation reception unit 111 executes the start process (see Figure 5 below). After the start process, the question answering device 100 executes the question answering process (see Figure 7 below) to receive and answer the user's questions.

[0031] When the "Stop System" button 515 is pressed, the administrator operation reception unit 111 stops the question answering process. In area 516, instructions are given on whether or not to update the response support information database 140. More specifically, when the "Start Update" button is pressed, the response support information update unit 114, described later, starts the response support information update process (see Figure 12, described later).

[0032] Control Unit: Answer Support Information Generation Unit, Question Answering Unit, Answer Support Information Update Unit The response support information generation unit 112 generates (constructs) the response support information database 140 based on the documents in the document database 130. Details of the generation process (see Figure 5) will be described later. The question answering unit 113 answers questions posed by the user. Details of the answering process (see Figure 7) will be described later. The response support information update unit 114 updates the response support information database 140. Details of the update process (see Figure 12) will be described later.

[0033] ≪Starting Process (Answer Support Information Generation Process)≫ Figure 5 is a flowchart of the startup process according to this embodiment. The startup process, including the process of generating answer support information after pressing the "Start System" button 514 (see Figure 4), will be explained with reference to Figure 5. The startup process is a preparatory process that is executed before the question answering device 100 receives and answers a question from the user.

[0034] In step S11, the administrator operation reception unit 111 accepts settings for the response support information reference level (see area 512 in Figure 4) and the size of the user evaluation cache 150 (see area 513). The administrator operation reception unit 111 stores the settings in the setting information 121.

[0035] In step S12, the response support information generation unit 112 starts the process of repeating steps S13 to S15 for each document in the document database 130. Hereinafter, the documents processed in this repeated process will be referred to as the documents to be processed. The target of processing may not be the document itself, but rather chapters, sections, etc., contained within the document.

[0036] In step S13, the answer support information generation unit 112 generates a question and answer generation prompt 410 (see Figure 6 below). Figure 6 is a diagram showing the question and answer generation prompt 410 according to this embodiment. Element 411 of the question and answer generation prompt 410 includes instructions to create a question and answer based on the document content in element 413, according to the format in element 412. Element 412 shows a format in which the question and answer are output on separate lines. Element 413 shows the document to be processed.

[0037] Returning to Figure 5, we continue the explanation of the startup process. In step S14, the answer support information generation unit 112 sends a question and answer generation prompt 410 to the language model server 210 to obtain the question and answer. In step S15, the response support information generation unit 112 stores the questions and answers obtained in step S14 in the response support information database 140. More specifically, the response support information generation unit 112 adds a record to the response support information database 140 and stores the new identification information and the questions and answers obtained in step S14 in the record. The response support information generation unit 112 also stores the name / identification information of the document to be processed in document 145 and sets the evaluation 146 to "Not yet".

[0038] In step S16, the administrator operation reception unit 111 instructs the question answering unit 113 to start the question answering process (see Figure 7 below). The process from steps S12 to S15 is also referred to as the response support information generation process. At the end of the response support information generation process, the method for generating records in the response support information database 140 is "A", and the evaluation is "Not yet completed".

[0039] ≪Question Answer Processing≫ Figure 7 is a flowchart of the question answering process according to this embodiment. The process by which the question answering device 100 receives and answers questions from the user will be explained with reference to Figure 7. In step S21, the question answering unit 113 starts the process of repeating steps S22 to S27.

[0040] In step S22, the question answering unit 113 displays the user screen 520 (see Figure 8 below) on the user terminal 240. Figure 8 is a diagram showing the user screen 520 according to this embodiment. The user enters a question into the text box in area 521 and presses the "Send" button 522. The user terminal 240 then sends the entered question to the question answering device 100.

[0041] Returning to Figure 7, we continue the explanation of the question answering process. In step S23, the question answering unit 113 receives the question sent by the user terminal 240. In step S24, the question answering unit 113 executes an answer acquisition process (see Figure 9 below) to obtain an answer to the question.

[0042] In step S25, the question answering unit 113 sends the answer obtained in step S24 to the user terminal 240. Upon receiving the answer, the user terminal 240 displays the answer in a text box in area 523 of the user screen 520 (see Figure 8). The user evaluates the accuracy of the answer, selects an evaluation result, and presses the "Send" button 524. The user terminal 240 then sends the evaluation result (hereinafter also referred to as evaluation / user evaluation) to the question answering device 100. The evaluation results range from the highest rating of 5, "The answer was very good," to the lowest rating of 1, "The answer was completely useless."

[0043] In step S26, the question answering unit 113 receives the evaluation sent by the user terminal 240. In step S27, the question answering unit 113 performs user evaluation processing (see Figure 11 below) and stores the evaluation in the response support information database 140 (see Figure 2).

[0044] ≪Response Acquisition Process≫ Figure 9 is a flowchart of the response acquisition process according to this embodiment. The details of the response acquisition process (see step S24 in Figure 7) for acquiring answers to questions posed by the user will be explained with reference to Figure 9.

[0045] In step S31, the question answering unit 113 searches the answer support information database 140 (see Figure 2) for records in which the question 143 matches the question posed by the user and which have a high evaluation 146 (labeled "answer support record" in Figure 9). A high evaluation 146 means that the evaluation 146 is equal to or greater than a predetermined threshold evaluation value. Note that if the evaluation 146 is "Not yet", it may also be considered to have a high evaluation. In this embodiment, the predetermined evaluation value is, for example, 4.

[0046] A record question 143 and a question posed by a user are considered to match if their similarity is above a predetermined high threshold. In this embodiment, a match is defined as a record question 143 and a question posed by a user whose similarity is above a first predetermined similarity. The first predetermined similarity is, for example, 0.98. The similarity between two texts, such as a question or a document, is the cosine similarity of the vectors obtained by transforming the text using the vectorization model 122 (see Vectorization Model 122).

[0047] In step S32, the question answering unit 113 proceeds to step S33 if a record is found in step S31 (step S32 → YES). If no record is found (step S32 → NO), the question answering unit 113 proceeds to step S34. In step S33, the question answering unit 113 completes the answer acquisition process by processing the answer 144 of the record found in step S31 as the processing result.

[0048] In step S34, the question answering unit 113 searches the answer support information database 140 for records where the similarity between question 143 and the question posed by the user is equal to or greater than the answer support information reference level, and where the evaluation 146 is high. The answer support information reference level is a setting value (second predetermined similarity) set by the administrator (see area 512 in Figure 4).

[0049] In step S35, the question answering unit 113 proceeds to step S36 if a record is found in step S34 (step S35 → YES). If no record is found (step S35 → NO), the question answering unit 113 proceeds to step S37.

[0050] In step S36, the question answering unit 113 generates an answer based on the document 145 of the record found in step S34 and processes it as the result. More specifically, the question answering unit 113 generates an answer generation prompt 420 (see Figure 10 below) and sends it to the language model server 210 to obtain and generate the answer.

[0051] Figure 10 shows the answer generation prompt 420 according to this embodiment. Element 421 of the answer generation prompt 420 contains instructions to create an answer to the question in element 422 based on the document content in element 423. Element 422 shows the question asked by the user. Element 423 is the content of the document shown in document 145.

[0052] Returning to Figure 9, we will continue the explanation of the response acquisition process. In step S37, the question answering unit 113 generates an answer based on the document in the document database 130 that has the highest similarity to the question posed by the user, and uses this as the processing result. More specifically, the question answering unit 113 generates an answer generation prompt 420 (see Figure 10) and sends it to the language model server 210 to obtain and generate the answer. The element 423 of the answer generation prompt 420 shows the content of the document that has the highest similarity to the question posed by the user.

[0053] As described above, the question answering device 100 includes a question answering unit 113 that refers to answer support information (see answer support information database 140) in which a question 143 and an answer 144 to the question 143 are stored in association, and outputs an answer 144 in the answer support information that corresponds to question 143 and has a similarity to a first predetermined similarity or higher, as an answer to the question asked by the user.

[0054] The question answering unit 113 outputs the answer 144 as the answer to the question posed by the user if the answer 143 in the answer support information has a similarity to the question posed by the user that is equal to or greater than a first predetermined similarity, and the evaluation value (see evaluation 146) of the answer 144 corresponding to the question 143 is equal to or greater than a predetermined evaluation value (see step S33). If the evaluation value is less than the predetermined evaluation value, it sends a prompt (see answer generation prompt 420) containing the question posed by the user to the language model server 210 to obtain and output the answer to that question (see steps S36, S37).

[0055] If there is no question 143 in the answer support information whose similarity to the question posed by the user is equal to or greater than a first predetermined similarity (see step S32 → NO), the question answer unit 113 sends a document 145 related to the question 143 and a prompt containing the question posed by the user (see answer generation prompt 420) to the language model server 210, and if there is a question 143 in the answer support information whose similarity to the question posed by the user is equal to or greater than a second predetermined similarity, and the evaluation value (see evaluation 146) of the answer 144 corresponding to the question 143 is equal to or greater than a predetermined evaluation value (see step S35 → YES), the unit 113 obtains and outputs an answer to the question. Note that the second predetermined similarity score is lower than the first predetermined similarity score.

[0056] If there is no question 143 in the answer support information whose similarity to the question posed by the user is equal to or greater than a first predetermined similarity (see step S32 → NO), the question answer unit 113, if there is a question 143 in the answer support information whose similarity to the question posed by the user is equal to or greater than a second predetermined similarity, and the evaluation value (see evaluation 146) of the answer 144 corresponding to the question 143 is less than a predetermined evaluation value (see step S35 → NO), sends the document with the highest similarity to the question posed by the user and a prompt containing the question posed by the user (see answer generation prompt 420) to the language model server 210 to obtain and output the answer to the question (see step S37).

[0057] <<User Evaluation Processing>> Figure 11 is a flowchart of the user evaluation process according to this embodiment. The details of the user evaluation process (see step S27 in Figure 7) will be explained with reference to Figure 11.

[0058] In step S41, the question answering unit 113 stores the user evaluation received in step S26 (see Figure 7) in the user evaluation cache 150. In step S42, the question answering unit 113 proceeds to step S43 if the result of the answer acquisition process in step S24 (see Figure 9) is answer 144 in the answer support record (step S42 → YES). If the result of the answer acquisition process is not answer 144 in the answer support record (step S42 → NO), the question answering unit 113 proceeds to step S44. The result of the answer acquisition process being answer 144 in the answer support record means that in step S33 (see Figure 9), answer 144 of the record was used as the answer result. The result of the answer acquisition process not being answer 144 in the answer support record means that in steps S36 and S37, the answer generated using the language model server 210 was used as the answer result.

[0059] In step S43, the question answering unit 113 overwrites the evaluation 146 of the record for which the answer was obtained in step S33 with the user evaluation received in step S26. In step S44, the question answering unit 113 adds a record to the answer support information database 140 (see Figure 2). More specifically, the question answering unit 113 adds a record to the answer support information database 140 and stores the new identification information, the question posed by the user, and the answer, which is the result of the answer acquisition process, in the record. The answer support information generation unit 112 also stores the name / identification information of the document used to generate the answer (see element 423 of the answer generation prompt 420 shown in Figure 10) in document 145. The evaluation 146 is the user evaluation received in step S26.

[0060] As explained above, the question answering unit 113 obtains the user's evaluation value for the outputted answer and stores it in the answer support information (see evaluation 146 in Figure 2) (see steps S43 and S44). If there is no question 143 in the answer support information whose similarity to the question posed by the user is equal to or greater than a first predetermined similarity, the question answering unit 113 sends a prompt (see answer generation prompt 420) containing the question posed by the user to the language model server 210, obtains an answer to the question (see steps S36, S37), outputs it (see step S25), associates the question posed by the user with the outputted answer, and adds it to the answer support information (see step S44).

[0061] ≪Response Support Information Update Process≫ Figure 12 is a flowchart of the response support information update process according to this embodiment. The process of updating the response support information database 140 will be explained with reference to Figure 12.

[0062] In step S51, if the update conditions described below are met for the answer support information database 140 (step S51 → YES), the answer support information update unit 114 proceeds to step S52. If the update conditions are not met (step S51 → NO), the answer support information update unit 114 terminates the answer support information update process.

[0063] The update conditions are that the following (1) and (2) are met: (1) User ratings are stored up to the size of the user rating cache 150 (see Figure 3). (2) The time-series changes in user ratings satisfy predetermined conditions. These predetermined conditions include, for example, a decrease and improvement in the variability of user ratings. Time-series changes can be considered as changes in the mean or variance of user ratings divided into a predetermined number or predetermined period.

[0064] Let's assume we divide the user evaluations into three equal parts based on the number of evaluations, creating three periods: Period 1, Period 2, and Period 3. Alternatively, we could divide the user evaluations into three equal parts based on the length of the period. If the variance decreases across Periods 1, 2, and 3, we can conclude that the variability of user evaluations has decreased. Furthermore, if the average increases, we can say that user evaluations have improved. The update conditions may include criteria for the magnitude of the decrease or increase.

[0065] In step S52, the response support information update unit 114 displays the administrator screen 510 (see Figure 4) on the administrator terminal 230. The administrator presses the "Start Update" button or the "Do not update" button in area 516 to indicate whether or not to proceed with the update. In step S53, if the response support information update unit 114 is instructed to update (step S53 → YES), it proceeds to step S54. If it is instructed not to update (step S53 → NO), the response support information update unit 114 finishes the response support information update process.

[0066] In step S54, the response support information update unit 114 determines the document to be updated. The response support information update unit 114 calculates the following score S3 for each document to determine the document to be updated.

[0067] S1 = (Total number of records associated with a document and whose generation method 142 is "B") / (Total number of records whose generation method 142 is "B") S2 = (Highest rating of 146) / (Average of 146 ratings for records associated with the document) S3 = S1 × S2

[0068] Here, the record associated with a document is the record in which document 145 is the document in question. The document score S1 indicates the proportion of questions related to that document. The score S2 indicates the low level of evaluation related to that document (the lower the evaluation, the higher the score S2).

[0069] The response support information update unit 114 will select a predetermined percentage of documents with a high score S3 from all documents as documents to be updated. The response support information update unit 114 may also select documents with a score S3 of a predetermined value or higher as documents to be updated. Alternatively, the response support information update unit 114 may select documents with a score S1 of a predetermined value or higher, or documents with a score S2 of a predetermined value or higher as documents to be updated.

[0070] In step S55, the response support information update unit 114 starts the process of repeating steps S56 and S57 for each document related to the update determined in step S54. Hereinafter, each document related to the update that is the subject of the repeated process will be referred to as the document to be processed.

[0071] In step S56, the response support information update unit 114 regenerates the response 144 for records where document 145 is the document to be processed and evaluation 146 is less than a predetermined value. The predetermined value may be a predetermined evaluation value (see steps S31 and S34 in Figure 9).

[0072] Figure 13 shows the answer regeneration prompt 430 according to this embodiment. The answer support information update unit 114 regenerates the answer using the answer regeneration prompt 430. Element 431 of the answer regeneration prompt 430 includes instructions to create an answer to the question in element 432 based on the document content in element 434 and according to the format in element 433. Element 432 includes a question 143 for a record where document 145 is the document to be processed and evaluation 146 is less than a predetermined value. Element 433 shows a format in which the question and answer are output on separate lines. Element 434 shows the document to be processed.

[0073] Returning to Figure 12, we continue the explanation of step S56. The answer support information update unit 114 sends an answer regeneration prompt 430 to the language model server 210 to obtain the question and answer. Subsequently, the answer support information update unit 114 updates the answer 144 of the record designated as question 143 with the obtained answer. The answer support information update unit 114 also sets the record generation method 142 to "A" and the evaluation 146 to "Not done".

[0074] In step S57, the answer support information update unit 114 adds an answer support information record. Figure 14 shows the question and answer addition prompt 440 according to this embodiment. The answer support information update unit 114 adds a question and an answer using the question and answer addition prompt 440. Element 441 of the question and answer addition prompt 440 includes instructions to create a question different from the question in element 442 and an answer to that question, based on the document content in element 444 and in accordance with the format in element 443. Element 442 includes a question 143 for records where document 145 is the document to be processed and evaluation 146 is greater than or equal to a predetermined value, and a question 143 for records where the answer 144 was regenerated in step S56. Element 443 shows a format in which the question and answer are output on separate lines. Element 434 shows the document to be processed.

[0075] Returning to Figure 12, we continue the explanation of step S57. The answer support information update unit 114 sends a question and answer addition prompt 440 to the language model server 210 to obtain the question and answer. Subsequently, the answer support information update unit 114 adds the obtained question and answer records. More specifically, the answer support information update unit 114 adds records to the answer support information database 140, storing the new identification information, obtained question, and answer in the record. The answer support information update unit 114 also stores the name / identification information of the document to be processed in document 145 and sets the evaluation 146 to "Not yet processed".

[0076] In step S58, the response support information update unit 114 clears the user evaluation cache 150 (see Figure 3).

[0077] As described above, the question answering device 100 includes a question answering information update unit 114 that, among the documents in the answer support information (see answer support information database 140), sends a prompt (question answer addition prompt 440) to the language model server 210 that includes instructions to generate questions and answers related to the document for documents in which the proportion of questions 143 related to the document (see score S1) is equal to or greater than a first predetermined value, the average value of evaluation values ​​(see evaluation 146) related to the document (see reciprocal of score S2) is equal to or less than a second predetermined value, or the quotient obtained by dividing the proportion of questions related to the document by the average value of evaluation values ​​related to the document (see score S3) is equal to or greater than a third predetermined value, to obtain new questions and answers, and updates the answer support information by adding the obtained questions and answers.

[0078] The response support information update unit 114 updates the response support information when the time-series change in the questioner's evaluation value for response 144 meets predetermined conditions. The predetermined conditions are based on the mean or variance of the divided evaluation value groups, obtained by dividing the time-series evaluation values ​​(see User Evaluation Cache 150) into predetermined lengths.

[0079] Features of the Question Answering Device The question answering device 100 provides an answer 144 from the question 143 in the answer support information database 140 that matches the question posed by the user (the similarity is equal to or greater than the first predetermined similarity). The question answering device 100 may also provide an answer 144 from a record with a higher evaluation 146 (equal to or greater than the predetermined evaluation value).

[0080] If there is no record that matches the question posed by the user, the question answering device 100 generates an answer to the user's question using a document 145 from the answer support information database 140 that has a high similarity to the question posed by the user (similarity of 2 predetermined similarity or higher), or a document that has a high similarity to the question posed by the user (see steps S36 and S37 in Figure 9). In this way, the question answering device 100 provides an answer based on an answer 144 or document 145 that corresponds to a question 143 that is similar to the question posed by the user with a similarity of 1 predetermined similarity or 2 predetermined similarity or higher. Therefore, a highly accurate answer can be expected.

[0081] The question answering device 100 also regenerates answers with a low evaluation score 146. Furthermore, for documents related to answers with a low evaluation score 146, or documents with many questions, the question answering device 100 generates additional questions and answers to update the answer support information database 140 (see steps S56 and S57 in Figure 12).

[0082] With such a question-answering device 100, users (questioners) can obtain highly accurate and well-rated answers to their questions. Furthermore, the question-answering device 100 can regenerate answers with low user ratings and add additional questions and answers to documents with many questions. This is expected to improve the accuracy of the answers.

[0083] <<Modification: Regenerate support information for low-rated answers>> In the embodiment described above, the response support information update unit 114 calculates a document score to determine the document to be updated (see step S54) and regenerates the response 144 for the low-rated response support record (see step S56). The response support information update unit 114 may also regenerate the low-rated response 144 regardless of the document. For example, the response support information update unit 114 may regenerate the response 144 for records where the evaluation 146 is less than a predetermined evaluation value, using a response regeneration prompt 430 (see Figure 13) that includes the document 145.

[0084] As described above, the question answering device 100 includes a question support information update unit 114 that, for questions 143 in the answer support information (see answer support information database 140) where the evaluation value (see evaluation 146) is less than a predetermined evaluation value, sends a prompt (answer regeneration prompt 430) including the question 143 and a document 145 related to the question 143 to the language model server 210 to obtain an answer to the question 143, and updates the answer 144 for the question 143 in the answer support information with the obtained answer.

[0085] <<Other variations>> Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention as described herein and elsewhere, as well as in the scope of the invention and its equivalents as described in the claims.

[0086] Hardware Configuration The question answering device 100 according to the above embodiment is implemented by a computer 900 having a configuration such as that shown in Figure 15. Figure 15 is a hardware configuration diagram showing an example of a computer 900 that implements the functions of the question answering device 100 according to the above embodiment. The computer 900 includes a CPU 901, ROM 902, RAM 903, SSD 904, and an input / output interface 905 (labeled as input / output I / F (Interface) in Figure 15). Furthermore, the computer 900 includes a communication interface 906 (labeled as communication I / F in Figure 15) and a media interface 907 (labeled as media I / F in Figure 15). The computer 900 may have an HDD (Hard Disk Drive) instead of the SSD 904, or it may have an HDD in addition to the SSD 904.

[0087] The CPU 901 operates based on programs stored in the ROM 902 or SSD 904 and is controlled by the control unit 110 in Figure 1. The ROM 902 stores boot programs executed by the CPU 901 when the computer 900 starts up, as well as programs related to the computer 900's hardware.

[0088] The CPU 901 controls input devices 910, such as a mouse and keyboard, and output devices 911, such as a display and printer, via the input / output interface 905. The CPU 901 acquires data from the input devices 910 and outputs the generated data to the output devices 911 via the input / output interface 905.

[0089] SSD904 stores programs executed by CPU901 and data used by those programs. Communication interface906 receives data from other devices (e.g., language model server210) not shown via the communication network and outputs it to CPU901, and also transmits data generated by CPU901 to other devices via the communication network.

[0090] The media interface 907 reads a program or data stored in the recording medium 912 and outputs it to the CPU 901 via the RAM 903. The CPU 901 loads the program from the recording medium 912 onto the RAM 903 via the media interface 907 and executes the loaded program. The recording medium 912 can be an optical recording medium such as a DVD (Digital Versatile Disk), a magneto-optical recording medium such as an MO (Magneto Optical Disk), a magnetic recording medium, a conductive memory tape medium, or a semiconductor memory.

[0091] For example, when computer 900 functions as a question answering device 100 according to the above embodiment, the CPU 901 of computer 900 realizes the functions of the question answering device 100 by executing a program 128 (see Figure 1) loaded on RAM 903. The CPU 901 reads the program from the recording medium 912 and executes it. In addition, the CPU 901 may read the program from another device via a communication network, or it may install the program 128 from the recording medium 912 to the SSD 904 and execute it. [Explanation of Symbols]

[0092] 100 question answering device 111 Administrator Operation Reception Department 112 Answer support information generation section 113 Question and answer section 114 Answer support information update department 130 Document Databases 140 Response Support Information Database (Response Support Information) 150 User Ratings Cache 121 Configuration Information 210 Language Model Servers 410 Question Answer Generation Prompt 420 Answer generation prompt 430 Answer regeneration prompt 440 Additional question and answer prompt 510 Administrator screen 520 User screen

Claims

1. By referring to answer support information that stores questions and their corresponding answers, The system includes a question answering unit that outputs an answer corresponding to a question in the answer support information whose similarity to the question posed by the user is equal to or greater than a first predetermined similarity, as the answer to the question posed by the user. Question answering device.

2. If there is no question in the answer support information whose similarity to the question posed by the user is equal to or greater than the first predetermined similarity, the question answering unit will: The prompt containing the question posed by the user is sent to the language model server, and the answer to the question is obtained and output. The question posed by the user and the outputted answer are associated and added to the answer support information. The question answering device according to claim 1.

3. The aforementioned response support information includes: The user's evaluation of the aforementioned response is further associated and stored. The aforementioned question answer section is, The user's evaluation value for the outputted answer is obtained and stored in the answer support information. If a question in the answer support information has a similarity to the question posed by the user that is equal to or greater than the first predetermined similarity, and the evaluation value of the answer corresponding to that question is equal to or greater than the predetermined evaluation value, then the answer is output as the answer to the question posed by the user. If the evaluation value is less than the predetermined evaluation value, a prompt including the question posed by the user is sent to the language model server, and the answer to the question is obtained and output. The question answering device according to claim 2.

4. The aforementioned response support information includes: Documents related to the aforementioned question are further associated and stored, If there is no question in the answer support information whose similarity to the question posed by the user is equal to or greater than the first predetermined similarity, the question answering unit will: If a question in the answer support information has a similarity to the question posed by the user that is equal to or greater than the second predetermined similarity, and the evaluation value of the answer corresponding to that question is equal to or greater than the predetermined evaluation value, The document related to the question and the prompt containing the question issued by the user are sent to the language model server, and the answer to the question is obtained and output. The second predetermined similarity is, Lower than the first predetermined similarity The question answering device according to claim 3.

5. If there is no question in the answer support information whose similarity to the question posed by the user is equal to or greater than the first predetermined similarity, the question answering unit will: If a question in the answer support information has a similarity to the question posed by the user that is equal to or greater than the second predetermined similarity, and the evaluation value of the answer corresponding to that question is less than the predetermined evaluation value, The document with the highest similarity to the question posed by the user, along with a prompt containing the question posed by the user, is sent to the language model server to obtain and output the answer to the question. The question answering device according to claim 4.

6. The aforementioned response support information includes: The user's evaluation of the answer and the document related to the question are further stored in association with each other. The aforementioned question answer section is, The user's evaluation value for the outputted answer is obtained and stored in the answer support information. Among the response support information, for questions in which the evaluation value is less than a predetermined evaluation value, a prompt including the question and a document related to the question is sent to the language model server to obtain a response to the question. The system includes an answer support information update unit that updates the answer to the question in the aforementioned answer support information with the obtained answer. The question answering device according to claim 1.

7. The aforementioned response support information includes: The user's evaluation of the answer and the document related to the question are further stored in association with each other. The aforementioned question answer section is, The user's evaluation value for the outputted answer is obtained and stored in the answer support information. In the documents included in the aforementioned support information, Documents in which the proportion of questions related to the document is equal to or greater than the first predetermined value, Documents in which the average value of the evaluation values ​​related to the document is less than or equal to the second specified value, For documents where the quotient obtained by dividing the proportion of questions related to the document by the average value of evaluation values ​​related to the document is equal to or greater than the value specified in Section 3, A prompt containing instructions to generate questions and answers related to the document is sent to the language model server to obtain new questions and answers. The system includes an answer support information update unit that updates the answer support information by adding the acquired questions and answers to it. The question answering device according to claim 1.

8. The aforementioned response support information update unit is: The response support information is updated when the time-series change in the questioner's evaluation value for the aforementioned response meets a predetermined condition. The question answering device according to claim 6 or 7.

9. The aforementioned predetermined conditions are: The evaluation values ​​in the time series are divided into predetermined lengths, and the condition is based on the mean or variance of the divided evaluation value groups. The question answering device according to claim 8.

10. The question answering device, By referring to answer support information that stores questions and their corresponding answers, The following step is performed: output an answer corresponding to a question in the answer support information whose similarity to the question posed by the user is equal to or greater than a first predetermined similarity, as the answer to the question posed by the user. How to answer questions.

Images