Response system and response method

Abstract

To obtain accurate answers to questions. [Solution] An answer system 10 comprising a computing device that performs a question reception process that receives text input of a question 41 related to business from a user; a data extraction process that extracts a first text and a second text related to question 41 from a database of texts of business-related sentences or words; an answer generation process that outputs an answer 42 to question 41 by inputting a prompt including the content indicated by the extracted first text and second text and the content indicated by the text of question 41 to a natural language processing model 17; an answer determination process that determines whether the outputted answer 42 is valid or not; and a user confirmation process that, if it is determined that the outputted answer 42 is not valid, generates text containing information related to question 41 and stores the generated text as a candidate for text related to question 41.

Description

Technical Field

[0001] The present invention relates to an answering system and an answering method.

Background Art

[0002] Operations using AI (Artificial Intelligence) such as large language models (LLMs: Large Language Models) are widely carried out. For example, companies provide AI services that receive access from customers and answer questions from customers.

[0003] As a technology for improving the accuracy of answers by such AI, for example, Patent Document 1 discloses a method of converting a question into a numerical value and searching for a highly relevant answer from a database. Further, Patent Document 2 discloses a technology for evaluating the performance of a question answering system.

[0004] And, Non-Patent Documents 1 and 2 disclose technologies for applying so-called RAG (Retrieval-Augmented Generation) to an LLM and inputting data obtained by expanding a question from a user into the AI.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] However, with the current LLM system, it may not be possible to obtain answers with sufficient accuracy depending on the type of work.

[0008] This invention has been made in view of these circumstances, and its purpose is to provide an answering system and an answering method that can obtain accurate answers to questions. [Means for solving the problem]

[0009] One of the present inventions for solving the above problems is an answer system comprising a storage device that stores a database of texts or words related to business operations and a natural language processing model that receives natural language text prompts and outputs text corresponding to the input prompts, and a computing device that performs a question reception process for receiving text input of a question related to the business operations from a user, a data extraction process for extracting a first text and a second text related to the input question from the database, an answer generation process for outputting an answer to the input question by inputting a prompt to the natural language processing model that includes the content indicated by the extracted first text and the second text and the content indicated by the input question text, an answer determination process for determining whether the output answer is valid using a predetermined algorithm, and a user confirmation process that, if it is determined that the output answer is not valid, generates text containing information related to the question using a predetermined algorithm and stores the generated text as a candidate text related to the input question. [Effects of the Invention]

[0010] According to the present invention, accurate answers to questions can be obtained. Other configurations and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the configuration of the business support system according to Example 1. [Figure 2] This diagram illustrates an example of the functions that the answer system possesses. [Figure 3] This figure shows an example of the hardware that the answering system will have. [Figure 4] This diagram illustrates the processing performed by the business support system in Example 1. [Figure 5] This diagram illustrates the details of the process for extracting information related to simulated answers. [Figure 6]This is a diagram for explaining the details of the suspicious question-related information extraction process. [Figure 7] This is a diagram for explaining the details of the answer generation process and the answer determination process. [Figure 8] This is a diagram for explaining the details of the user confirmation process. [Figure 9] This is a diagram for explaining the details of the answer generation process and the answer determination process in Example 2. [Figure 10] This is a diagram for explaining the details of the answer generation process and the answer determination process in Example 3. [Figure 11] This is a diagram for explaining the processes performed by the business support system in Example 4.

Modes for Carrying Out the Invention

[0012] Embodiments of the present invention will be described with reference to the drawings.

[0013] [Example 1] FIG. 1 is a diagram showing an example of the configuration of a business support system 1 according to Example 1. The business support system 1 includes an answer system 10, which is an information processing system managed by an enterprise, and one or more user terminals 20 used by users who are customers of the enterprise.

[0014] The answer system 10 stores a natural language processing model described later, receives a question (query) regarding the business being conducted by the enterprise from the user terminal 20, generates an answer to the question, and returns it to the user terminal 20. At this time, the answer system 10 uses a chunk text database described later and returns an appropriate answer to the user based on the so-called Retrieval Augmented Generative (RAG) technology.

[0015] The answer system 10 and the user terminal 20 are communicably connected by a wired or wireless communication network 5 such as the Internet, a Local Area Network (LAN), a Wide Area Network (WAN), or a dedicated line.

[0016] Figure 2 illustrates an example of the functions provided by the response system 10.

[0017] The response system 10 stores a chunk text database 16 and a natural language processing model 17.

[0018] The chunked text database 16 is a database of text (hereinafter referred to as "chunked text") consisting of documents or words related to the business of a company. In this embodiment, the company's business is assumed to be insurance business, but this is not intended to limit the type of business. Chunked text is, for example, text extracted from business documents, regulations, and legal texts.

[0019] The natural language processing model 17 is a mathematical model (e.g., a Large Language Model (LLM)) that takes natural language text prompts as input and outputs text corresponding to the input prompts. The natural language processing model 17 can be constructed using, for example, BERT (Bidirectional Encoder Representations from Transformers), XLNet, or GPT (Generative Pre-trained Transformer).

[0020] Next, the response system 10 includes the following functional units: a question reception unit 11, a data extraction unit 12, a response generation unit 13, a response determination unit 14, and a user confirmation unit 15.

[0021] The question reception unit 11 receives text input of questions from users (user terminals 20) regarding their work.

[0022] The data extraction unit 12 extracts a first text (hereinafter also referred to as the first chunk text) and a second text (hereinafter also referred to as the second chunk text) related to the question entered in the question reception unit 11 from the chunk text database 16.

[0023] Specifically, the answer generation unit 13 includes a pseudo-answer related information extraction unit 131 and a pseudo-question related information extraction unit 132.

[0024] The pseudo-answer related information extraction unit 131 obtains answers (pseudonyms) to questions entered by the question reception unit 11 using a predetermined algorithm, and extracts text from the chunk text database 16 whose similarity to the obtained pseudo-answer is greater than or equal to a predetermined value as the first text (first chunk text).

[0025] In this embodiment, the pseudo-answer related information extraction unit 131 obtains a pseudo-answer to the question entered by the question reception unit 11 by inputting a prompt to the natural language processing model 17 requesting an answer to the question entered by the question reception unit 11.

[0026] The pseudo-question related information extraction unit 132 generates text for each question (pseudo-question) whose answer is text in the chunk text database 16 using a predetermined algorithm, and extracts the pseudo-question text from the generated pseudo-question texts that have a similarity to the question text input by the question reception unit 11 that is equal to or greater than a predetermined value as the second text (second chunk text).

[0027] In this embodiment, the pseudo-question related information extraction unit 132 generates the text of a pseudo-question by inputting a prompt to the natural language processing model 17 that requests the generation of a question whose answer is text from the chunk text database 16.

[0028] The response generation unit 13 outputs a response to the question entered in the question reception unit 11 by inputting a prompt to the natural language processing model 17 that includes the content indicated by the first text and the second text extracted by the data extraction unit 12 and the content indicated by the text of the question entered in the question reception unit 11.

[0029] The response determination unit 14 determines whether the response output by the response generation unit 13 is valid or not using a predetermined algorithm.

[0030] In this embodiment, the response determination unit 14 generates a prompt asking for the validity grade value of the response output by the response generation unit 13, and inputs the generated prompt to the natural language processing model 17 to determine whether the response output by the response generation unit 13 is valid or not.

[0031] If the user verification unit 15 determines that the answer output by the answer generation unit 13 is not valid, it generates text containing information related to the question entered by the question reception unit 11 using a predetermined algorithm, and stores the generated text as a candidate for text (chunk text) related to the question entered by the question reception unit 11.

[0032] Specifically, the user verification unit 15 obtains text indicating the details of the question by inputting a prompt to the natural language processing model 17 requesting the generation of a question sentence that asks for details of the question from the question reception unit 11. The user verification unit 15 then receives input from the user for the text of the answer corresponding to the obtained text, and generates text containing information related to the question from the text of the answer entered by the user and the text indicating the details of the question obtained above (hereinafter referred to as conversation history text).

[0033] Figure 3 shows an example of the hardware provided by the response system 10. The response system 10 is an information processing device comprising a computing unit 31 such as a CPU (Central Processing Unit), a main memory 32 such as RAM (Random Access Memory) or ROM (Read Only Memory), an external storage device 33 such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), an input device 34 such as a keyboard, mouse, or touch panel, a display device 35 such as a display or touch panel, and a communication device 36 consisting of a NIC (Network Interface Card), wireless communication module, USB (Universal Serial Interface) module, or serial communication module. The user terminal 20 is equipped with similar hardware.

[0034] The functions of the functional unit of the answer system 10, as described above, are realized by the arithmetic unit 31 reading programs from the main memory 32 or external memory 33. Each program can be recorded on, for example, a portable or fixed recording medium and distributed. In addition, each program in the answer system 10 may be implemented in whole or in part using virtual information processing resources provided using virtualization technology, process space isolation technology, etc., such as a virtual server provided by a cloud system. Furthermore, all or part of these programs may be implemented by services provided by the cloud system via, for example, an API (Application Programming Interface). Next, we will explain the processes performed in business support system 1.

[0035] Figure 4 is a diagram illustrating the process performed by the business support system 1 in Example 1.

[0036] First, the question receiving unit 11 of the answering system 10 obtains the text of a question 41 (hereinafter simply referred to as "the question") related to the company's business from the user terminal 20 (s10). For example, the question receiving unit 11 accepts the input of a question about insurance business from the user terminal 20. The question is, for example, "I subscribed to a comprehensive insurance plan. Last month, the insurance premium was not paid due to insufficient funds. What should I do?"

[0037] The pseudo-answer related information extraction unit 131 executes a pseudo-answer related information extraction process s30 to extract a first chunk text based on the question 41 input in s10. The pseudo-question related information extraction unit 132 executes a pseudo-question related information extraction process s40 to extract a second chunk text based on the question input in s10. The answer generation unit 13 then outputs an answer 42 to the question 41 input in s10 based on the first and second chunk texts.

[0038] The first chunk text consists of a predetermined number (M in this case) of chunk texts extracted from the chunk text database 16 in descending order of similarity to the pseudo-answers described later. The second chunk text consists of a predetermined number (N in this case) of chunk texts extracted from the chunk text database 16 in descending order of similarity to the pseudo-questions described later.

[0039] Then, the answer generation unit 13 executes an answer generation process s50 that outputs an answer 42 to the question 41 input in s10, based on the extracted first chunk text and second chunk text.

[0040] The response determination unit 14 executes a response determination process s60 to determine whether the response 42 output in the response generation process s50 is valid or not. If the response 42 is determined to be valid (s60: accepted), the response determination unit 14 sends the output response 42 to the user terminal 20, and the user terminal 20 displays the received response 42 on the display device screen (s70).

[0041] On the other hand, if the response 42 is determined to be invalid (s60: rejection), the user verification unit 15 executes a user verification process s80 that generates conversation history text based on the question 41 entered in s10. The user verification unit 15 adds the conversation history text to the chunk text database 16. After that, the process in s10 is repeated.

[0042] The details of each process are explained below. <Processing to extract information related to simulated answers> Figure 5 is a diagram illustrating the details of the pseudo-response related information extraction process s30.

[0043] The pseudo-answer related information extraction unit 131 inputs the question 41 entered in s10 into the natural language processing model 17, thereby obtaining the text of the answer (pseudonym 51) corresponding to the question 41 from the natural language processing model 17 (s31).

[0044] For example, mock answer 51 states, "If premium payments are in arrears, the total amount of premiums paid in the past will be deducted from the surrender value, and the insurance contract will be continued. If the surrender value is less than the unpaid premiums, the insurance contract will be terminated." Mock answer 51, as shown above, is not necessarily appropriate to the question.

[0045] Then, the pseudo-response related information extraction unit 131 converts the text of the pseudo-response 51 obtained in s11 into vector data (embedding process) (s32).

[0046] Meanwhile, the pseudo-response related information extraction unit 131 retrieves each chunk text 52 from the chunk text database 16 and converts each retrieved chunk text 52 into vector data (embedding process) (s33). The chunk text is data such as, for example, "In the case of an insurance policyholder under Article 1...<Article 2, Paragraph 2>If the transfer of insurance premiums cannot be completed by the payment due date using the account transfer method, we will re-bille the insurance premiums to the designated account the following month."

[0047] The pseudo-answer related information extraction unit 131 calculates the similarity between the vector data of the pseudo-answer 51 converted in s32 and the vector data of each chunk text 52 converted in s33 (s34). For example, the pseudo-answer related information extraction unit 131 calculates the cosine similarity between the vector data of the pseudo-answer 51 and the vector data of the chunk text 52.

[0048] Then, the pseudo-response related information extraction unit 131 extracts M chunk texts (first chunk text 53) from each chunk text 52 whose similarity to the pseudo-response vector data was calculated in s34, in descending order of similarity.

[0049] <Simulated Question Related Information Extraction Process s40> Figure 6 is a diagram illustrating the details of the pseudo-question-related information extraction process s40.

[0050] The pseudo-question related information extraction unit 132 retrieves each chunk text 61 from the chunk text database 16. Then, for each retrieved chunk text 61, the pseudo-question related information extraction unit 132 inputs a prompt to the natural language processing model 17 that generates a question sentence in which the content of the chunk text 61 is used as the answer, thereby retrieving the text of the question (pseudo-question 62) corresponding to each chunk text 61 (s41). The pseudo-question related information extraction unit 132 converts the text of each retrieved pseudo-question 62 into vector data (embedding process) (s42).

[0051] The prompt is, for example, "Please create one or more questions that would answer with something like, 'In the case of an insurance policyholder under Article 1... <Article 2, Paragraph 2> If the transfer of insurance premiums cannot be completed by the payment due date via bank transfer, we will re-bille the insurance premiums to the designated account the following month.'" Then, the mock question 62 is, for example, "Q1: When is the payment deadline for the insurance premiums?", "Q2: Will late payment penalties be incurred if payment is delayed?", "Q3: Can I set up regular payments?"

[0052] Meanwhile, the pseudo-question related information extraction unit 132 converts the text data of the user question 41 entered in s10 into vector data (embedding process) (s43).

[0053] Then, the pseudo-question related information extraction unit 132 calculates the similarity between the vector data of question 41 converted in s43 and the vector data of each pseudo-question 62 converted in s42 (s44). For example, the pseudo-question related information extraction unit 132 calculates the cosine similarity between the vector data of question 41 and the vector data of pseudo-question 62.

[0054] Then, the pseudo-question related information extraction unit 132 extracts N vector data of pseudo-questions 62 (second chunk text 63) in descending order of similarity from the vector data of each pseudo-question 62 whose similarity to the vector data of question 41 was calculated in s44.

[0055] <Answer generation process, answer judgment process> Figure 7 is a diagram illustrating the details of the answer generation process s50 and the answer determination process s60.

[0056] The answer generation unit 13 generates a prompt requesting an answer 42 to question 41. The answer determination unit 14 then inputs the generated prompt into the natural language processing model 17 to obtain the text of the answer 42 to question 41 (s51).

[0057] This prompt indicates that, for example, the text of question 41 should have the first chunk text extracted in the pseudo-answer related information extraction process s30 and the second chunk text extracted in the pseudo-question related information extraction process s40 added as supplementary information, and that an answer to the text of question 41 is requested. This can be achieved, for example, by RAG (Retrieval-Augmented Generation).

[0058] Then, the response determination unit 14 determines whether the response 42 obtained in s51 is valid or not using a predetermined model (s61). In this embodiment, the response determination unit 14 determines the validity of the response 42 by inputting a prompt to the natural language processing model 17 asking about the validity of the text of the response 42 obtained in s51.

[0059] Specifically, the response determination unit 14 inputs a prompt to the natural language processing model 17 asking which of the following four grades the response 42 falls into. The response determination unit 14 then stores that the response 42 is valid if the output value from the natural language processing model 17 is grade 1 or grade 2, and stores that the response 42 is invalid if the output value from the natural language processing model 17 is grade 3 or grade 4.

[0060] Grade 1: The answer contains all the information that should be present in an ideal answer. Grade 2: The answer contains only some of the information that should be present in an ideal answer, but is acceptable as an answer to the question. Grade 3: The answer contains only a fraction of the information found in an ideal answer and is insufficient as a response to the question. Grade 4: The answer does not address the question.

[0061] Note that the grades used to indicate the validity of Answer 42 shown here are just examples. For example, you could use only two grades: "The answer addresses the question" or "The answer does not address the question."

[0062] <User verification process> Figure 8 is a diagram illustrating the details of the user confirmation process s80.

[0063] If the user verification unit 15 stores that the answer is invalid in the answer determination process s60, it performs the following processing:

[0064] First, the user verification unit 15 generates a prompt requesting the generation of a detailed question 82 that asks for a more detailed explanation of the question 41 entered in s10. Then, the user verification unit 15 inputs the generated prompt into the natural language processing model 17 to obtain the detailed question 82.

[0065] For example, if question 41 is, "I have joined Stage A. Can I change my premium payment method (monthly, annual, or semi-annual)?" and the answer 42 is, "The document does not contain specific details about changing the premium payment method (monthly, annual, etc.)," ​​then the generated follow-up question 82 would be, "What is your current premium payment method?"

[0066] The user verification unit 15 displays the content of the question generated in s81 on the screen and accepts text input of the answer to the question from the user terminal 20 (s82). The user verification unit 15 stores the text of the input answer 83.

[0067] The user verification unit 15 stores the conversation text (conversation history text 84), which consists of the text of the detailed question 82 generated in s82 and the text of the answer 83 entered in s82, as a new chunk text. The user verification unit 15 may add this new chunk text to the chunk text database 16, or it may perform the data extraction process s20 again based on the list of chunk texts that include this new chunk text without adding it to the chunk text database 16. In this way, the user verification unit 15 considers this new chunk text as a candidate for chunk text containing information related to question 41. Note that the conversation history text 84 may be broken down into sentences or words.

[0068] The user verification unit 15 may repeat the processes s81 and s82 multiple times.

[0069] As described above, the response system 10 of this embodiment extracts a first text and a second text related to the business question 41 entered by the user from the chunk text database 16, and inputs a prompt to the natural language processing model 17 that includes the content indicated by the extracted first text and the second text and the content indicated by the text of the question 41, thereby outputting a response to the question 41. If it is determined that the output response 42 is not valid, it generates a text containing information related to the question 41 (a conversation history text 84 including the response 83 to the detailed question 82), and adds the generated text to the chunk text database 16.

[0070] In other words, the answer system 10 of this embodiment obtains an answer by inputting a question prompt containing two chunk texts from the chunk text database 16 related to a question 41 from the user into the natural language processing model 17. If the answer 42 is not valid, additional information regarding the question is registered in the chunk text database 16. This makes it possible to obtain an accurate answer 42 to question 41.

[0071] Specifically, the answer system 10 in this embodiment obtains a pseudo-answer 51 to a question 41 from the user, and extracts text (first chunk text 53) from the chunk text in the chunk text database 16 that has a high degree of similarity to the pseudo-answer 51. The answer system 10 also generates a pseudo-question 62 for each chunk text, using the content of the chunk text in the chunk text database 16 as the answer, and extracts the text of the pseudo-question 62 that is similar to question 41 (second chunk text 63) from among the generated pseudo-question 62 texts.

[0072] In this way, by using the chunk text corresponding to the pseudo-answer obtained by extending question 41, and the chunk text obtained from the pseudo-question, as input data to the natural language processing model, it is possible to obtain accurate answers to the questions.

[0073] Specifically, the answer system 10 of this embodiment obtains a pseudo-answer 51 to question 41 by inputting a prompt requesting an answer 42 to question 41 to the natural language processing model 17.

[0074] This allows us to obtain a simulated answer 51 with appropriate content.

[0075] Specifically, the answer system 10 of this embodiment generates the text of a pseudo-question by inputting a prompt to the natural language processing model 17 that requests the generation of a pseudo-question 62 whose answer is the text in the chunk text database 16.

[0076] This allows us to obtain a suitable mock question 62.

[0077] Furthermore, the response system 10 of this embodiment generates a prompt asking for the validity grade value of the output response 42, and inputs the generated prompt into the natural language processing model 17 to determine whether or not the output response 42 is valid.

[0078] This allows for an objective determination of the validity of the output response 42.

[0079] Specifically, the response system 10 of this embodiment inputs a prompt to the natural language processing model 17 asking which of the four validity grade values ​​the output response 42 corresponds to, and determines that the output response 42 is valid if it determines that the output response 42 corresponds to grade 1 or 2 (grade 1 or 2).

[0080] This allows for accurate determination of the validity of the output response 42.

[0081] Furthermore, if the response system 10 of this embodiment determines that the output response 42 is not valid, it inputs a prompt to the natural language processing model 17 requesting the generation of a question asking for details of question 41 (detailed question 82), thereby obtaining the detailed question 82. The response system 10 then receives the text input of the response corresponding to the detailed question 82 from the user, generates the input response 83 text and the detailed question 82 text as conversation history text 84, and stores them as candidate texts related to question 41.

[0082] In this way, by asking the user for details about question 41, generating the answer as conversation history text 84 along with the corresponding question, and using it as a candidate for chunk text related to question 41, the accuracy of the answer to the question can be improved.

[0083] Furthermore, if the response system 10 of this embodiment determines that the output response 42 is valid, it displays the output response 42 on the display device of the user terminal 20.

[0084] This allows the user to see answer 42 to the question.

[0085] [Example 2] Next, we will describe the business support system 1 of Example 2.

[0086] In this embodiment of the business support system 1, the answer determination unit 14 of the answer system 10 differs from that in Embodiment 1. The other configurations are the same as in Embodiment 1.

[0087] In other words, the response determination unit 14 determines, using a predetermined algorithm, whether the response 42 output by the response generation unit 13 contains the content indicated by the first chunk text or the second chunk text. If it determines that the output response 42 contains the content indicated by the first chunk text or the second chunk text, it determines that the response 42 output by the response generation unit 13 is valid.

[0088] <Answer generation process, answer judgment process> Figure 9 is a diagram illustrating the details of the response generation process s50 and the response determination process s60 in Example 2.

[0089] The answer generation unit 13 generates a prompt 101 that requests an answer 42 to question 41 and also requests information about the chunk of text that the answer 42 quotes (transcribes). The answer determination unit 14 then inputs the generated prompt into the natural language processing model 17 to obtain the text of the answer 102 to question 41 and the text of the quoted portion 103 of the answer 42 (s52).

[0090] This prompt 101 is data that includes, for example, the answer to question 41, a request for the chunk text portion cited in that answer, the first chunk text extracted in the pseudo-answer related information extraction process s30, and the second chunk text extracted in the pseudo-question related information extraction process s40 (hereinafter, both texts are referred to as comparison chunk texts).

[0091] The answer determination unit 14 compares the comparison chunk text with the text of the quoted portion 103 obtained in s52 (s62). The answer determination unit 14 then determines whether or not the text of the quoted portion 103 obtained in s41 is included in the comparison chunk text.

[0092] If the text of quoted portion 103 is included in the comparison chunk text, the answer determination unit 14 stores that the answer 102 obtained in s52 is valid. If the text of quoted portion 103 is not included in the comparison chunk text, the answer determination unit 14 stores that the answer 102 obtained in s52 is invalid.

[0093] Note that the determination method in s52 described here is just one example. For example, the answer determination unit 14 may determine whether the text of the quoted portion 103 is included in the comparison chunk text in a certain proportion or higher (for example, on a word-based or sentence-based basis).

[0094] As described above, the response system 10 of this embodiment determines, using a predetermined algorithm, whether the response 42 output in the response generation process s50 contains the content indicated by the first text (chunk text) or the second text (chunk text). If it is determined that the response output in the response generation process s50 contains the content indicated by the first text or the second text, it determines that the response output in the response generation process s50 is valid.

[0095] In this way, by determining whether the output response 42 quotes chunk text, it is possible to prevent situations (hallucination) where response 42 contains unsubstantiated information from the chunk text database 16.

[0096] Specifically, the answer system 10 of this embodiment outputs the answer to question 41 and the quoted portion of the answer 42 by inputting a prompt to the natural language processing model 17 that includes the content indicated by the first text (chunk text) and the second text (chunk text), the content indicated by the text of question 41, and a request for the portion of the chunk text database 16 that is cited in the answer to question 41.

[0097] In this way, by using the natural language processing model 17 to output the conversation history text 84 of response 42, it is possible to accurately determine whether chunk text has been quoted.

[0098] [Example 3] Next, we will describe the business support system 1 of Example 3.

[0099] In this embodiment of the business support system 1, the answer determination unit 14 of the answer system 10 differs from that in Embodiment 1. The other configurations are the same as in Embodiment 1.

[0100] In other words, the response determination unit 14 calculates the similarity between the content of the response output by the response generation unit 13 and the content indicated by the first text and the second text, determines whether the calculated similarity is equal to or greater than a predetermined threshold, and if it determines that the calculated similarity is equal to or greater than the predetermined threshold, it determines that the response output by the response generation unit 13 is valid.

[0101] <Answer generation process, answer judgment process> Figure 10 is a diagram illustrating the details of the response generation process s50 and the response determination process s60 in Example 3.

[0102] The response generation unit 13 obtains the text of the response 42 to question 41 based on the natural language processing model 17, similar to the first embodiment (s51).

[0103] Furthermore, the answer generation unit 13 converts the text consisting of M first chunk texts extracted in the pseudo-answer related information extraction process s30 and N second chunk texts extracted in the pseudo-question related information extraction process s40 into vector data (embedding process) (s53). Also, the answer generation unit 13 converts the text of the answer 42 into vector data (embedding process).

[0104] The answer determination unit 14 calculates the similarity between the answer vector 42 and the chunk text vector (s54). For example, the answer determination unit 14 calculates the cosine similarity between the answer vector data 42 and the chunk text vector data.

[0105] The response determination unit 14 stores that the response 42 obtained in s51 is valid if the similarity calculated in s54 is equal to or greater than a predetermined threshold, and stores that the response obtained in s51 is invalid if the similarity calculated in s51 is less than a predetermined threshold.

[0106] In this embodiment, the response system 10 determines that the response output by the response generation unit 13 is valid if it determines that the similarity between the content of the response output by the response generation unit 13 and the content indicated by each chunk text (first text, second text) is equal to or greater than a predetermined threshold.

[0107] This allows for accurate determination of whether the outputted response is correct or not, based on the chunked text.

[0108] [Example 4] Next, we will describe the business support system 1 of Example 4.

[0109] In the business support system 1 of this embodiment, the response system 10 has a text modification unit in addition to the functions of Embodiment 1. The other configurations are the same as in Embodiment 1.

[0110] If the text modification unit determines that the answer 42 output by the answer generation unit 13 is not valid, it extracts a third text and a fourth text from the chunk text database 16 that are related to the input question 41 and are different from the first text and the second text.

[0111] Figure 11 is a diagram illustrating the process performed by the business support system 1 in Example 4.

[0112] The processing in s10~s60 and s70 is the same as in Example 1.

[0113] In the response determination process s60, if the response 42 output in the response generation process s50 is determined to be invalid (s60: rejection), the text modification unit changes parts of the first chunk text and the second chunk text extracted in the pseudo-response related information extraction process s30 and the pseudo-question related information extraction process s40, respectively, to other chunk texts in the chunk text database 16 (referred to as the third text and the fourth text) (s90).

[0114] For example, the text modification unit replaces the lower half of the chunk text in the first chunk text with chunk text from the low-similarity chunk text database 16 that was not extracted. Also, for example, the text modification unit replaces the lower half of the chunk text in the second chunk text with chunk text from the low-similarity chunk text database 16 that was not extracted.

[0115] The method of modifying chunk text described here is merely an example and is not particularly limited. For example, the text modification section may replace all chunk text in the first (second) chunk text with chunk text from the chunk text database 16 that was not extracted and has low similarity.

[0116] The text modification unit then repeats the pseudo-answer related information extraction process s30 and the pseudo-question related information extraction process s40 and subsequent processes based on the modified chunk text (referred to as the third text and the fourth text). Note that, as in Example 1, a user confirmation process may also be performed.

[0117] As described above, if the response system 10 of this embodiment determines that the response 42 output by the response generation unit 13 is not valid, it extracts texts (third text and fourth text) from the chunk text database 16 that are different from the first text and second text and are related to the question 41, and inputs a prompt to the natural language processing model 17 that includes the content indicated by the extracted third text and fourth text and the content indicated by the text of question 41, thereby outputting a response 43 to question 41.

[0118] Thus, if an inappropriate answer is determined to have been obtained, a more accurate answer can be obtained by using different chunk text.

[0119] The present invention is not limited to the embodiments described above, and can be implemented using any components without departing from its essence. The embodiments and modifications described above are merely examples, and the present invention is not limited to these as long as the features of the invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0120] For example, some of the hardware provided in each device of this embodiment may be provided in other devices.

[0121] Furthermore, each program of each device may be provided in other devices, a program may consist of multiple programs, or multiple programs may be integrated into a single program.

[0122] Furthermore, the algorithms used to determine the validity of answer 42, as described in each embodiment, are examples only, and other methods (for example, comparison with a predetermined template database) may be employed.

[0123] Furthermore, the algorithm for generating detailed questions 82 in the user confirmation process s80 described in Example 1 is an example. Other methods (for example, generating detailed questions using a predetermined template database) may also be employed.

[0124] Furthermore, the algorithm for obtaining simulated answers and simulated questions described in Example 1 is merely an example. Other methods (for example, generating simulated answers and simulated questions using a predetermined template database) may also be employed. [Explanation of Symbols]

[0125] 10 Answer System, 11 Question Reception Unit, 12 Data Extraction Unit, 13 Answer Generation Unit, 14 Answer Judgment Unit, 15 User Verification Unit

Claims

1. A storage device that stores a database of texts or words related to business operations, and a natural language processing model that takes natural language text prompts as input and outputs text corresponding to the input prompts, and A question reception process that accepts text input from users regarding the aforementioned business operations, A data extraction process that extracts a first text and a second text related to the input question from the aforementioned database, The answer generation process involves inputting a prompt to the natural language processing model that includes the content indicated by the extracted first text and second text, and the content indicated by the text of the input question, thereby outputting an answer to the input question. A response determination process that determines whether the outputted response is valid or not using a predetermined algorithm, If the outputted answer is determined to be invalid, the arithmetic unit performs a user confirmation process in which it generates text containing information related to the question using a predetermined algorithm and stores the generated text as a candidate text related to the input question. A response system equipped with the following features.

2. The aforementioned computing device is In the aforementioned data extraction process, The process involves extracting pseudo-answers, which are answers to the input questions, using a predetermined algorithm, and extracting texts from the database whose similarity to the acquired pseudo-answers is greater than or equal to a predetermined value, as first texts. The process involves generating pseudo-question texts for each text in the database using a predetermined algorithm, and then extracting pseudo-question texts from the generated pseudo-question texts that have a similarity to the input question text of a predetermined value or higher as a second text. The answer system according to claim 1.

3. The aforementioned computing device is In the pseudo-answer related information extraction process, a pseudo-answer to the input question is obtained by inputting a prompt requesting an answer to the input question into the natural language processing model. The answer system according to claim 2.

4. The aforementioned computing device is In the pseudo-question related information extraction process, the text of the pseudo-question is generated by inputting a prompt to the natural language processing model that requests the generation of a question whose answer is text from the database. The answer system according to claim 2.

5. The aforementioned computing device is In the response determination process, a prompt is generated to ask for the validity grade value of the output response, and the generated prompt is input to the natural language processing model to determine whether the output response is valid or not. The answer system according to claim 1.

6. The aforementioned computing device is In the response determination process, a prompt is generated asking which of the four validity grades the output response corresponds to. By inputting the generated prompt into the natural language processing model, it is determined whether the output response corresponds to grade 1 or grade 2. If it is determined that the output response corresponds to grade 1 or grade 2, the output response is determined to be valid. The answer system according to claim 5.

7. The aforementioned computing device is In the user verification process described above, if it is determined that the outputted answer is not valid, a prompt is input to the natural language processing model requesting the generation of a question asking for details of the question, thereby obtaining text indicating the details of the question, receiving input from the user for the answer text corresponding to the obtained text, and generating text containing information related to the question from the user's input answer text and the obtained text indicating the details of the question. The answer system according to claim 1.

8. The aforementioned computing device is In the response determination process, a predetermined algorithm is used to determine whether the output response contains the content indicated by the first text or the second text, and if it is determined that the output response contains the content indicated by the first text or the second text, the output response is determined to be valid. The answer system according to claim 1.

9. The aforementioned computing device is In the response generation process, by inputting a prompt to the natural language processing model that includes the content indicated by the extracted first text and second text, the content indicated by the input question text, and a request for the portion of the database cited in the response to the input question, the model outputs the response to the input question and information of the quoted portion of the response. In the response determination process, it is determined whether the outputted quoted portion is included in the content indicated by the first text or the second text, and if it is determined that the outputted quoted portion is included in the content indicated by the first text or the second text, it is determined that the outputted response is valid. The answer system according to claim 8.

10. The aforementioned computing device is In the response determination process, the similarity between the content of the output response and the content indicated by the first text and the second text is calculated, and it is determined whether the calculated similarity is equal to or greater than a predetermined threshold. If it is determined that the calculated similarity is equal to or greater than the predetermined threshold, the output response is determined to be valid. The answer system according to claim 1.

11. The aforementioned computing device is If the outputted answer is determined to be invalid, a text modification process is executed to extract a third text and a fourth text from the database that are different from the first text and the second text and are related to the input question. In the response generation process, a prompt including the content of the extracted third and fourth texts and the content of the input question text is input to the natural language processing model to output a response to the input question. The answer system according to claim 1.

12. The aforementioned computing device is If the outputted response is deemed valid, the outputted response will be displayed on the display device. The answer system according to claim 1.

13. A method for providing an answer using an information processing device comprising a storage device that stores a database of texts or words related to business operations, a natural language processing model that takes natural language text prompts as input and outputs text corresponding to the input prompts, and an arithmetic unit, The aforementioned computing device A question reception process that accepts text input from users regarding the aforementioned business operations, A data extraction process that extracts a first text and a second text related to the input question from the aforementioned database, The answer generation process involves inputting a prompt to the natural language processing model that includes the content indicated by the extracted first text and second text, and the content indicated by the text of the input question, thereby outputting an answer to the input question. A response determination process that determines whether the outputted response is valid or not using a predetermined algorithm, If the outputted answer is determined to be invalid, a user confirmation process is performed in which a text containing information related to the question is generated using a predetermined algorithm, and the generated text is stored as a candidate text related to the input question. How to answer.

Images